Skip to content

Draw from CopilotKit's standard catalog, governed like the gallery - #873

Closed
guidovizoso wants to merge 6 commits into
mainfrom
standard-ui-governance
Closed

guidovizoso wants to merge 6 commits into
mainfrom
standard-ui-governance

Conversation

@guidovizoso

Copy link
Copy Markdown
Collaborator

What this changes

Bots can now draw interfaces from CopilotKit's standard catalog: A2UI 1.0, 68 components for layout, text, inputs, tables, charts, media and composites. Admins govern them the way they govern the gallery, so a Bot never draws a component an admin hasn't approved.

Five commits:

  1. Upgrade CopilotKit to 1.83.0. Moves @copilotkit/runtime and react-core from 1.77.0, and @ag-ui/* to 1.0.2. Tests now read SDK tools lazily, because the provider no longer re-renders its consumers after mount.
  2. Draw from the standard catalog. Uses standardUI without Map, which needs a map library OpenBot doesn't ship. The theme is mapped to OpenBot's tokens, so it follows the colours, radii and font in light and dark. The runtime's generativeUI follows OPENBOT_GENERATIVE_UI.
  3. Govern it on the server.
    • Rows: the 68 components are grouped into 57 families. A family keeps together parts that only work as a set, such as Table with its rows and cells. Each family is a row in the existing components table, named like a2ui.Table, so publishing, per-Bot withholding and edited descriptions work as they do for gallery components.
    • Rebuilt per run: a beforeRequestMiddleware throws away the catalog the browser sends. It rebuilds it from the server's own copy (shared/standard-ui-catalog.json), keeping only the families that Bot can use, in the published words.
    • Enforced: CopilotKit's GenerativeUIMiddleware then refuses any other component, whatever the browser sent.
    • Defaults: Layout, text and input families arrive published. Data, media, overlay and composite families arrive unpublished.
  4. List the families in Admin → UI components and in the components gallery. One section per group after the gallery tiles. Each row shows "On for every Bot", "Off for N Bots" or "Off" and opens the existing detail page.
  5. Preview each family. A family's page shows it drawn by the real SDK from a sample. The samples render in a provider of their own, apart from any conversation.

The browser keeps every standard component for drawing. A conversation from before a family was withdrawn still draws it.

Worth knowing

  • Favicon and LinkCard call Google. The SDK loads their icons from https://www.google.com/s2/favicons, so the site's domain goes to Google. Both arrive unpublished.
  • The full catalog still crosses the wire. The browser sends the whole catalog with every run because the tool transport has no includeCatalog: false. The server discards it. This needs an upstream change to stop.
  • Old surfaces no longer draw. A2UI 0.9 surfaces from before the upgrade don't draw when a conversation is reopened. The messages around them stay. This is in the changelog's "Before upgrading".
  • Rebuilding the catalog file. After an SDK upgrade, run bun run --cwd app export:standard-ui. A test fails if the file drifts from what the SDK sends.

Where it runs

  • New state that outlives a request? Only rows in the existing components table in Postgres, one per family. The rebuilt catalog is computed per request from those rows. There's no in-memory cache.
  • What happens on the second replica? Both replicas run syncCatalogue at boot. The insert is onConflictDoNothing on the unique name, so the second replica inserts nothing and reports nothing added. Every replica reads the same rows on every run.
  • Anything serialised? No. The insert above is the only write, and the unique index decides it.
  • Anything fanned out to a browser? None.
  • New listener, port, or schedule? None. The middleware runs inside the existing CopilotKit runtime route.

Boundary and audit

  • Every acting call still goes through the gateway. Drawing a surface isn't an acting call, and no gateway path changed.
  • New refusals and failures are recorded. A component outside the Bot's catalog is refused by the runtime and returned to the model in the tool result. Publishing and withholding families use the existing component routes, which already write audit rows. PUT /catalogue refuses the reserved a2ui. prefix with a 400.
  • Nothing new is trusted from the client. The server replaces the catalog the browser sends instead of filtering it. When a Bot has no families, the request names no agent, or its permissions can't be read, the run gets no catalog at all.

Changelog

  • Under Unreleased: "Bots draw from CopilotKit's standard components" and "CopilotKit 1.83". The "Before upgrading" list is updated.

Proof

  • Checks: format:check, lint and typecheck pass on all 6 packages, after a rebase onto 4bb0c0b4.
  • App tests (bun test ./app from the root): 1752 pass, 10 fail.
    • All 10 failures are in new-bot-sheet.test.tsx. Unmodified main fails the same 10 when the whole suite runs.
    • They pass when the file runs alone, on this branch and on main.
  • Server and toolkit tests against a fresh pgvector container: 5945 pass, 19 skip, 3 fail. The failures are the Mastra bridge test and the langgraph and bot "real provider sees the learned catalog" tests. They need per-package installs in the agent packages, which aren't workspaces.
  • Server tests for the catalog rewrite use the real GenerativeUIMiddleware: a granted Text draws and an ungranted Image is refused.
  • Previews: a test draws all 57 family samples and fails if the SDK rejects any of them.
  • Live, with a real Bot:
    • While Table was unpublished, the Bot said it couldn't use Table.
    • After I published it, the Bot drew one.
    • I unpublished it again.
    • I also checked the Admin and components gallery screens and the previews.

Bumps @copilotkit/react-core, @copilotkit/runtime and
@copilotkit/a2ui-renderer from 1.77.0 to 1.83.0 in app, server and
toolkit, and @ag-ui/core and @ag-ui/client to the 1.0.2 they require.

Two tests assumed 1.77 behavior:

- The provider no longer re-renders its consumers once after mount, so
  the tool-card tests now read the SDK's registry when they ask for a
  tool instead of when their capture component renders.
- "openai/..." models now use Chat Completions when OPENAI_BASE_URL is
  not an OpenAI or Azure host, as aimock is in tests, and a gpt-5
  model's instructions arrive as a developer message there.
A Bot with generative UI on now builds interfaces from CopilotKit's
standard catalog (A2UI 1.0) instead of the A2UI 0.9 basic catalog. The
browser passes it as the provider's generativeUI prop, still only when
the deployment has the capability, as openbot/ui@1: standardUI without
Map, which needs maplibre-gl. The theme points the --a2ui-* variables at
OpenBot's own tokens, which switch under .dark themselves, and the charts
at the gallery's series colours. The CSS overrides on the old catalog's
inline styles go.

The runtime sets generativeUI to the deployment's setting, so an
opted-out deployment refuses render_ui even from a client that sends a
catalog, and turns the A2UI 0.9 middleware off.

Surfaces drawn before this no longer render when a conversation is
reopened; the changelog says so.
Each family of CopilotKit's standard components (a component and the
parts that only work with it, such as Table with its rows and cells) is
now a row of the component gallery, named a2ui.<Head>. The server
registers the rows at boot from shared/standard-ui.ts; layout, text and
input families arrive published, the rest unpublished. The catalogue
route refuses the a2ui. prefix, so a browser cannot create those rows in
its own words.

The browser still sends the whole catalog with every run, because the
SDK's tool transport gives it no way not to. A beforeRequestMiddleware
now replaces it with one built from the server's copy
(shared/standard-ui-catalog.json, generated from the SDK by
`bun run --cwd app export:standard-ui`): only the families the Bot
holds, each head in its published description. It fails closed: no
families, no agent named, or an unreadable grant mean the run carries no
catalog and no render_ui tool. The runtime then refuses any component
the rebuilt catalog does not list, which a test drives through the
SDK's own GenerativeUIMiddleware.

The browser keeps every component for drawing, so a conversation still
renders what it drew before a family was withdrawn.
The admin component list and the people's gallery now show the
standard catalog's families after the gallery's own components, one
section per group (Layout, Text, Inputs, Data, Media, Feedback and
overlays, Composites). They are rows rather than tiles: there are 57 of
them, and a Bot arranges them into an interface of its own, so a tile
showing one alone says little. In Admin each row says where the family
stands (On for every Bot, Off for 2 Bots, Off) and opens the existing
detail page, where publishing, the description and the per-Bot grants
work as for any component. The people's gallery lists only published
families, with their published description.

On a family's page the data-functions row is gone, since building blocks
read no data, and Details names the group and the components the family
includes. Its preview no longer says this build cannot draw it.
A family's page in Admin and in the people's gallery now shows it drawn
by the SDK, from a small sample interface in
app/src/components/building-block-samples.ts, on a card as a conversation
draws it. Each preview runs in a CopilotKitProvider of its own, with no
runtime and no inspector, on a thread nobody talks on, so it cannot
reach or disturb a conversation. The people's gallery rows now open
that page, which names the family's group and members.

app/tests/building-block-samples.test.tsx draws every sample through
the SDK and fails on any operation it rejects, so a sample cannot drift
from the catalog unnoticed.
The artwork's viewBox was 1200x800, but its gradient stops at 754. On a gallery tile, which is wider than 3:2, the whole viewBox showed and left a white band about 12px tall along the bottom.
@guidovizoso

Copy link
Copy Markdown
Collaborator Author

Closing in favour of a smaller PR that only upgrades to A2UI 1.0. The governance work stays on this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant