Skip to content

The OpenBot Recorder and the OpenBot Browser: teach a Bot by doing it in your own browser - #858

Merged
davidmckayv merged 485 commits into
mainfrom
david/recorder-and-browser
Oct 28, 2026
Merged

davidmckayv merged 485 commits into
mainfrom
david/recorder-and-browser

Conversation

@davidmckayv

Copy link
Copy Markdown
Contributor

Brings the OpenBot Recorder and the OpenBot Browser into OpenBot. Both were built and reviewed in a private repository as two stacked pull requests; this lands them here as one, on top of current main.

The Recorder. A Chrome extension a person uses to teach a Bot by doing: they press Record, do a task in their own browser while saying what they are doing, and the clicks, typing, pages, network calls and narration go straight into their own Learning space in Intelligence as a trajectory, where Learning turns it into a skill a Bot can use. Settings → Browsers connects a browser; each person gets their own Learning space and chooses how their skills are shared.

The OpenBot Browser. A button in OpenBot opens a real Chrome window the person uses as their own, with the Recorder already installed and connected, and the Browser page records a named task in it. It records the person, never the Bot. OpenBot Desktop opens and connects it on macOS, Windows and Linux.

Before upgrading. Two migrations add the Recorder's tables, after main's 0065:

  • 0066_recorder_extension: six new tables (recorder_connect_codes, recorder_connections, recorder_trajectory_bindings, learning_person_spaces, learning_skill_sharing, learning_open_skills). Nothing existing changes.
  • 0067_recorder_activity: two nullable columns on recorder_connections.

The CopilotKit runtime moves to 1.79.1, with COPILOTKIT_OPENAI_API choosing the API an OpenAI-compatible endpoint is called with; the CHANGELOG's Unreleased section has every item.

Reviewing. Most of the size is the extension (its source, and recorded agent traffic it is tested against in extension/tests/fixtures) and the two drizzle snapshots. The server's part is the Recorder and Browser routes, connect codes and the per-person Learning spaces; the app's part is Settings → Browsers and the Browser page.

Checked before publishing. No keys or tokens in the added lines, no customer names or internal links in the code or the commit messages, every dependency public (including the patched @copilotkit/learning@0.0.1), and no new CI secret. The extension's signing key is not in the repository.

…table code, each message within a deadline below the extension's wait

The ping no longer runs the handoff's process ownership checks (lsof, or PowerShell and netstat on Windows): it reads whether a deployment is selected, asks the app's port for /api/capabilities, and in organization mode reads whether Desktop holds a sign-in for the organization, without the network. Every reply now carries ok, and a failed one carries error, one of origin-not-allowed, not-installed, not-running, not-signed-in, organization-unavailable, handoff-failed, timed-out, host-failed or unknown-message. Each message runs on its own thread and is answered timed-out at 2 s for a ping and 7 s for a handoff, inside the extension's 3 s and 8 s. On this machine a release ping now takes about 3 ms (it took about 175 ms) and a handoff about 175 ms. The host reads the host name and extension id from the install module's one copy, and a test runs a full organization handoff and checks that the host's stderr holds no connect code and no part of a cookie.
…ment

Desktop never builds or runs the extension; the frozen bun install needs only its workspace manifest, so its source, tests and assets now stay out of every deployment. A simulated deployment with only that file still installs with bun install --frozen-lockfile.
…its host registrations, macOS and Linux keep the per-person host file until someone deletes it
…Recorder records no screen, so nothing may call tabCapture
…utcome naming why: its live events are already in Intelligence, so it never reads as a recording the person finished
…olicy pin elsewhere both end the live recording, halt the document's streams for its token and keep everything for its person; a pin also revokes the token, a new gateway list reaches running streams, and Connect and Record re-check the policy
… connecting revokes the old token, so the live recording, its stored session and the document's stream take the new one, and a stream the old token already signed out resumes from the outbox
… once its port stays down past its reconnect window, reopen it for a Stop that finds it gone, and refuse Start over while it cannot be reached, so events are not dropped and Stop does not wait for a browser restart
…s it and counts the dropped events, and Stop reports that nothing reached Intelligence (a resumed recording keeps what it stored, parked) instead of Sent to Intelligence
…ne and the service worker told, so closed records no longer pile up and a discard's walk stays short; a recover that finds a recording's records gone sends nothing again
…m before the rework left (its other stores, other meta keys and demonstration trajectories with their events), and read stored recordings through a shape check so an old build's demonstration is never sent
…how a plain sentence for it in the popup (a general one for a code the Recorder does not know), since a good ping no longer promises a good handoff; the host-deadline comments now cite its 2 s ping and 7 s handoff
… progress and the stopped recordings still sending to Intelligence, as well as the kept events
…stead of again under the Bot picker, pin Stop and Start over to the bottom so the 50-action notice cannot push them past Chrome's 600px, and have the popup and the page notice both say the count it opened at, "or more"
… a policy value of the wrong type, which then reads as no restriction: a schema names exactly one type, so IT is told to check chrome://policy, and a test keeps the schema to the rules Chrome loads
…he wrong type, which then means no restriction, so IT must see status OK in chrome://policy; only a value of the right type that the Recorder cannot use is refused
…s to it in the changelog, .env.example and the docs index
…the release tree, so the check accepts the Recorder extension's package.json and still fails when a workspace's manifest is missing
… shared length, the longest that fits, so the larger body is cut first and a small response survives a large upload whole instead of being emptied before the request is touched
…er instead of the offscreen document calling chrome.runtime.getManifest, which Chrome does not give an offscreen document, so kept recordings are sent again after a reconnect or a browser restart; the chrome fake now gives each context its own chrome, the offscreen document only runtime messaging, getURL and id, and a new test runs the real service worker and document together through both paths
… still stands and end the recording through the sign-out path on a revoked or signed-out answer, so a browser revoked in OpenBot's Settings stops streaming within 30 s instead of at the next popup open or 12-hour keep-alive; an unreachable OpenBot or an error page keeps the recording going, and a restarted service worker resumes the checks
…n in ten minutes so an ordinary connect, reconnect, Disconnect and revoke run never meets it, and answer a connect over the limit as throttled with the time left instead of as busy, so Settings says to try again in so many minutes rather than that another connect is under way
… number or typed secret in spoken numbers, which speech recognition writes with commas, dashes or a currency sign, then runs the usual text masking
…he offscreen document opens the microphone and runs the browser's own speech recognition on it, each final phrase goes into the recording as a masked transcript event through the same outbox and stream, and Stop, Disconnect, a sign-out or a policy move ends it with the recording; OpenBot receives nothing of it
…itle says it is listening with the last words heard, or why it is not in a plain sentence, with a switch to mute it; the first Record opens a microphone page once to ask for the lasting grant the hidden document needs, and Record sends the page's language
…cript now redacts every spoken run of six digits or more whatever its checksum (a date said in full excepted), and every number after a word that names a secret (card, PIN, CVV, account, password and the like) however short, including in the phrase after it, since Chrome splits a phrase at a pause; the popup's interim words go through the same masking with the phrase before them, so a card number read aloud no longer reaches Intelligence or the Voice row as "4012-888888-1881" or "4012-8"
…no words: as Edge does, recognition starts, hears speech and raises no error while every final phrase comes back empty, so the row said Listening forever; now, 20 s after speech is first heard with no phrase in words, the row says this browser is not turning your speech into words and recording carries on without narration, keeps listening, and drops the sentence when a real phrase arrives
… Record records: website access is now a required host permission that the install grants (the debugger permission already implies every site, so Chrome and Edge show the same install warnings and an update is no privilege increase), and a fresh install opens one setup page that asks for the microphone as it loads, with a button to ask again and the answer in plain words, worded for Chrome's or Edge's prompt; the access page, the website-access prompts at Connect and at Record, and the microphone tab the first Record opened are gone, Record never opens a tab, and a recording without the microphone goes on while its Voice row says the microphone is not allowed and offers the setup page on a click
…s joins no Learning space: the popup said Learning may not see it whenever OpenBot could not check the Bot's space with Intelligence (Learning delivery off, its skills endpoint answering 404, or Intelligence not answering), although the pass still joins the Bot's container and the trajectory lands in it; one shared rule, joinedLearningContainer, now decides both which container the server's selector joins and when the popup warns, so a Bot with no space, a container Intelligence says does not exist, or settings OpenBot could not read still warn, and after Stop an unchecked space says the recording goes into the Bot's Learning space
The mark is the sign that a tab is being recorded, and it lives in the page's DOM with an open shadow
root, so a page could hide it with one CSS rule, remove it, or reach inside, and nothing put it back.
Its shadow root is now closed (the end-to-end build keeps it open for its harness). What makes it seen
is set on its host from inside the shadow root, important, which outranks every important page rule
and any inline style on the host (CSS Cascade 5, context; measured in Chromium), and where it hangs is
set on the mark inside. Removed, or the document replaced, it is put back at once, hidden and inert are
taken off it again, and it stays the root element's last child, so nothing added later is drawn over
it at the same z-index: at most ten moves a look, so a script that also wants to be last cannot make
the two trade places as fast as they can.
A key held when control was handed back, the screen closed or its stream ended never had its release
sent, so a Shift or a Control could stay down in the remote browser and change every key and click
after it. The keys sent down are kept with what was sent, and released when forwarding stops, before
the socket closes.
…d wait on it

openbot.connect.status kept answering cancelled for the whole browser session after a Cancel, until
the same origin asked for a web connect again, so a page that later connected another way (as the
organization's pinned OpenBot, or OpenBot Desktop's) still read cancelled. A Cancel is now news within
the five minutes a request lasts and until that origin is connected, and is forgotten after.
Every row's Disconnect dialog said OpenBot signs the Recorder in this browser out, on another
browser's row too. It now says this browser only on the row of the copy in the browser the page is
open in, and names every other by the name its row shows (your OpenBot browser for that one).
Tests and a few of Desktop's comments said "the code-owner review", "blocker 3" or "end-to-end run 1
of the review fixes" where the reason belonged. Each now says the defect it pins or what a person
saw, in the code's own terms. A guard reads the sources and tests of those parts for the phrases.
…ign out half way

Sign-out no longer fails open. Ending the person's Recorder connections is
retried three times; when it still fails, recorder.sign_out_failed goes on
the audit trail and the sign-out does not happen: the person stays signed
in and is told "Signing out did not finish. Try again." That follows the
code's own rule for ending access: a removal ends sessions and deny-lists in
one transaction, a removed person's linked accounts retire with their rows
in one transaction, and a connection ends exactly when its row is written.
A sign-out reported done while a Recorder went on recording as the person
would break it. With Better Auth the Recorders end in a before hook on
/sign-out, since the sign-out route itself answers success when deleting the
session fails; in organization mode they end before the organization is
asked to end its session. The app's Sign out now says the sentence in a
toast and stays put instead of failing silently.

Every other end of a sign-in ends the Recorders too: Better Auth's session
delete hook (signing out everywhere, ending one or the other sessions, a
password reset or change, SCIM deactivation, single logout, an account
proving the address), a person's removal (with what else a removal
retires), and SSO-required being turned on (before the sessions, and for
everybody holding a connection, so saving again finishes a failed save). In
organization mode a connection ends when the organization refuses the
session it holds a copy of, however that session ended there. An expired
session that Better Auth tidies away is not an end, and ends nothing.

The organization session cookie now carries its own expiry, set at the
handoff and enforced when it is read, and a connection's sealed copy ends
at that expiry (kept in organization_session_expires_at, never later than
the cookie's Max-Age from connecting) instead of a week from connecting.

The docs say a Recorder connection's lifetime, and that recordings cannot
be deleted from OpenBot yet and stay in Intelligence after the person is
removed, because Intelligence has no API a server credential can call for
either.
@davidmckayv

Copy link
Copy Markdown
Contributor Author

Thanks for the re-review and the approval. The follow-ups are in the latest push, each with a regression test that fails without it.

  1. Sign-out fails open. Ending a person's Recorder connections is retried, and if it still fails a recorder.sign_out_failed row is written and the sign-out does not happen ("Signing out did not finish. Try again."), since ending access here is all or nothing. In Better Auth this runs in a hook before /sign-out, because Better Auth answers success even when deleting the session fails.
  2. Every other end of a sign-in ends the Recorders. A hook before any live session is deleted covers sign out everywhere, ending other sessions, password reset and change, SCIM deactivation, IdP single logout and an email proved by another account; removing a person, and turning SSO-required on, end them too; in organization mode a connection ends when the organization refuses the session it holds a copy of. If the Recorders cannot be ended, the session is not deleted either.
  3. The mark. A closed shadow root; its visibility rules are set !important on its host from inside the root, so no page rule or inline style outranks them; removing it, replacing the document, or setting hidden or inert puts it back at once, and it stays the root's last child (bounded, so it cannot loop with another script doing the same). The top layer (a modal dialog) can still cover it, as it covers anything.
  4. An OpenBot page starting a recording with no confirmation. Kept as designed: only the OpenBot the Recorder is connected to can do it, from its own page's Record button, and the mark shows in the recorded tab from the first step.
  5. Native host spawns. Every openbot.connect is counted before Desktop is asked for its address (30 requests from a loopback page now start the host 10 times, not 30).
  6. Narration choice. A choice still loading is no longer "never chosen": Record waits for it (up to 3 s) and, if it cannot be read, records without narration and leaves the saved choice alone. Record-here passes the account's choice through openbot.record.
  7. Still open from the first round:
    • Deleting a recording, and a person's trajectories outliving their removal: still needs an Intelligence API a server credential can call. It is now documented in docs/configuration.md and docs/automatic-learning.md.
    • Organization mode's session copy: now bounded by the organization cookie's own expiry (written when the cookie is issued, capped at 7 days from now), in a new column (migration 0068_recorder_organization_session_expiry).
    • macOS and AppImage uninstall: still no hook to run; documented, and part of the same Desktop path we will come back to.
    • restrictStorageToTrustedContexts(): a refusal is logged and retried at the next worker start, the rule recovery.ts already sets for top-level work.
    • The two Chrome switches: they matter for the person's own Chrome on Desktop. For now the browser inside OpenBot is the focus, so that path stays as it is, switches included, and I will decide them when we come back to it.
  8. Scope. We kept the 1.79.1 bump here. Looking into it found a real regression: behind an OpenAI-compatible proxy, 1.79.1 sent gpt-5.6-terra and other Responses-only models to Chat Completions, where every turn with tools failed. OpenBot now sets COPILOTKIT_OPENAI_API=responses at boot for those models when OPENAI_BASE_URL is set and the operator has not chosen, with a test against recorded OpenAI traffic for both APIs (18 cases) and one live check through a proxy.

Nits: held keys are released in the remote browser when the screen closes or control goes back; a Cancel is reported only within the connect request's five minutes; the Disconnect dialog names the row's own browser; and comments now state the defect rather than citing the review, with a test that keeps it that way.

Also merged main again (#853's full-width audit log now lives in the split page parts).

…aving `via` for how a connection was made

`via` on Recorder rows says how a connection was made (web, desktop, openbot-browser), as `recorder.connected` and
`recorder.connect_refused` carry it. The sign-out follow-ups also put the way a person's sign-in ended (a sign-out,
sessions ended another way, the person removed, SSO required, the organization) in `via` on `recorder.revoked` and
`recorder.sign_out_failed`, so one field meant two things. That is `ending` now, through the store, the sign-out and
every caller, and the trail's descriptions say so. The organization ending a session is pinned on the trail too.
davidmckayv added a commit that referenced this pull request Oct 28, 2026
guidovizoso
guidovizoso previously approved these changes Oct 28, 2026

@guidovizoso guidovizoso left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at b4a39b4f. Approving again. The new commits close most of the follow-ups:

  • Sign-out fails closed. Ending the Recorders is retried, a failure writes recorder.sign_out_failed, and the sign-out doesn't happen. That's stricter than I asked for, and I prefer it.
  • Every end of a sign-in ends the Recorders. That covers sign out everywhere, password reset, SCIM, single logout, removal and SSO-required. I checked the two places that delete session rows directly (sso-required.ts, people/store.ts). Both end the Recorders first.
  • The organization session copy now ends when the cookie does.
  • The recording mark can't be hidden or removed by the page.
  • The connect request is counted before the native host is asked for its address.
  • restrictStorageToTrustedContexts() is now caught.
  • The narration choice is no longer asked again while it loads, and Record-here passes it on.
  • Keys are released when the live screen stops forwarding them.
  • The stale "cancelled" is gone, the Disconnect dialog names the right browser, and the test comments no longer cite the review.

Before merging:

  1. It conflicts with main in 5 files, one of them server/src/agents/agent-resolver.ts. That's where actor.id goes into acquireLearnedSkills, so please check per-person Skills still reach a run after the rebase.

Still open (not blocking):
2. An OpenBot page can still start a recording with no confirmation in the extension (openbot-page.ts, page.record). If you think that's acceptable because OpenBot's own top frame is trusted, please say so here. An XSS on OpenBot would then be a silent recording.
3. The OpenAI proxy fix (79316f55) looks right and the regression test is good. Two notes:

  • chooseOpenAIApi sets COPILOTKIT_OPENAI_API for the whole process, so every openai/* model the runtime resolves goes to /responses, the plan model included (copilot.ts:465). The configuration.md line reads as if it's decided per model.
  • The wider requiresResponsesApi also makes agent-bot refuse the -pro and -codex models at boot. That's correct, but it's a behaviour change outside this feature and worth a CHANGELOG line.

It also fixes a regression the 1.79.1 bump caused, which is the case for making that bump revertable on its own. I still won't block on the split.
4. Deleting recordings and recordings outliving the person are now documented. Please open an Intelligence ticket for a delete API, so the docs point somewhere.
5. macOS still has no uninstall cleanup. Windows and Linux are done.
6. Someone on the product side should still sign off on the AutomationControlled / --enable-unsafe-extension-debugging flags.

Resolved conflicts:
- CopilotKit 1.83.0 from main (app, server, toolkit), and the lockfile regenerated.
- The resolver keeps #858's person-bound learned Skills and main's name context.
- Admin keeps the Skill sharing switch under main's reworded Network copy.
- The changelog's upgrade list is counted from what it now holds (fourteen).
- The tool-capture test helpers and both CI test fixes take main's versions.
- The assertion-shape guard points at the shifted line.
Brings in the skins (#856). The account keeps both the narration choice and
the skin, the sidebar draws the shared navigation list with the Browser button
after Connections, and PageHeader carries the page-header and page-title hooks
the skins read.
@davidmckayv
davidmckayv merged commit f678b33 into main Oct 28, 2026
26 checks passed
@davidmckayv
davidmckayv deleted the david/recorder-and-browser branch October 28, 2026 21:41
davidmckayv added a commit that referenced this pull request Oct 28, 2026
davidmckayv added a commit that referenced this pull request Oct 28, 2026
Brings in the Recorder and the OpenBot Browser (#858). The migration is renumbered
0069_isolated_bots, after #858's 0066 to 0068. A person's own browser is always a
computer of its own, never a seat on the person's computer beside their Bots: the
seat resolver says so, the browser routes locate it by its own key, and the
computer refuses it by the id named rather than by the seat made from it. A
Sandbox is cut by computer key: an isolated Bot's names its Bot and keeps its files
at the root of the volume it had, and a person's names no Bot and gives each Bot
a folder.
dustywusty added a commit to dustywusty/openbot that referenced this pull request Oct 29, 2026
Upstream added `extension` (CopilotKit#858) to the desktop's package list; v0.2.0 has no
`extension` either, so it joins `shared` and `toolkit` as a later package that
is read only when the release includes it.
dustywusty added a commit that referenced this pull request Oct 29, 2026
…ering

A failed save rolls the cache back after cleanup. Unregistering happy-dom
at once removes window under it, and React's read of window.event throws
between tests, failing the run with every test passed. Same change as
main's #858.
dustywusty added a commit that referenced this pull request Oct 29, 2026
Takes main's engine choice (#831): Docker Desktop is opened only when it
has run before, and each installation stays on the engine it was set up
on. The old copy of #831 on this branch is dropped, including
after_engine_install, which duplicated main's answering_after_install
and ignored the saved engine. Kept from this branch: path_with_docker,
downloads chosen by the machine's architecture on Windows on ARM, and
the per-user Docker Desktop location on Windows.

Takes main's #823 and #821 (StartFresh replaces DatabaseReset; the local
Intelligence screen offers it too), #860, and #858's Recorder step label.

The web app keeps main's theme and skin picker (#856); this branch's
CopilotKit reskin of the app and its two fonts are dropped. Its desktop
CSS (room for the macOS window buttons, sidebar resize) stays.
dustywusty added a commit that referenced this pull request Oct 29, 2026
…containers

Main's OpenBot Browser (#858) runs agent-computer on the host with Bun, to
open the person's own Chrome or Edge. The server and worker are containers
now, so the browser is the only host process left:

- Start prepares Bun and the packages the first time it has a browser to
  run, as the Intelligence sign-in helper does, after the server answers.
  It spawns the browser under the startup lock, records it so a later
  window can end it, and holds it.
- The server container reaches it at host.docker.internal
  (OPENBOT_BROWSER_CONTAINER_URL, which docker-compose.yml prefers), since
  the computer listens on the host's loopback.
- It is restarted with backoff when it dies. Giving up says why, with the
  tail of its log, and leaves OpenBot running.
- Stop closes the held browser before the recorded processes are ended,
  so Chrome closes its windows and profile first.
dustywusty added a commit that referenced this pull request Oct 29, 2026
…containers

Main's OpenBot Browser (#858) runs agent-computer on the host with Bun, to
open the person's own Chrome or Edge. The server and worker are containers
now, so the browser is the only host process left:

- Start prepares Bun and the packages the first time it has a browser to
  run, as the Intelligence sign-in helper does, after the server answers.
  It spawns the browser under the startup lock, records it so a later
  window can end it, and holds it.
- The server container reaches it at host.docker.internal
  (OPENBOT_BROWSER_CONTAINER_URL, which docker-compose.yml prefers), since
  the computer listens on the host's loopback.
- It is restarted with backoff when it dies. Giving up says why, with the
  tail of its log, and leaves OpenBot running.
- Stop closes the held browser before the recorded processes are ended,
  so Chrome closes its windows and profile first.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants