Repository navigation
The OpenBot Recorder and the OpenBot Browser: teach a Bot by doing it in your own browser - #858
Conversation
…table code, each message within a deadline below the extension's wait The ping no longer runs the handoff's process ownership checks (lsof, or PowerShell and netstat on Windows): it reads whether a deployment is selected, asks the app's port for /api/capabilities, and in organization mode reads whether Desktop holds a sign-in for the organization, without the network. Every reply now carries ok, and a failed one carries error, one of origin-not-allowed, not-installed, not-running, not-signed-in, organization-unavailable, handoff-failed, timed-out, host-failed or unknown-message. Each message runs on its own thread and is answered timed-out at 2 s for a ping and 7 s for a handoff, inside the extension's 3 s and 8 s. On this machine a release ping now takes about 3 ms (it took about 175 ms) and a handoff about 175 ms. The host reads the host name and extension id from the install module's one copy, and a test runs a full organization handoff and checks that the host's stderr holds no connect code and no part of a cookie.
…ment Desktop never builds or runs the extension; the frozen bun install needs only its workspace manifest, so its source, tests and assets now stay out of every deployment. A simulated deployment with only that file still installs with bun install --frozen-lockfile.
…its host registrations, macOS and Linux keep the per-person host file until someone deletes it
…Recorder records no screen, so nothing may call tabCapture
…utcome naming why: its live events are already in Intelligence, so it never reads as a recording the person finished
…olicy pin elsewhere both end the live recording, halt the document's streams for its token and keep everything for its person; a pin also revokes the token, a new gateway list reaches running streams, and Connect and Record re-check the policy
… connecting revokes the old token, so the live recording, its stored session and the document's stream take the new one, and a stream the old token already signed out resumes from the outbox
… once its port stays down past its reconnect window, reopen it for a Stop that finds it gone, and refuse Start over while it cannot be reached, so events are not dropped and Stop does not wait for a browser restart
…sage, so the extension typechecks
…s it and counts the dropped events, and Stop reports that nothing reached Intelligence (a resumed recording keeps what it stored, parked) instead of Sent to Intelligence
…ne and the service worker told, so closed records no longer pile up and a discard's walk stays short; a recover that finds a recording's records gone sends nothing again
…m before the rework left (its other stores, other meta keys and demonstration trajectories with their events), and read stored recordings through a shape check so an old build's demonstration is never sent
…how a plain sentence for it in the popup (a general one for a code the Recorder does not know), since a good ping no longer promises a good handoff; the host-deadline comments now cite its 2 s ping and 7 s handoff
… progress and the stopped recordings still sending to Intelligence, as well as the kept events
…stead of again under the Bot picker, pin Stop and Start over to the bottom so the 50-action notice cannot push them past Chrome's 600px, and have the popup and the page notice both say the count it opened at, "or more"
… a policy value of the wrong type, which then reads as no restriction: a schema names exactly one type, so IT is told to check chrome://policy, and a test keeps the schema to the rules Chrome loads
…he wrong type, which then means no restriction, so IT must see status OK in chrome://policy; only a value of the right type that the Recorder cannot use is refused
…s to it in the changelog, .env.example and the docs index
…the release tree, so the check accepts the Recorder extension's package.json and still fails when a workspace's manifest is missing
… shared length, the longest that fits, so the larger body is cut first and a small response survives a large upload whole instead of being emptied before the request is touched
…er instead of the offscreen document calling chrome.runtime.getManifest, which Chrome does not give an offscreen document, so kept recordings are sent again after a reconnect or a browser restart; the chrome fake now gives each context its own chrome, the offscreen document only runtime messaging, getURL and id, and a new test runs the real service worker and document together through both paths
… still stands and end the recording through the sign-out path on a revoked or signed-out answer, so a browser revoked in OpenBot's Settings stops streaming within 30 s instead of at the next popup open or 12-hour keep-alive; an unreachable OpenBot or an error page keeps the recording going, and a restarted service worker resumes the checks
…n in ten minutes so an ordinary connect, reconnect, Disconnect and revoke run never meets it, and answer a connect over the limit as throttled with the time left instead of as busy, so Settings says to try again in so many minutes rather than that another connect is under way
… number or typed secret in spoken numbers, which speech recognition writes with commas, dashes or a currency sign, then runs the usual text masking
…he offscreen document opens the microphone and runs the browser's own speech recognition on it, each final phrase goes into the recording as a masked transcript event through the same outbox and stream, and Stop, Disconnect, a sign-out or a policy move ends it with the recording; OpenBot receives nothing of it
…itle says it is listening with the last words heard, or why it is not in a plain sentence, with a switch to mute it; the first Record opens a microphone page once to ask for the lasting grant the hidden document needs, and Record sends the page's language
…cript now redacts every spoken run of six digits or more whatever its checksum (a date said in full excepted), and every number after a word that names a secret (card, PIN, CVV, account, password and the like) however short, including in the phrase after it, since Chrome splits a phrase at a pause; the popup's interim words go through the same masking with the phrase before them, so a card number read aloud no longer reaches Intelligence or the Voice row as "4012-888888-1881" or "4012-8"
…no words: as Edge does, recognition starts, hears speech and raises no error while every final phrase comes back empty, so the row said Listening forever; now, 20 s after speech is first heard with no phrase in words, the row says this browser is not turning your speech into words and recording carries on without narration, keeps listening, and drops the sentence when a real phrase arrives
… Record records: website access is now a required host permission that the install grants (the debugger permission already implies every site, so Chrome and Edge show the same install warnings and an update is no privilege increase), and a fresh install opens one setup page that asks for the microphone as it loads, with a button to ask again and the answer in plain words, worded for Chrome's or Edge's prompt; the access page, the website-access prompts at Connect and at Record, and the microphone tab the first Record opened are gone, Record never opens a tab, and a recording without the microphone goes on while its Voice row says the microphone is not allowed and offers the setup page on a click
…s joins no Learning space: the popup said Learning may not see it whenever OpenBot could not check the Bot's space with Intelligence (Learning delivery off, its skills endpoint answering 404, or Intelligence not answering), although the pass still joins the Bot's container and the trajectory lands in it; one shared rule, joinedLearningContainer, now decides both which container the server's selector joins and when the popup warns, so a Bot with no space, a container Intelligence says does not exist, or settings OpenBot could not read still warn, and after Stop an unchecked space says the recording goes into the Bot's Learning space
The mark is the sign that a tab is being recorded, and it lives in the page's DOM with an open shadow root, so a page could hide it with one CSS rule, remove it, or reach inside, and nothing put it back. Its shadow root is now closed (the end-to-end build keeps it open for its harness). What makes it seen is set on its host from inside the shadow root, important, which outranks every important page rule and any inline style on the host (CSS Cascade 5, context; measured in Chromium), and where it hangs is set on the mark inside. Removed, or the document replaced, it is put back at once, hidden and inert are taken off it again, and it stays the root element's last child, so nothing added later is drawn over it at the same z-index: at most ten moves a look, so a script that also wants to be last cannot make the two trade places as fast as they can.
A key held when control was handed back, the screen closed or its stream ended never had its release sent, so a Shift or a Control could stay down in the remote browser and change every key and click after it. The keys sent down are kept with what was sent, and released when forwarding stops, before the socket closes.
…d wait on it openbot.connect.status kept answering cancelled for the whole browser session after a Cancel, until the same origin asked for a web connect again, so a page that later connected another way (as the organization's pinned OpenBot, or OpenBot Desktop's) still read cancelled. A Cancel is now news within the five minutes a request lasts and until that origin is connected, and is forgotten after.
Every row's Disconnect dialog said OpenBot signs the Recorder in this browser out, on another browser's row too. It now says this browser only on the row of the copy in the browser the page is open in, and names every other by the name its row shows (your OpenBot browser for that one).
Tests and a few of Desktop's comments said "the code-owner review", "blocker 3" or "end-to-end run 1 of the review fixes" where the reason belonged. Each now says the defect it pins or what a person saw, in the code's own terms. A guard reads the sources and tests of those parts for the phrases.
…ign out half way Sign-out no longer fails open. Ending the person's Recorder connections is retried three times; when it still fails, recorder.sign_out_failed goes on the audit trail and the sign-out does not happen: the person stays signed in and is told "Signing out did not finish. Try again." That follows the code's own rule for ending access: a removal ends sessions and deny-lists in one transaction, a removed person's linked accounts retire with their rows in one transaction, and a connection ends exactly when its row is written. A sign-out reported done while a Recorder went on recording as the person would break it. With Better Auth the Recorders end in a before hook on /sign-out, since the sign-out route itself answers success when deleting the session fails; in organization mode they end before the organization is asked to end its session. The app's Sign out now says the sentence in a toast and stays put instead of failing silently. Every other end of a sign-in ends the Recorders too: Better Auth's session delete hook (signing out everywhere, ending one or the other sessions, a password reset or change, SCIM deactivation, single logout, an account proving the address), a person's removal (with what else a removal retires), and SSO-required being turned on (before the sessions, and for everybody holding a connection, so saving again finishes a failed save). In organization mode a connection ends when the organization refuses the session it holds a copy of, however that session ended there. An expired session that Better Auth tidies away is not an end, and ends nothing. The organization session cookie now carries its own expiry, set at the handoff and enforced when it is read, and a connection's sealed copy ends at that expiry (kept in organization_session_expires_at, never later than the cookie's Max-Age from connecting) instead of a week from connecting. The docs say a Recorder connection's lifetime, and that recordings cannot be deleted from OpenBot yet and stay in Intelligence after the person is removed, because Intelligence has no API a server credential can call for either.
… and what ends a Recorder's connection
|
Thanks for the re-review and the approval. The follow-ups are in the latest push, each with a regression test that fails without it.
Nits: held keys are released in the remote browser when the screen closes or control goes back; a Cancel is reported only within the connect request's five minutes; the Disconnect dialog names the row's own browser; and comments now state the defect rather than citing the review, with a test that keeps it that way. Also merged main again (#853's full-width audit log now lives in the split page parts). |
…aving `via` for how a connection was made `via` on Recorder rows says how a connection was made (web, desktop, openbot-browser), as `recorder.connected` and `recorder.connect_refused` carry it. The sign-out follow-ups also put the way a person's sign-in ended (a sign-out, sessions ended another way, the person removed, SSO required, the organization) in `via` on `recorder.revoked` and `recorder.sign_out_failed`, so one field meant two things. That is `ending` now, through the store, the sign-out and every caller, and the trail's descriptions say so. The organization ending a session is pinned on the trail too.
guidovizoso
left a comment
There was a problem hiding this comment.
Re-reviewed at b4a39b4f. Approving again. The new commits close most of the follow-ups:
- Sign-out fails closed. Ending the Recorders is retried, a failure writes
recorder.sign_out_failed, and the sign-out doesn't happen. That's stricter than I asked for, and I prefer it. - Every end of a sign-in ends the Recorders. That covers sign out everywhere, password reset, SCIM, single logout, removal and SSO-required. I checked the two places that delete session rows directly (
sso-required.ts,people/store.ts). Both end the Recorders first. - The organization session copy now ends when the cookie does.
- The recording mark can't be hidden or removed by the page.
- The connect request is counted before the native host is asked for its address.
restrictStorageToTrustedContexts()is now caught.- The narration choice is no longer asked again while it loads, and Record-here passes it on.
- Keys are released when the live screen stops forwarding them.
- The stale "cancelled" is gone, the Disconnect dialog names the right browser, and the test comments no longer cite the review.
Before merging:
- It conflicts with main in 5 files, one of them
server/src/agents/agent-resolver.ts. That's whereactor.idgoes intoacquireLearnedSkills, so please check per-person Skills still reach a run after the rebase.
Still open (not blocking):
2. An OpenBot page can still start a recording with no confirmation in the extension (openbot-page.ts, page.record). If you think that's acceptable because OpenBot's own top frame is trusted, please say so here. An XSS on OpenBot would then be a silent recording.
3. The OpenAI proxy fix (79316f55) looks right and the regression test is good. Two notes:
chooseOpenAIApisetsCOPILOTKIT_OPENAI_APIfor the whole process, so everyopenai/*model the runtime resolves goes to/responses, the plan model included (copilot.ts:465). Theconfiguration.mdline reads as if it's decided per model.- The wider
requiresResponsesApialso makesagent-botrefuse the-proand-codexmodels at boot. That's correct, but it's a behaviour change outside this feature and worth a CHANGELOG line.
It also fixes a regression the 1.79.1 bump caused, which is the case for making that bump revertable on its own. I still won't block on the split.
4. Deleting recordings and recordings outliving the person are now documented. Please open an Intelligence ticket for a delete API, so the docs point somewhere.
5. macOS still has no uninstall cleanup. Windows and Linux are done.
6. Someone on the product side should still sign off on the AutomationControlled / --enable-unsafe-extension-debugging flags.
Resolved conflicts: - CopilotKit 1.83.0 from main (app, server, toolkit), and the lockfile regenerated. - The resolver keeps #858's person-bound learned Skills and main's name context. - Admin keeps the Skill sharing switch under main's reworded Network copy. - The changelog's upgrade list is counted from what it now holds (fourteen). - The tool-capture test helpers and both CI test fixes take main's versions. - The assertion-shape guard points at the shifted line.
Brings in the skins (#856). The account keeps both the narration choice and the skin, the sidebar draws the shared navigation list with the Browser button after Connections, and PageHeader carries the page-header and page-title hooks the skins read.
Brings in the Recorder and the OpenBot Browser (#858). The migration is renumbered 0069_isolated_bots, after #858's 0066 to 0068. A person's own browser is always a computer of its own, never a seat on the person's computer beside their Bots: the seat resolver says so, the browser routes locate it by its own key, and the computer refuses it by the id named rather than by the seat made from it. A Sandbox is cut by computer key: an isolated Bot's names its Bot and keeps its files at the root of the volume it had, and a person's names no Bot and gives each Bot a folder.
Upstream added `extension` (CopilotKit#858) to the desktop's package list; v0.2.0 has no `extension` either, so it joins `shared` and `toolkit` as a later package that is read only when the release includes it.
…ering A failed save rolls the cache back after cleanup. Unregistering happy-dom at once removes window under it, and React's read of window.event throws between tests, failing the run with every test passed. Same change as main's #858.
Takes main's engine choice (#831): Docker Desktop is opened only when it has run before, and each installation stays on the engine it was set up on. The old copy of #831 on this branch is dropped, including after_engine_install, which duplicated main's answering_after_install and ignored the saved engine. Kept from this branch: path_with_docker, downloads chosen by the machine's architecture on Windows on ARM, and the per-user Docker Desktop location on Windows. Takes main's #823 and #821 (StartFresh replaces DatabaseReset; the local Intelligence screen offers it too), #860, and #858's Recorder step label. The web app keeps main's theme and skin picker (#856); this branch's CopilotKit reskin of the app and its two fonts are dropped. Its desktop CSS (room for the macOS window buttons, sidebar resize) stays.
…containers Main's OpenBot Browser (#858) runs agent-computer on the host with Bun, to open the person's own Chrome or Edge. The server and worker are containers now, so the browser is the only host process left: - Start prepares Bun and the packages the first time it has a browser to run, as the Intelligence sign-in helper does, after the server answers. It spawns the browser under the startup lock, records it so a later window can end it, and holds it. - The server container reaches it at host.docker.internal (OPENBOT_BROWSER_CONTAINER_URL, which docker-compose.yml prefers), since the computer listens on the host's loopback. - It is restarted with backoff when it dies. Giving up says why, with the tail of its log, and leaves OpenBot running. - Stop closes the held browser before the recorded processes are ended, so Chrome closes its windows and profile first.
…containers Main's OpenBot Browser (#858) runs agent-computer on the host with Bun, to open the person's own Chrome or Edge. The server and worker are containers now, so the browser is the only host process left: - Start prepares Bun and the packages the first time it has a browser to run, as the Intelligence sign-in helper does, after the server answers. It spawns the browser under the startup lock, records it so a later window can end it, and holds it. - The server container reaches it at host.docker.internal (OPENBOT_BROWSER_CONTAINER_URL, which docker-compose.yml prefers), since the computer listens on the host's loopback. - It is restarted with backoff when it dies. Giving up says why, with the tail of its log, and leaves OpenBot running. - Stop closes the held browser before the recorded processes are ended, so Chrome closes its windows and profile first.
Brings the OpenBot Recorder and the OpenBot Browser into OpenBot. Both were built and reviewed in a private repository as two stacked pull requests; this lands them here as one, on top of current main.
The Recorder. A Chrome extension a person uses to teach a Bot by doing: they press Record, do a task in their own browser while saying what they are doing, and the clicks, typing, pages, network calls and narration go straight into their own Learning space in Intelligence as a trajectory, where Learning turns it into a skill a Bot can use. Settings → Browsers connects a browser; each person gets their own Learning space and chooses how their skills are shared.
The OpenBot Browser. A button in OpenBot opens a real Chrome window the person uses as their own, with the Recorder already installed and connected, and the Browser page records a named task in it. It records the person, never the Bot. OpenBot Desktop opens and connects it on macOS, Windows and Linux.
Before upgrading. Two migrations add the Recorder's tables, after main's 0065:
0066_recorder_extension: six new tables (recorder_connect_codes,recorder_connections,recorder_trajectory_bindings,learning_person_spaces,learning_skill_sharing,learning_open_skills). Nothing existing changes.0067_recorder_activity: two nullable columns onrecorder_connections.The CopilotKit runtime moves to 1.79.1, with
COPILOTKIT_OPENAI_APIchoosing the API an OpenAI-compatible endpoint is called with; the CHANGELOG's Unreleased section has every item.Reviewing. Most of the size is the extension (its source, and recorded agent traffic it is tested against in
extension/tests/fixtures) and the two drizzle snapshots. The server's part is the Recorder and Browser routes, connect codes and the per-person Learning spaces; the app's part is Settings → Browsers and the Browser page.Checked before publishing. No keys or tokens in the added lines, no customer names or internal links in the code or the commit messages, every dependency public (including the patched
@copilotkit/learning@0.0.1), and no new CI secret. The extension's signing key is not in the repository.