Repository navigation
Close a computer's browsers before its memory runs out, and a Bot's idle tabs - #879
Conversation
…le background tabs A computer capped its browsers by count and closed them after half an hour untouched, and neither looked at memory. A few heavy pages could take a container past its limit, and on Kubernetes the OOM killer then kills everything in the container together, so every Bot's browser went at once. The computer now reads its own working set the way the kubelet does: memory.current less inactive_file on cgroup v2, the v1 files where only those exist, measured against the machine's memory when the container has no limit, and nothing at all outside a container. Before each launch and on each idle-sweep tick, at or above COMPUTER_MEMORY_HIGH_PERCENT (default 85) it closes browsers one at a time, least recently used first by the cap's own order, reading memory again after each. It never closes the browser being launched, the most recently used one, or the browser of a Bot whose wheel a person holds or has been asked to take. Each close logs computer-browser-closed-for-memory with the working set and the limit. The idle sweep also closes a Bot's background tabs that have gone untouched for COMPUTER_TAB_IDLE_MS (default ten minutes). A Bot is always on its newest page, so an older tab is one it cannot return to. The page the Bot is on, a person's own browser, extension and DevTools pages, and any tab of a Bot whose wheel a person holds are never closed. There is no per-browser tab count: memory is now watched directly, and a count would close tabs a site opened moments ago. Both settings are wired through the chart, compose, the all-in-one image and the docs. The chart renders a zero and a whole number from a values file as the computer reads them, which the new-values-key check now asserts on every target that runs the shared computer.
…inds of computer computers.browserIdleMs and computers.maxBrowsers were rendered with `with`, so `browserIdleMs: 0` (keep browsers open) was dropped, and a whole number from a values file arrived as a float: an hour rendered as `3.6e+06`, which the computer refused for its default. All four browser settings now go through the shared computer environment, written in digits and kept at zero, so the computers `computers.mode: sandbox` makes get them too. The values check covers browserIdleMs alongside the two new keys.
guidovizoso
left a comment
There was a problem hiding this comment.
The approach holds up:
- Memory is measured as the kubelet does: the working set without inactive file pages, for cgroup v2 and v1.
- No cgroup, or one it can't read, closes nothing.
- Browsers close gracefully, with cookies saved first.
- The idle-tab close leaves alone the Bot's current page, the last tab, and any browser a person holds.
- The Helm
wholeNumberfix is correct, and the sandbox computers get the settings too.
A few things to fix, mostly because #871 will put all of a person's Bots on one computer:
- Launching one Bot can close another Bot that is mid-run. Before a launch,
relieveMemorycounts the Bot asking as the newest (profiles.ts:~597), andchooseForMemoryspares only the newest (browser-eviction.ts:116). So any other browser can be closed, including one used a second ago.usedAtonly moves onprofiles.page(), so a Bot waiting on the model or running a long command looks idle. Its next action then fails with "Target closed", and it relaunches on a blank page. Your own end-to-end run closed 3 of 4 in one launch. The description anddocs/configuration.mdsay the most recently used browser is never closed. Please make that true on launch: spare the newest existing browser too, or anything used in the last N seconds, or anything mid-run. Otherwise correct the wording. - Nothing stops a close/reopen cycle.
relieveMemorystops as soon as memory reads under 85%, and there's no minimum time a browser stays open. Three busy Bots whose pages together sit above the line will close each other on every relaunch (profiles.ts:948), and with no OOM nobody sees it. Please close down to a lower target (say 75%) and/or skip browsers launched or used in the last minute. - Simultaneous launches close more browsers than needed.
relieveMemoryhas no lock, and a closed browser takes about 2s to free its memory (closeAndWait). So K launches in a burst, like a schedule firing several Bots, close K browsers where one may have been enough. Please run one relieve at a time behind a shared in-flight promise. - A person's own cloud browser can be closed for memory, mid-recording too.
heldByPersonis false for person browser ids, andkeptOpenonly covers native windows. It's usually near the end of the order while the person is on the Browser page, but it's still eligible, and closing it ends a Recorder session without warning. Please spare it while it's being viewed or recording, or always close it last. - The "instead of OOM-kill" claim is wider than the code. Memory is checked only every 60s (
IDLE_SWEEP_MS) and before a launch, so a running browser that grows (a heavy single-page app, a big PDF) can still hit the OOM between checks. Please say that in the description and docs, or react tomemory.events/memory.high. - In the all-in-one s6 image, the 85% includes Postgres and the API server. The container's working set covers every service in it (
docker/s6/s6-rc.d/computer/run), so browsers get closed to make room the database is using. Please document this, or default it off there.
Tests: there's nothing for concurrent launches, the close/reopen cycle, a Bot mid-action when its browser closes, or a person's cloud browser being closed.
…own to ten points under Review found four ways closing browsers for memory could hurt the work it was protecting. A launch counted only itself as in use, so a Bot that asked for its page a second ago and was waiting on its model could be closed in the middle of a run. The most recently used browser already running is now spared as well, and so is any browser used in the last minute (RECENTLY_USED_MS), which also keeps a launch from closing the browser the launch before it just started. Closing stopped at the line it started at, so the next launch crossed it again; a pass now starts at COMPUTER_MEMORY_HIGH_PERCENT and closes on to ten points under it. Every launch ran a pass of its own and read memory before any close had released it, so a burst of launches closed one browser each; one pass runs at a time and every caller waits on the one in flight. A person's own browser, cloud or native, is never closed for memory, because closing it ends their Recorder session without a word. The docs now call this a check before each launch and once a minute rather than a guarantee, and say that in the all-in-one image the share counts the database and the API server too, on purpose, because the kernel's out-of-memory kill counts them as well. createProfiles takes a clock, so the browser tests can make a browser a minute quiet without waiting a minute.
|
@guidovizoso thanks, all fair. Addressed in bec3d4e:
Tests for each, in real Chromium (
|
guidovizoso
left a comment
There was a problem hiding this comment.
Re-reviewed at bec3d4e1. Approving: all six are addressed.
- Closing a Bot mid-run.
chooseForMemorynow spares the newest running browser, anything used withinRECENTLY_USED_MS, and every person's browser, and the browser being launched is no longer a candidate. "The most recently used browser is never closed" now holds. - Close/reopen cycle. It starts at 85 and closes down to
memoryTargetPercent(75), reading memory again after each close finishes. - Concurrent launches.
relieveMemoryshares one in-flight pass, and launches wait on it. - A person's own browser is never chosen for memory.
- The docs now say it's a check before each launch and once a minute, not a guarantee against OOM.
- The s6 image: your reasoning holds. The OOM killer counts the database too, so closing a browser first is the better failure. And it's documented.
One note for later, not blocking: "recently used" still moves only on page actions. A Bot that has been waiting on the model or running a command for over a minute can still be closed under it. If that shows up in practice, marking a browser used while its Bot has a run in progress would close that gap.
Brings in #879: a Bot's browser now also closes when the computer's memory runs high and its idle background tabs close, sparing any browser whose wheel a person holds, so a recording is never cut off by either.
A computer runs one browser per Bot that is browsing, and the only limits were a count (
COMPUTER_MAX_BROWSERS, 8) and half an hour untouched. Neither looked at memory, so a few heavy pages could take a container past its limit well under the count, and the out-of-memory kill then took every Bot's browser down at once. The industry standard for an always-on agent's computer has exactly this failure today: a fixed machine per user, a few Bots browsing at once is the comfortable limit, and when memory runs out nothing can reach the computer. This closes browsers before that happens, and closes the tabs Bots leave behind.Memory
memory.currentminusinactive_file, againstmemory.maxon cgroup v2, with a v1 fallback. It reads before every browser launch and on every idle sweep.COMPUTER_MEMORY_HIGH_PERCENT(85 by default,0turns it off), it closes browsers one at a time, least recently used first, until memory is back under. A closed Bot starts again where it left off, still signed in, because its profile is on disk.Tabs
COMPUTER_TAB_IDLE_MS(10 minutes by default,0keeps them).Helm fix found on the way
computers.browserIdleMs: 3600000rendered as3.6e+06, which the computer ignored for its default, andbrowserIdleMs: 0was dropped.computers.mode: sandboxmakes, which had none of them.Tests (each fails before the change and passes after)
memory.statcapture.browser-memory.test.tsin real Chromium, added to CI's real-browser list.browserIdleMs.Locally: 97 real-browser tests, 523 computer unit tests, the chart check on every CI target, and format, lint and types all clean.
End to end with the computer image from this branch, in a container with a 1 GB memory limit: