Close a computer's browsers before its memory runs out, and a Bot's idle tabs

.env.example

@@ -359,6 +359,20 @@ COMPUTER_TOKEN=
 # COMPUTER_BROWSER_BACKEND=managed
 # COMPUTER_BROWSER_MODE=headed
 
+# When a computer closes a browser besides the cap (COMPUTER_MAX_BROWSERS) and the idle timeout
+# (COMPUTER_BROWSER_IDLE_MS). At or above this share of its memory limit, read as the kubelet reads it
+# (the working set, so the page cache does not count), it closes browsers nobody has used for a
+# minute, least recently used first, until it is ten points under. Memory is read before each launch
+# and once a minute, so a page that grows between two checks can still run it out of memory. A
+# container with no memory limit is measured against the machine's memory; a computer outside a
+# container closes nothing for memory. In the all-in-one image the share counts the database and the
+# API server too, on purpose: the out-of-memory kill counts them as well. 0 switches it off.
+# COMPUTER_MEMORY_HIGH_PERCENT=85
+# How long a Bot's background tab may sit untouched before it is closed, in ms: a tab the Bot left
+# behind, such as the page a sign-in popup opened from. The page the Bot is on, a person's own
+# browser, and any tab of a Bot whose wheel a person holds are never closed. 0 keeps them.
+# COMPUTER_TAB_IDLE_MS=600000
+
 # Web Bot Auth: every request a Bot's browser makes is signed as coming from this deployment
 # (Ed25519, HTTP Message Signatures), and the public key is published at
 # <OPENBOT_PUBLIC_URL>/.well-known/http-message-signatures-directory. Off unless set to on, because

.github/workflows/ci.yml

@@ -141,7 +141,7 @@ jobs:
           bun test tests/sign-in.test.ts tests/session-cookies.test.ts tests/follows-popup.test.ts
           tests/demonstration-live.test.ts tests/webrtc-proxy.test.ts tests/web-bot-auth-browser.test.ts
           tests/handoff-page-restore.test.ts tests/enter-text.test.ts tests/person-browser.test.ts tests/person-browser-computer.test.ts
-          tests/screencast-keys-browser.test.ts
+          tests/screencast-keys-browser.test.ts tests/browser-memory.test.ts
         working-directory: agent-computer
         env:
           OPENBOT_COMPUTER_BROWSER: '1'

CHANGELOG.md

@@ -8,7 +8,7 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.
 
 ## Unreleased
 
-**Before upgrading.** Fourteen things change for an existing deployment:
+**Before upgrading.** Fifteen things change for an existing deployment:
 - A hosted computer's browser is now headed by default. It uses a little more memory than headless
   did; set `COMPUTER_BROWSER_MODE=headless` (Helm: `computers.browserMode: headless`) to keep the old
   behaviour.
@@ -61,6 +61,55 @@ Newest first. `Unreleased` is what is on `main` and not yet tagged.
   to keep the old behaviour.
 - Interfaces a Bot drew before the upgrade (A2UI 0.9 surfaces) no longer draw when a conversation is
   reopened; the messages around them stay. New ones use A2UI 1.0 (below).
+- A computer now closes browsers nobody is using, least recently used first, when its memory
+  reaches 85% of its limit, and closes a Bot's background tabs after ten minutes untouched (below).
+  In the all-in-one image that share counts the database and the API server too.
+  `COMPUTER_MEMORY_HIGH_PERCENT=0` and `COMPUTER_TAB_IDLE_MS=0` (Helm: `computers.memoryHighPercent`
+  and `computers.tabIdleMs`) keep the old behaviour.
+
+### A computer closes browsers nobody is using when its memory runs high
+
+A computer runs one browser for each Bot that is browsing, and the only limits were a count and half
+an hour untouched. Neither looked at memory, so a few heavy pages could take a container past its
+memory limit well under the count, and the kernel then killed processes without regard to which Bot
+was busy. On Kubernetes it kills everything in the container at once, so every Bot's browser closed
+together and nothing could reach the computer until it came back. A computer now reads its own
+memory the way the kubelet does (the working set, which leaves out the file cache the kernel can
+drop), before it starts a browser and once a minute. At or above 85% of its limit it closes browsers
+one at a time, least recently used first, until it is under 75%, so the next launch does not cross
+the line again straight away. Launches that arrive together share one pass rather than closing a
+browser each. A Bot whose browser was closed starts again where it left off on its next step, still
+signed in, because its profile is on disk.
+
+Only a browser nobody is using is closed. Never closed for memory: the browser being started, the
+most recently used one, any browser used in the last minute (a Bot between two steps asks for
+nothing while its model works out the next), a person's own browser, cloud or native, whose Recorder
+session would end, and the browser of a Bot whose wheel a person holds or has been asked to take.
+When every browser is in use, none is closed.
+
+This is a check, not a guarantee. Memory is read before each launch and once a minute, so a page
+that grows between two checks can still run the container out of memory. A container with no memory
+limit is measured against the machine's memory, and a computer outside a container, such as OpenBot
+Desktop's, closes nothing for memory. In the all-in-one image the database and the API server run in
+the same container as the browsers, and the share counts them too. That is deliberate: the kernel's
+out-of-memory kill counts everything in the container and can take the database down, and closing a
+browser nobody is using first is how the database is protected. Each close is logged as
+`computer-browser-closed-for-memory` with the working set and the limit in bytes.
+`COMPUTER_MEMORY_HIGH_PERCENT` (Helm: `computers.memoryHighPercent`) sets the share where closing
+starts, and `0` switches it off.
+
+The tabs a Bot leaves behind are closed too. A Bot always works in its newest page, so an older tab,
+such as the page a sign-in popup was opened from or a link a site opened in a new tab, is one it
+cannot go back to, and it held memory for as long as the browser stayed up. A background tab that
+nothing has used, navigated or brought to the front for ten minutes is now closed. The page the Bot
+is on is never closed, and neither is any tab of a person's own browser or of a Bot whose wheel a
+person holds. `COMPUTER_TAB_IDLE_MS` (Helm: `computers.tabIdleMs`) sets the time, and `0` keeps them.
+
+Through Helm, `computers.browserIdleMs` and `computers.maxBrowsers` now reach the computer as
+written. An hour (`browserIdleMs: 3600000`) used to render as `3.6e+06`, which the computer ignored
+for its default of thirty minutes, and `browserIdleMs: 0` (keep browsers open) was dropped. All four
+of these settings now also reach the computers `computers.mode: sandbox` makes, which had none of
+them.
 
 ### Generated interfaces use A2UI 1.0
 

agent-computer/src/browser-eviction.ts

@@ -1,5 +1,5 @@
 /**
- * Which Bots' browsers to close, and when.
+ * Which Bots' browsers to close, and when, and which of a Bot's tabs.
  *
  * Its own file, with no Playwright import, for the reason `authorisation.ts` and `bot-id.ts` are:
  * `profiles.ts` imports Playwright at module scope, so anything living there needs a browser merely
@@ -62,21 +62,96 @@ export function chooseEvictions(
   const entries = [...running];
   if (entries.length <= max) return [];
   // A cap of zero keeps nothing, the newest included; that is the mistake reading the setting guards.
-  const newest =
-    max > 0
-      ? entries.reduce((latest, entry) =>
-          entry[1].usedAt >= latest[1].usedAt ? entry : latest,
-        )
-      : undefined;
+  const newest = max > 0 ? newestOf(entries) : undefined;
 
+  return inClosingOrder(entries, newest, goesLast)
+    .slice(0, entries.length - max)
+    .map(([botId]) => botId);
+}
+
+/** The most recently used, which is the one a launch has just added. Ties go to the later entry. */
+function newestOf(
+  entries: [string, Evictable][],
+): [string, Evictable] | undefined {
+  if (entries.length === 0) return undefined;
+  return entries.reduce((latest, entry) =>
+    entry[1].usedAt >= latest[1].usedAt ? entry : latest,
+  );
+}
+
+/**
+ * The order browsers are closed in when some have to go: every Bot's before a person's native window
+ * (`keptOpen`), and least recently used first within each. `spare` is left out, which is the newest.
+ */
+function inClosingOrder(
+  entries: [string, Evictable][],
+  spare: [string, Evictable] | undefined,
+  goesLast: (botId: string) => boolean,
+): [string, Evictable][] {
   return entries
-    .filter((entry) => entry !== newest)
+    .filter((entry) => entry !== spare)
     .sort(
       ([aId, a], [bId, b]) =>
         Number(goesLast(aId)) - Number(goesLast(bId)) || a.usedAt - b.usedAt,
+    );
+}
+
+/**
+ * How long after its last use a browser still counts as in use, and so is never closed for memory.
+ *
+ * A browser's use is recorded when something asks for its page: a Bot's step, a person's input, the
+ * live screen. Between two steps a Bot asks for nothing while its model works out the next one, which
+ * can take tens of seconds, and nothing here can tell that pause from a Bot that has finished.
+ * Closing the browser then closes it in the middle of a run: the page the Bot was on is gone, and its
+ * next step starts a cold browser, which is the very jump in memory the close was meant to head off.
+ * A minute covers a model's turn between steps. It also covers a browser that has just launched, so
+ * the next launch never closes the one before it, which is what would otherwise turn a full computer
+ * into a loop of each Bot's launch closing the last Bot's browser.
+ */
+export const RECENTLY_USED_MS = 60_000;
+
+/**
+ * Which to close next because the computer's memory runs high (`container-memory.ts`).
+ *
+ * One at a time, because the caller reads memory again after each close and stops as soon as it is
+ * enough: a count says in advance how many have to go, and bytes do not. Least recently used first,
+ * the cap's order, so the two never disagree about who goes first.
+ *
+ * Never a browser in use, which here means four things:
+ *   - The one being launched. The caller runs this before a launch and passes only the browsers
+ *     already running, so the Bot that asked is not among them.
+ *   - The most recently used of those, however long ago that was. So at least one browser is always
+ *     left, because one browser that is on its own too big for the computer is the limit's problem,
+ *     and closing it would only start it again on its next step.
+ *   - Any used within `recentlyUsedMs` ({@link RECENTLY_USED_MS}), which is a Bot in the middle of a
+ *     run or a browser that has just started.
+ *   - Any `spared` names, which is a Bot whose wheel a person holds or has been asked to take: closing
+ *     it closes it under somebody's hands, or under a page the Bot is waiting for somebody to work.
+ *
+ * And never a person's own browser, cloud or native. Their Recorder runs in it, and closing it ends
+ * what they were recording without a word to them. It still counts toward the cap, which closes it
+ * last; memory pressure is not a reason to end somebody's session.
+ */
+export function chooseForMemory(
+  running: Iterable<[string, Evictable]>,
+  now: number,
+  {
+    recentlyUsedMs = RECENTLY_USED_MS,
+    spared = () => false,
+  }: {
+    recentlyUsedMs?: number;
+    spared?: (botId: string) => boolean;
+  } = {},
+): string | undefined {
+  const entries = [...running];
+  const cutoff = now - recentlyUsedMs;
+  return inClosingOrder(entries, newestOf(entries), () => false)
+    .filter(
+      ([botId, entry]) =>
+        entry.usedAt <= cutoff && !isPersonBrowserId(botId) && !spared(botId),
     )
-    .slice(0, entries.length - max)
-    .map(([botId]) => botId);
+    .map(([botId]) => botId)
+    .at(0);
 }
 
 /**
@@ -104,3 +179,40 @@ export function chooseIdle(
     .filter(([botId, entry]) => entry.usedAt <= cutoff && !stays(botId))
     .map(([botId]) => botId);
 }
+
+/** Enough of a tab for the choice. Keeps this file independent of what else is on a page. */
+export type IdleTab = {
+  /** When the Bot or a person last acted on it, it navigated, or it was the page in front. */
+  usedAt: number;
+  /** The page the Bot is on, which is the one the live screen shows and its next action lands in. */
+  front: boolean;
+  /** A page that is never the Bot's to lose: an extension's own (the Recorder's) or DevTools. */
+  kept?: boolean;
+};
+
+/**
+ * Which of a Bot's background tabs to close because nothing has touched them.
+ *
+ * A Bot is always on its newest page (`chooseLivePage`): a window a site opens is where its work
+ * went, and it has no way back to an older tab except that newer ones close. So a tab left behind
+ * (the page a sign-in popup was opened from, a link that opened in a new tab and was never closed)
+ * holds a renderer and its memory for nothing, for as long as the browser stays up. Closed past a
+ * timeout rather than at once, because an opener whose popup is still mid sign-in is waiting for
+ * that popup to come back to it.
+ *
+ * Never the page in front, which is the one the Bot is on, so closing tabs never moves it: it stays
+ * the newest open page, which is what the Bot follows. And never anything when which page is in
+ * front is not exactly one, because then any tab could be the one the Bot is on. That also means the
+ * last tab is never closed, since a single tab is the one in front. A timeout of zero or less
+ * switches this off.
+ */
+export function chooseIdleTabs<T extends IdleTab>(
+  tabs: readonly T[],
+  idleTimeoutMs: number,
+  now: number,
+): T[] {
+  if (idleTimeoutMs <= 0) return [];
+  if (tabs.filter((tab) => tab.front).length !== 1) return [];
+  const cutoff = now - idleTimeoutMs;
+  return tabs.filter((tab) => !tab.front && !tab.kept && tab.usedAt <= cutoff);
+}