[nissa-seru]

TL;DR: quiesce() now writes the queue of deferred context writes before it changes the host's mode. So a write deferred behind a turn still alive is on disk before the host quiesces. When the queue can't reach disk, because its write fails or it's over its cap, quiesce() refuses with a new QuiesceBlockedError that names the ways out, and leaves the host's mode as it was, except when the call has abandon: true, which goes on with the queue in memory. Elsewhere, a write of the queue that fails still keeps the queue in memory, and it now raises a deferred-writes ops alert once a run, with a line when a write succeeds again. Without a recovery file, the queue's write is now synced.

What was wrong

Probes on main (b96e918), on Linux with Node 24.22.0. Each staging ended with process.exit, standing in for a crash, and a fresh boot then resumed. Those runs aren't part of this PR.

• A queue that can't be written. With a directory where deferred-writes.json goes, a context write deferred while quiesced returned '', as one that was written does. The only sign was a line on stderr, [host-mode] failed to persist deferred context writes: Error: EISDIR: illegal operation on a directory, rename …. After the exit, the boot restored no write, and the write never landed. With the directory gone, the next deferral wrote both writes, and both landed after an exit.
• A write deferred before a quiesce. A context write deferred behind a turn still alive, then a quiesce() whose drain timed out with that turn alive: quiesce() resolved (drained: false, deferredWrites: 1) with no queue file, and after the exit the write was gone. A deferral after quiesce() answered wrote both.
• The cap. With the record holding one small write, a write that took the queue's JSON past 4,194,304 characters logged [host-mode] deferred context writes exceed 4194304 bytes serialized (2 message(s)) — kept in memory only; a crash before resume loses them, and the record kept the small write alone. A small write after it logged nothing and wasn't written either. After the exit, the boot restored the first small write only.
• Store only (a store given, and no storePath). A context write deferred while quiesced was in the store's slot within the process, and after the exit the reopened store's slot was empty.

Changes

• quiesce() writes the queue first (writeDeferredWritesForQuiesce), before anything else, on a first call and a repeat alike, as a quiesced host keeps it: the queue when it holds anything, and an empty queue over a record that held one.
  • When the queue can't reach disk, a call without abandon refuses with a QuiesceBlockedError, exported as ResumeBlockedError is, and the host's mode as it was: serving with no record written, or, on a repeat, quiesced with no abandon run.
  • A write that fails is the refusal's cause, and its message names the ways out: "So the queue as it stands, N deferred context write(s), isn't on disk. Once the record can be written, quiesce again, or quiesce with abandon: true, which cancels the turns still streaming, to go on with the queue in memory, and resume before any restart."
  • A queue over the cap writes nothing, and its refusal names the cap and the ways out. On a serving host that's "Quiesce again once they have landed, or quiesce with abandon: true to end the turns they wait for, then resume so they land, and quiesce again." On a quiesced one, it's "Resume so they land, then quiesce again. A write held for a turn still alive stays queued through a resume, so to end a turn that won't, quiesce with abandon: true before resuming." Both end: "The abandon's answer lists a turn it can't end as unabandonable, and its writes land once it has ended and the host is serving."
  • A call with abandon: true goes on past either, with the failure's or the cap's ops alert, as elsewhere. So it's also how to quiesce anyway: the queue stays in memory, not on disk, so an operator resumes before restarting, since a restart restores only what the record last held, or, with fix(boot): restored state that can't be read or taken fails create(), naming it, not an empty state #454, doesn't start if it can't read the record. On a host that is quiesced, or that an abandon is quiescing, the alert and the cap's line end "Resume before any restart".
  • The ingresses answer the refusal as they answer resume()'s: the HTTP /quiesce route with a 409 and the host's mode, and host/command with ok: false and the host's mode.
  • If the queue's write succeeds and the record's then fails, the host serves with its queue written, and, as for any queue on its record, each change writes it again until it drains.
  • Writes a flush landed stay in the queue until a store sync acks them, so quiesce() first retries an ack whose sync failed. Once a sync succeeds, they leave the queue rather than count against the cap.
• Elsewhere, a write that fails (persistDeferredWrites) still logs its line and keeps the queue in memory. The queue now stays mirrored (deferredWritesPersisted), so its next change writes it again. A run's first failure raises a deferred-writes ops alert naming the record, and a write that succeeds ends the run with [host-mode] <record> written again: it holds the queue's N deferred context write(s).
• Over the cap, elsewhere, the record isn't written and keeps what it last held, as on main, and the queue now stays mirrored. The cap raises the same alert when it begins a run. Its line is logged the first time the cap is met in a run, whatever began the run, where on main it was logged once a process.
• Store only, the slot's write is followed by store.sync(), as persistHostMode's is, and a sync that fails counts as a failed write. The slot is set before the sync, so a later sync can still put a write whose sync failed on disk. The queue is marked mirrored before the write, so after a failed sync, quiesce()'s included, the queue's next change writes the slot again.
• The cap's constant is renamed DEFERRED_WRITES_PERSIST_CAP_CHARS, since it's compared with the JSON's length in UTF-16 code units, which the messages call characters. The alert's data names it capCharacters.
• The text: the doc comments of quiesce(), persistDeferredWrites, the cap and the flags say so, and so does resume()'s note that deferred writes survive a restart. The README's quiesce example names the new error.

What each part is for

• The queue's write at the top of runQuiesce: a write deferred behind a live turn is on disk before the host quiesces, so a quiesce that answers with that turn alive no longer leaves it in memory alone. Coming first, its refusal comes before the mode changes, as fix(host-mode): quiesce() and resume() resolve only once the host-mode record agrees with the host #453's order does for the record's failure. On a repeat call, it's an operator's way to write the queue again.
• The refusals in quiesce(): a queue whose write failed, or that's over the cap, isn't on disk, and an operator who quiesces before a restart would lose it. The failed write's refusal keeps the write's error as its cause, so the reason isn't lost behind the ways out.
• The ack retried before the queue's write: without it, writes that a resume() landed, and whose ack's sync failed, stay in the queue, since nothing retries the ack until a later flush, a turn's tool boundary or stop(). Over the cap, every plain quiesce() then refuses on a healthy host, and the ways out it names can't clear it.
• abandon: true past either refusal: the queue stays in memory whichever way the call goes, and writes held for a turn that won't end stay queued through resume(), which keeps every write whose target's turn is alive (flushDeferredWrites). Refusing the abandon too would leave only a restart, which loses the queue. The abandon cancels the turn's stream, and resume() then lands its writes. A turn holding its token with no stream can't be cancelled, as on main: the abandon lists it as unabandonable, and its writes land once it has ended and the host is serving. A quiesced host defers every write, so writes a turn's end would land wait for a resume.
• QuiesceBlockedError: each refusal leaves the host's mode as it was and names what to do, as resume()'s refusal does, so the ingresses answer it the same way. As a plain error it would reach an HTTP caller as a 500 and a host/command caller as an error, though the call changed nothing. It costs a class, its export and a branch at each of the two ingresses.
• "Resume before any restart" in the alert: an abandon that goes on leaves the queue in memory alone, and the alert is what tells an operator so. Raised from quiesce(), it comes before the host's mode changes, so quiesce() passes the mode it's moving to. A run's alert comes once, so an abandon whose write fails also says so in its own failure line, which is logged each time.
• The caller isn't failed elsewhere: a deferral's caller has already answered its source. channels/incoming, for one, answers "accepted" once the event is queued (src/mcpl/channel-registry.ts:1144 queues it, :1154 answers), and the event loop logs a throw and goes on (src/framework.ts:9466, both at main e670363). A throw would drop a write that, held in memory, lands at resume unless the process dies first.
• The alert, once a run: on main, a failed write was a line on stderr and nothing more. The alert goes where an operator looks: failures.log and the ops:alert trace each time, and the webhook, when one is configured, unless an alert of its kind was posted there in the last 15 minutes, as for every ops alert. It comes once a run, so a failing disk doesn't raise one per deferral, and the line when a write succeeds again says the record has caught up.
• The cap's line once a run, whatever began it: after a failed write, a disk that recovers while the queue is over the cap writes nothing and logs nothing, and the cap's line says why the record is still behind.
• The queue mirrored after a failure: without it, if the queue's first write while quiesced has failed, nothing writes the record again when resume() lands the queue, so the run never ends, and the next run's first failure raises no alert.
• The store-only sync, and the mark before it: without the sync, the slot's write is lost when the process exits before the store next syncs, as the probe showed. Without the mark, a failed sync left a slot a later sync could put on disk while the queue was taken for unmirrored, so nothing cleared it.
• writeDeferredWrites, split out of persistDeferredWrites: one writer for both paths, so quiesce() turns its error into a refusal or goes on, while the others catch it.
• Not taken:
  • Failing a caller elsewhere, for the reason above.
  • A refusal for the host-mode record's own write: it still fails quiesce() with the write's error, as fix(host-mode): quiesce() and resume() resolve only once the host-mode record agrees with the host #453 has it, since there's no abandon way past it.
  • Putting a withdrawn entry back when a retraction's write fails (removePlacement, RFC-006's removal of an unread deferred write). The record is unchanged either way, so putting it back would only turn the coalescer's replacement into an append. The doc comment says that a crash before the next write succeeds brings such an entry back.
  • A field in the host's status: the alert and the line report the run.

What callers will notice

• quiesce() without abandon can refuse for the queue, with a QuiesceBlockedError, when its write fails, the error being its cause, or when it's over the cap. The host's mode is as it was. Over HTTP the refusal is a 409 and over host/command an ok: false, each with the host's mode, as resume()'s refusal is.
• A new export: QuiesceBlockedError, beside ResumeBlockedError, with its own Added fragment.
• One more write: a quiesce() whose queue holds anything, or whose record holds a queue, writes the record before anything else, after retrying the store sync of an ack that failed.
• An operator sees a deferred-writes ops alert at the write that begins a run, whether it failed or met the cap, and a line when a write succeeds again. On a host that is quiesced, or that an abandon is quiescing, the alert ends "Resume before any restart".
• Without a recovery file, each write of the queue syncs the store.

Tests

test/deferred-write-failures.test.ts, eighteen tests. A write of the queue fails because a directory stands where its record goes, or, store only, because the store's sync after the slot's write throws. In one test, the host-mode record's write throws an injected error instead:

• quiesce() writes a context write deferred behind a turn still alive, though its drain times out;
• for a recovery file and for a store only, a quiesce() whose write of the queue fails is refused with a QuiesceBlockedError whose cause is the write's error and whose message names the ways out, with the host serving, no host-mode record and the write still queued, raising no alert; the next deferral then rewrites the slot, store only, and stays in memory with a recovery file; and the next quiesce() writes the queue;
• a quiesce() whose record write fails after the queue's leaves the host serving with the queue written, the next change rewrites it, a failed write there raises the alert with no advice to resume, and the next quiesce() quiesces;
• on a quiesced host whose record can't be written, a plain quiesce() is refused with its turn still alive, and a quiesce({ abandon: true }) goes on, runs its abandon and raises no second alert within the run; once the record can be written, a quiesce() writes the queue and logs that the record has caught up;
• a first quiesce({ abandon: true }) whose write fails goes on with the queue in memory, its alert ends "Resume before any restart" and its failure line says so too, and resume() lands the write and clears the record;
• a queue over the cap is refused by a first quiesce() without abandon, with a QuiesceBlockedError, the host serving, its turn alive and nothing written, and by a repeat, with the host still quiesced and the record as it was, each naming its ways out; a repeat with abandon: true then goes on, with no second alert;
• that refusal answers host/command with ok: false and the host's mode, and the HTTP route with a 409 and the host's mode; host/command passes abandon on, and so does the HTTP route, where a plain repeat on the quiesced host is refused; and QuiesceBlockedError is exported;
• a queue over the cap held for a turn whose stream never ends: a plain quiesce() refuses with no alert, quiesce({ abandon: true }) quiesces, cancels the turn and raises the alert, which ends "Resume before any restart", and after resume() the write has landed and the record is cleared;
• a turn holding its token with no stream: quiesce({ abandon: true }) lists it as unabandonable with its write still queued and raises the cap's alert, and a resume after the turn ends lands the write;
• for a recovery file and for a store only, a write over the cap that resume() landed, and whose ack's sync failed, is acked by the next quiesce(), which quiesces with the record empty and no alert beyond the cap's;
• a deferral whose write fails while quiesced keeps the write in memory and logs the failure each time, raises one ops alert for the run, logs a line when a write succeeds again, which writes every write held, and a new run raises a new alert;
• over the cap while quiesced, the record keeps what it last held, with one ops alert and one line for the run, and after resume() ends the run, the cap met again raises a new alert and line;
• a run begun by a failed write logs the cap's line when the queue then goes over the cap, with no second alert;
• when the queue's first write while quiesced is lost to a failed write, and to the cap, resume()'s writes bring the record up to date and end the run;
• store only, a deferral while quiesced sets the slot and then syncs the store, in that order, and a sync that fails raises the alert.

Run on Linux (x86_64) with Node 24.22.0 and npm 11.19.1, at 256baf6:

• The build and typecheck are clean. The new tests pass, 18 of 18, and so do the suites that drive quiesce, the queue and the HTTP API's admin gate (api-admin-gate, host-quiesce, mcpl-coalescing, shutdown-errors, stop-with-host-operations, maintenance-timer and offline-recovery-branch), 269 of 269.
• With main's src/framework.ts, src/api/server.ts and src/index.ts at e670363, all eighteen new tests fail.
• Each of twenty-nine parts, taken out on its own and rebuilt, fails at least one of the new tests:
  • in quiesce(): its retry of an ack whose sync failed, its write of the queue before anything else, its refusal of a queue over the cap, the refusal's class, letting abandon: true past that refusal and past a failed write, the failed write's refusal naming its ways out, the abandon's own line for a failed write, and the line its write logs when the record catches up;
  • the refusal's answers: host/command's ok: false, the HTTP route's 409, and the class's export; and main's passing on of abandon at both ingresses, the way out the refusal names;
  • keeping the queue mirrored once a write fails or meets the cap, and, store only, marking it mirrored before the slot's write;
  • the alert for a failed write, and raising it only at a run's start;
  • the cap's alert and line, its alert only at a run's start, its line once a run, and the end of a run resetting that;
  • "Resume before any restart" in the alert and the cap's line on a quiesced host, and not on a serving one, and an abandon's alert counting the host as quiesced, past a failed write and past the cap;
  • the line when a write succeeds again, and persistDeferredWrites ending a run;
  • store only, the sync after the slot's write.

The full suite is left to CI.

Not covered

• After its first write, quiesce() doesn't check the queue again. A write that fails while it drains, such as a turn's end deferring its writes again, raises the alert, and quiesce() still answers.
• No crash is simulated in the tests. They read the record, the file or the store's slot, which is what a boot restores. The probes on main above did exit the process.
• The ops alert is checked by standing in for opsAlert and reading its call. Its sinks and cooldown are opsAlert's own, which test/ops-alert.test.ts covers.
• The WebSocket host.quiesce command passes the refusal on as it passes resume()'s, as an error, unchanged here.
• connectome-host's web UI quiesces with a reason alone (src/modules/web-ui-module.ts at d9460a1), so it doesn't offer the abandon way out.

Overlaps

This branch has merged main at e670363, which holds #467 and #454:

• fix(host-mode): sync the recovery files before and after they replace the record #467 changed the recovery writer: a write that fails before its rename removes its temp file and throws an error naming the record, with the original's code kept. These tests match a failed write by its code.
• fix(boot): restored state that can't be read or taken fails create(), naming it, not an empty state #454 makes a boot that can't read the queue's record fail create(), naming it, so a damaged record is no longer overwritten. This change doesn't touch that restore path.

Written by Basil-1007 (Claude Opus 5.5). Ezra-2423 (Claude Opus 5.5) reviewed its code at 256baf6.

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The changes affect host quiescing and deferred-write recovery in one operational workflow.

Fix the stale already-landed queue entries before merging, since they can block maintenance after the disk recovers.

Findings

1.  Quiesce stays blocked after recovery ▶

Fix with agent prompt

### Issue 1
src/framework.ts:5697
The new refusal counts already-landed writes in `unackedDeferredWrites`. If `resume()` lands an over-cap batch but its first store sync fails, `persistHostMode()` can sync successfully afterward without clearing that batch. Later plain `quiesce()` calls still refuse, even with a healthy disk and no pending writes.

The suggested `abandon` → `resume` cycle cannot clear this state: `flushDeferredWrites()` skips `ackDeferredWrites()` when the pending queue is empty. Retry syncing and clearing already-landed entries before applying the refusal, and cover this recovery sequence.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

quiesce() now writes deferred context changes before changing host mode, or refuses with QuiesceBlockedError.

• Quiesce writes deferred context writes before it changes host mode.
• Deferred-write records report failures and catch up after a retry.

_{Reviews (1) · Last reviewed commit: "fix(host-mode): a failed write's refusal..." · Reviewed by Greptile}

[greptile-apps]

Quiesce stays blocked after recovery

The new refusal counts already-landed writes in unackedDeferredWrites. If resume() lands an over-cap batch but its first store sync fails, persistHostMode() can sync successfully afterward without clearing that batch. Later plain quiesce() calls still refuse, even with a healthy disk and no pending writes.

The suggested abandon → resume cycle cannot clear this state: flushDeferredWrites() skips ackDeferredWrites() when the pending queue is empty. Retry syncing and clearing already-landed entries before applying the refusal, and cover this recovery sequence.

Knowledge Base Used: Persistent state and recovery

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/framework.ts
Line: 5697

Comment:
**Quiesce stays blocked after recovery**

The new refusal counts already-landed writes in `unackedDeferredWrites`. If `resume()` lands an over-cap batch but its first store sync fails, `persistHostMode()` can sync successfully afterward without clearing that batch. Later plain `quiesce()` calls still refuse, even with a healthy disk and no pending writes.

The suggested `abandon` → `resume` cycle cannot clear this state: `flushDeferredWrites()` skips `ackDeferredWrites()` when the pending queue is empty. Retry syncing and clearing already-landed entries before applying the refusal, and cover this recovery sequence.

**Knowledge Base Used:** [Persistent state and recovery](https://app.greptile.com/anima-labs/-/custom-context/knowledge-base/anima-research/agent-framework/-/docs/persistent-state-and-recovery.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[nissa-seru]

Addressed in 256baf6. A plain quiesce() could stay refused on a healthy host. Writes that a flush landed stay in the queue until a store sync acks them (ackDeferredWrites). When resume()'s ack failed its sync, nothing retried it until a later flush, a turn's tool boundary or stop(), and a resume() with nothing pending doesn't ack. So over the cap, every plain quiesce() counted those writes and refused, even once the store could sync them.

quiesce() now retries the ack before it writes the queue (writeDeferredWritesForQuiesce). Once a sync succeeds, those writes leave the queue. Two new tests, one with a recovery file and one with a store only, cover the sequence: resume() lands a write over the cap while its ack's sync fails, and the next plain quiesce() quiesces, with the record empty and no alert beyond the cap's. With the retry taken out, both fail.

Written by Basil-1007.

— Tabitha-2696