Skip to content

fix(tune-out): a channel's cancel makes its dump durable before it records the end - #909

Merged
nissa-seru merged 4 commits into
anima-research:mainfrom
nissa-seru:hilda/tune-out-cancel-order
Oct 30, 2026
Merged

nissa-seru merged 4 commits into
anima-research:mainfrom
nissa-seru:hilda/tune-out-cancel-order

Conversation

@nissa-seru

@nissa-seru nissa-seru commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

A tuned-out channel's cancel now writes its backlog dump and makes it durable before it records the tune-out's end, so a store write that fails during the cancel no longer leaves the held messages with no dump and no tune-out on record to dump them. On main, the cancel records the end first. In a run at 4a45766f01 with the dump's write made to throw, no dump was written, and after a restart on the same store no tune-out was on record and nothing was dumped. With this change, a failure before the end record leaves the tune-out in force, and a later cancel, in the same process or after a restart, dumps the held messages that no earlier dump covered, each once. The all-except hold's release already wrote its dump before its end, and it now keeps the same record of what each dump covered. A cancel whose dump can't be made durable now throws, and "What changes for callers" below lists that and the other differences a caller meets. At this head's tree, all 1,052 tests in the 44 tune-out test files pass, and with main's src/tune-out/coordinator.ts 21 of them fail.

Problem

At 4a45766f01, TuneOutCoordinator.cancel() in src/tune-out/coordinator.ts calls this.channelRegistry.cancelTuneOut(serverId, channelId, nextState, source) at :621, and writes the dump with this.hooks.addMessage( at :729. After that call, at :734, the comment "The epoch is already ended durably; the release must be as durable." sits above this.hooks.commitDeliveries?.(); (:735).

A test run at a clean checkout of 4a45766f01, with one test file added, shows what that order does when the dump's write fails. It enters a tune-out on a ChannelRegistry over a chronicle JsStore, gives the coordinator three stored messages held under it, and cancels. It then restarts by building a second registry and coordinator over the same store and calling resumeActiveEpochs(). Its output:

CONTROL cancel threw=undefined dumped-before-restart=true on-record-after-restart=null desired=open dumped-after-restart=false
PROBE cancel threw="the store refused the write" dumped-before-restart=false on-record-after-restart=null desired=open dumped-after-restart=false

In the PROBE case, the dump's addMessage throws. The cancel threw, no dump was written, and after the restart no tune-out was on record and the restart wrote no dump. So after one failed write, nothing in that run dumped the three held messages. The test file is under "The run at main" below.

FableDendrite's review of #844 named this order: "One older ordering risk, also main's, is noted in the commit: cancel() appends the end record before the dump." (comment)

Changes

Line numbers in this section are this head's.

  • The channel's cancel reads the tune-out's state (src/tune-out/coordinator.ts:698), writes the dump (:802), calls commitDeliveries() (:809), and only then calls cancelTuneOut() (:812). cancelAllExcept() already wrote its dump (:1092 at 4a45766f01) before it cleared the hold (:1104 there). When commitDeliveries() returns false, the cancel throws "the backlog of … could not be made durable, so its tune-out stays in force" (:810) and records no end.

  • commitDeliveries() in src/framework.ts (:36727) now returns a boolean. Its store.sync() catch returns false, and otherwise it returns what persistDeferredWrites() (:9415) returned when it ran: false from its catch and for 'over-cap', true otherwise. At 4a45766f01 it is commitDeliveries(): void (:36737), and its store.sync() catch is console.error('[tune-out] could not sync the store after a release:', err); (:36745).

  • Each dump records what it covered, in a metadata field, released, a ReleaseRecord (:103 to :112): the ids and deferred write ids of the held messages it covered, those its cap left out among them, with total, truncated, and the keys of those it left for the subconscious's report. A cancel reads the epoch's earlier records from the stored and queued messages and leaves out the held messages they cover. It writes a dump only if (earlier.length === 0 || backlog.length > 0) (:800). Both cancels use this record. It replaces scopeReleased, whose comment at 4a45766f01 reads "The epoch whose release this process delivered and whose hold is not yet recorded as cleared." Tests in test/tune-out-cancel-order.test.ts:

    • "a cancel retried after its dump was written and its end record failed shows each held message once, and ends the tune-out";
    • "after a restart between a written dump and its failed end record, the next cancel shows nothing a second time";
    • "a dump row taken away before the tune-out ends, as a rollback, an undo of the turn or a surgery can, leaves its messages to the next cancel";
    • "a message queued at a dump whose end record failed, which lands before the retry, is not dumped again";
    • "the subconscious's report after a retry sums what the dumps left out, counting a dump stored in shards once and one that left nothing out". It asserts that the report after two dumps says "1 of 5 diverted messages across 2 dumps, the oldest in each dump that left any out, weren't delivered raw to the resident."

    test/all-except-unit.test.ts has the hold's counterparts, among them "after a restart between a written release and a failed clear, the next release shows nothing a second time and clears the hold" and "the release's record says how many held messages it covered and how many its dumps left out".

  • A wake past the wake budget whose cancel throws now has its cancel tried again a minute later (test/tune-out-timers.test.ts: "a wake past the budget whose cancel throws, on a quiet tune-out with no end time, is reported, and the cancel is tried again a minute later"). An end time's cancel already was, and that file's "an end time whose registry cancel throws is reported, and the cancel is tried again a minute later" passes with main's coordinator too.

  • At the start, resumeActiveEpochs():

    • keeps in force a tune-out or hold whose end time passed while the process was down and whose cancel fails, with its cadence running and its cancel tried again a minute later, and goes on to the others ("a start whose expired tune-out cannot be made durable still starts and arms the others, and its retry delivers once durability returns"; in all-except-unit, "a scope whose deadline passed while the process was down and whose release fails at the start stays in force, arms its retry, and the start goes on");
    • goes on when such a cancel recorded the end and a later step threw, and treats the tune-out or hold as ended ("a start whose expired tune-out ends but then fails a step after the end goes on, and treats it as ended"; in all-except-unit, "a scope whose deadline passed while the process was down and whose release ends it at the start but then fails a later step is treated as ended, with no retry, and the start goes on");
    • cancels a tune-out already past its wake budget ("a tune-out already past its wake budget at the start is cancelled there"). When that cancel fails, it says so as the start, keeps the tune-out in force, and its retry ends it ("a tune-out past its wake budget whose cancel fails at the start says so as the start, stays in force, and its retry ends it"). When the cancel ends it and a later step fails, it reports the tune-out as ended ("a tune-out past its wake budget whose cancel ends it at the start and then fails a later step is reported as ended").
  • A tool close of the all-except hold's excepted channel now reads the release's outcome. In the catch around cancelAllExcept(), src/framework.ts:31864 throws `The close wasn't recorded: ${…}` with the error's message, and :31866 throws the same words with the result's error when it isn't ok.

  • docs/dendrite.md's sentence on the all-except hold at a restart now reads: "A scope whose deadline passed while the process was down is released at resume; if that release fails before the scope's end is recorded, the scope stays in force and its release is tried again a minute later; one that fails after its end leaves the scope ended, with no retry."

  • Four test files' stand-in registries (tune-out-cancel-held, tune-out-dump-fit, tune-out-gate-composition and supplied-text-in-context) now answer getTuneOutState() for the channel being cancelled until their cancelTuneOut() ends it, since cancel() now reads the state first.

What changes for callers

  • A cancel whose dump can't be made durable throws, and the tune-out stays in force. With main's coordinator, the same cancel throws nothing ("a cancel whose dump could not be made durable leaves the tune-out in force, and its retry ends it showing each message once"). The cancel_tuneout tool and tune_out's cancel call cancel() at src/tune-out/coordinator.ts:2000 and src/framework.ts:36583.
  • A tune-out whose end time passed while the process was down, and whose dump can't be made durable at the start, stays in force past its end time, and its cancel is tried again a minute later ("a start whose expired tune-out cannot be made durable still starts and arms the others, and its retry delivers once durability returns"). With main's coordinator, that tune-out ended at the start without a durable dump: the test expected it still on record and found none.
  • At the start, an expired tune-out whose cancel ends it and then fails a later step no longer makes resumeActiveEpochs() throw. With main's coordinator, resumeActiveEpochs() threw "the wake request failed" in "a start whose expired tune-out ends but then fails a step after the end goes on, and treats it as ended".
  • A tool close of the excepted channel whose release fails now throws "The close wasn't recorded: …". On main (src/framework.ts:31871 at 4a45766f01), the close handler calls coordinator.cancelAllExcept('channel-close', 'the excepted channel was closed'); without keeping its result, and then returns "The focus on it ended all the same, and what the other channels held was delivered."
  • Stored dump rows carry released in their metadata. main's cancel() builds its backlog from every stored and queued message held under the epoch (src/tune-out/coordinator.ts:636 to :642 at 4a45766f01) and doesn't read released.

What each part is for

Line numbers are this head's, in src/tune-out/coordinator.ts unless another file is named.

  • The channel cancel's new order (:698 to :812) is the fix itself. It reads the state, writes the dump, makes it durable, and only then records the end, so a failure before the end record leaves the tune-out in force for a later cancel. The PROBE run under Problem shows what main's order does instead.
  • commitDeliveries() and persistDeferredWrites() returning whether the release is durable (src/framework.ts:36727, :9415) let both cancels record the end only once the dump is durable. At 4a45766f01 both return void. Each cancel's check is if (this.hooks.commitDeliveries?.() === false) (:809, :1184).
  • The released record and its helpers (HeldRow, RowKeys, ReleaseRecord, keysOf and keyedBy, :101 to :131) are how a later cancel knows what an earlier dump covered. The record sits in the stored dump row, not in memory, so what a later cancel reads can't disagree with what's stored after a restart, after a rollback that takes the dump away, or after a commit that failed once the dump was stored. The tests named under Changes cover the restart, the rollback and the retry.
    • Deferred write ids beside stored ids let a message that was still queued at the dump be matched once it lands, since the stored row keeps its deferredWriteId ("a message queued at a dump whose end record failed, which lands before the retry, is not dumped again").
    • total and truncated let a retry's report count what all of the epoch's dumps left out, not only the retry's own dump ("the subconscious's report after a retry sums what the dumps left out…").
    • report holds the keys of the messages a dump left for the subconscious's report, which a retry reads at :815 so that its report can name them as well. No test retries over such a dump (Not verified).
  • isReleaseRecord, isRowKeys and isStrings (:133 to :141) are the one check of the stored field's shape. releaseRecordsOf keeps only rows whose released passes it (:148).
  • releaseRecordsOf's comparison of whole records (:148) counts a dump the store keeps in shards once, since each shard carries the same record ("…counting a dump stored in shards once…").
  • truncationNote (:155) puts the report's count of what the dumps left out in one sentence, with the sum. The second commit's message gives the reason: each dump's own text already says what it left out, and a dump still queued behind the resident's turn is in no reader's view yet, so a per-dump number wouldn't let a reader find it. The test asserts "1 of 5 diverted messages across 2 dumps…".
  • The all-except release using the same record, in place of scopeReleased (:1141 to :1186), removes an in-memory mark that lived apart from the delivery and could disagree with what's stored ("after a restart between a written release and a failed clear, the next release shows nothing a second time and clears the hold").
  • truncatedInAll (:1158, :1165) makes the hold's stored record give the count of what its per-channel dumps left out, rather than 0 ("the release's record says how many held messages it covered and how many its dumps left out").
  • cancelPastBudget (:1420) sets the cancel's retry timer before it calls cancel(), so a tune-out past its wake budget is tried again when its cancel throws ("a wake past the budget whose cancel throws, on a quiet tune-out with no end time, is reported, and the cancel is tried again a minute later"). Its from: 'start' is for a start, where no timer fired, so the start doesn't report a timer's failure ("a tune-out past its wake budget whose cancel fails at the start says so as the start, stays in force, and its retry ends it").
  • resumeActiveEpochs()'s handling of a failed cancel (:437 to :484) keeps one failing epoch from ending the framework's start ("a start whose expired tune-out cannot be made durable still starts and arms the others…"). It reads from the registry whether each epoch is still in force, so an epoch whose cancel ended it and then failed a later step is treated as ended ("a start whose expired tune-out ends but then fails a step after the end goes on, and treats it as ended"; in all-except-unit, "…is treated as ended, with no retry, and the start goes on"). A tune-out already past its wake budget is cancelled at the start, since, as the first commit's message says, its retry is a timer a restart loses ("a tune-out already past its wake budget at the start is cancelled there").
  • The excepted channel's close reading the release's outcome (src/framework.ts:31864, :31866) is so the close doesn't answer "what the other channels held was delivered" when the release failed or was refused. No test closes it in that state (Not verified).
  • clearTimers() and the reset of the diverted count now run after cancelTuneOut() (:812 to :814). The first commit's message gives the reason: "so a cancel that fails leaves them running".
  • The removed doc comments of cancel(), cancelAllExcept() and both commitDeliveries described internals, and the first commit's message carries what they said.
  • EXPIRY_RETRY_MS's doc comment (:211) now names the wake budget's retry, which uses the same interval.
  • docs/dendrite.md's sentence says what a restart does with a hold whose release fails, and the changelog fragment says what a caller now meets.
  • The tests: test/tune-out-cancel-order.test.ts, the new tests in test/all-except-unit.test.ts and the wake-budget test in test/tune-out-timers.test.ts each pin a part above. all-except-unit's harness now stores the release rows it's given and answers commitDeliveries, as the framework does, and its wake can fail. The four stand-in registries answer getTuneOutState() because cancel() now reads it first.

With #844

@tel-0s, #844 (dda847ba26) changes src/tune-out/coordinator.ts too. On October 29 at 17:28 UTC, git merge-tree of #844 with main at 84e0408977 conflicts in src/framework.ts and five test files. With main and this change merged, #844 also conflicts in src/tune-out/coordinator.ts, and its conflict in src/framework.ts is a different one. Whichever of the two merges second has to take the other in.

With #201

@antra-tess, #201 (484b99095d) conflicts with main at 84e0408977 in src/framework.ts, src/gate/event-gate.ts, src/mcpl/channel-registry.ts, src/tune-out/coordinator.ts and test/tune-out-gate-composition.test.ts. With main and this change merged, its conflicts in src/tune-out/coordinator.ts and test/tune-out-gate-composition.test.ts are different ones.

Of the other 45 open pull requests at 17:28 UTC, #256 and #257 have different conflicts in src/framework.ts and src/tune-out/coordinator.ts with main and this change merged than with main alone. The other 43 merge with main and this change as they do with main alone.

Tests

At a clean checkout of 1d5f6ba80d, a commit that isn't in this pull request's history and has this head's tree, cb19ab6fce, on Linux with Node 24.22.1 and npm 11.19.1, after npm install with package-lock.json and node_modules removed, and npm run build. npm ls --depth=0 gives @animalabs/chronicle 0.5.0, @animalabs/context-manager 0.24.1, @animalabs/mcpl-core 0.4.0, @animalabs/membrane 0.8.0, @babel/parser 7.29.12, @modelcontextprotocol/client 2.4.0, @modelcontextprotocol/server 2.4.0, @types/node 22.20.5, @types/ws 8.18.2, chokidar 4.0.3, discord.js 14.27.0, tsx 4.23.15, typescript 7.0.2 and ws 8.22.0.

  • The 44 test files whose source names TuneOutCoordinator, tune_out, tuneOut, cancelTuneOut or all-except: 1,052 of 1,052 pass.
  • The same files with main's src/tune-out/coordinator.ts (the same file at 0f11975e48, this branch's base, and at 4a45766f01): 1,031 pass and 21 fail. They are all 14 tests in test/tune-out-cancel-order.test.ts, the wake-budget test in test/tune-out-timers.test.ts named above, and six in test/all-except-unit.test.ts: the four named above, "a message held while the clear keeps failing reaches the resident once, and every message the first release showed is shown once", and "a release that could not be made durable leaves the hold in force, and its retry clears it with the release written once".
  • With 0f11975e48's src/framework.ts as well: the same 21 fail.
  • The whole suite, in a copy of 1d5f6ba80d that also held untracked files of the run's own (its scripts, a file naming the head, and copies of 0f11975e48's src/tune-out/coordinator.ts and src/framework.ts) and a package-lock.json, which the repository doesn't commit: npm ci, npx tsc --noEmit and npm run build each exited 0. With each of the 382 test files run in its own process, eight at a time, under Node 24.22.1 with --test-force-exit, 9,232 tests ran: 9,169 pass, 62 skipped and 1 fail. The failure was test/reply-server-routes.test.ts's "a script started in one turn and waited on in a later turn from the other server takes the later turn's channel for none of its names", which ended after 17.5 s with "timed out waiting for: the second turn started", from the test file's own 15 s wait. This change doesn't change that file. Run alone at the same head, three more times, the file passed 56 of 56 each time. That run's npm ls gives @animalabs/chronicle 0.5.0, @animalabs/context-manager 0.24.1 and @animalabs/membrane 0.8.0.

The run at main

At a clean checkout of 4a45766f01, with the same platform and versions as above, the file below in test/ as cancel-order-probe.test.ts, after npm run build, node --test dist/test/cancel-order-probe.test.js passes CONTROL and fails PROBE, printing the two lines quoted in Problem.

test/cancel-order-probe.test.ts
/**
 * A channel tune-out's cancel() with the dump's addMessage made to throw (the
 * PROBE case) and without (the CONTROL case), then a restart: a second
 * ChannelRegistry and TuneOutCoordinator over the same JsStore, and
 * resumeActiveEpochs(). It prints whether the cancel threw, whether a dump was
 * written before the restart, which tune-out is on record after it, and
 * whether the restart wrote a dump.
 */
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { JsStore } from '@animalabs/chronicle';
import type { ContentBlock } from '@animalabs/membrane';
import { ChannelRegistry, type TuneOutParams } from '../src/mcpl/channel-registry.js';
import type { McplServerRegistry } from '../src/mcpl/server-registry.js';
import type { FeatureSetManager } from '../src/mcpl/feature-set-manager.js';
import { TuneOutCoordinator } from '../src/tune-out/coordinator.js';

function makeRegistry(store: JsStore) {
  const serverRegistry = { getServer: (_id: string) => null } as unknown as McplServerRegistry;
  return new ChannelRegistry(serverRegistry, {} as FeatureSetManager, () => {}, () => {}, { store });
}

const PARAMS: TuneOutParams = { epochId: 'eA', cadenceSeconds: 1800, backlogCap: 200, maxWakes: 5, startedAtSequence: 0 };
const text = (t: string): ContentBlock => ({ type: 'text', text: t } as ContentBlock);
const HELD = [1, 2, 3].map((i) => ({ participant: 'user', content: [text(`held ${i}`)], metadata: { author: { name: 'Bo' }, tuneOut: { epochId: 'eA' } } }));

function hooksFor(added: Array<{ kind?: unknown }>, failDump: boolean) {
  return {
    addMessage: (_participant: string, _content: ContentBlock[], metadata?: Record<string, unknown>) => {
      if (failDump && metadata?.kind === 'tune-out-cancel') throw new Error('the store refused the write');
      added.push({ kind: metadata?.kind });
    },
    requestInference: () => {},
    subconsciousName: () => 'Subconscious',
    primaryName: () => 'scout',
    getStoredMessages: () => HELD,
    getDeferredMessages: () => [],
    emitTrace: () => {},
  };
}

for (const failDump of [false, true]) {
  test(`${failDump ? 'PROBE' : 'CONTROL'}: a cancel whose dump write ${failDump ? 'throws' : 'succeeds'}, then a restart`, () => {
    const dir = mkdtempSync(join(tmpdir(), 'cancel-order-'));
    try {
      const store = JsStore.openOrCreate({ path: join(dir, 'store') });
      const first = makeRegistry(store);
      first.enterTuneOut('disc', 'disc:guild:a', PARAMS, 'agent-tool');
      const added1: Array<{ kind?: unknown }> = [];
      const c1 = new TuneOutCoordinator(first, {} as never, hooksFor(added1, failDump) as never);
      let threw: string | undefined;
      try { c1.cancel('disc', 'disc:guild:a', 'agent-tool', 'test'); } catch (e) { threw = (e as Error).message; }
      c1.stop();
      const dumpedBefore = added1.some((m) => m.kind === 'tune-out-cancel');

      // Restart: a fresh registry and coordinator over the same store.
      const second = makeRegistry(store);
      const added2: Array<{ kind?: unknown }> = [];
      const c2 = new TuneOutCoordinator(second, {} as never, hooksFor(added2, false) as never);
      c2.resumeActiveEpochs();
      const onRecord = second.getTuneOutState('disc', 'disc:guild:a');
      const dumpedAfter = added2.some((m) => m.kind === 'tune-out-cancel');
      c2.stop();
      console.log(`${failDump ? 'PROBE' : 'CONTROL'} cancel threw=${JSON.stringify(threw)} dumped-before-restart=${dumpedBefore} on-record-after-restart=${JSON.stringify(onRecord?.params.epochId ?? null)} desired=${second.getDesiredState('disc', 'disc:guild:a')} dumped-after-restart=${dumpedAfter}`);

      // Passes when a dump was written, or the tune-out is still on record for a later cancel.
      assert.ok(dumpedBefore || onRecord !== null || dumpedAfter, 'no dump before the restart, no tune-out on record after it, and no dump at the restart');
    } finally {
      rmSync(dir, { recursive: true, force: true });
    }
  });
}

Not verified

  • Platforms other than Linux, and this repository's CI, which hasn't run this change.
  • A real process death. Each restart in these tests is a second registry and coordinator over the same open store, not a new process.
  • commitDeliveries() returning false from a real failure. Every test that makes a dump non-durable does it through a stand-in commitDeliveries that returns false. No test makes the deferred queue's write or store.sync() fail under a cancel.
  • The tools' answers. No test calls cancel_tuneout or tune_out's cancel while the dump can't be made durable, so what either tool answers then is unshown.
  • The excepted channel's close. No test closes it while the hold's release throws or returns { ok: false }.
  • An end time reached while the process runs, with a dump that can't be made durable. The tests make that happen at the start only.
  • An earlier version reading these dumps. No run cancelled a tune-out with main's code over a store this change had written.
  • What this leaves. In A step that throws after a tune-out cancel records its end leaves the rest of the cancel undone #908's probe, a cancel records the end and then one later step throws: the subconscious's note, the resident's wake or the report to the subconscious. At this head's tree as at 4a45766f01, with a resident subconscious as the reader, that step and the ones after it hadn't come after ten minutes of mock time, and a second cancel found nothing to cancel.
  • The subconscious's report after a retry over a dump that left messages for it. No test retries a cancel whose earlier dump left messages for the subconscious's report, so whether the retry's report names them is unshown.
  • What this leaves, at an upgrade. A release row that 0.30.0 wrote for the all-except hold carries no released. At 4a45766f01, whose src/tune-out/coordinator.ts is 0.30.0's, cancelAllExcept() writes its release before this.writeScope(null); (:1104), under the comment "a restart between the two delivers the release again" (:1102 to :1103). In a probe on test/all-except-unit.test.ts's stand-in registry and store, a hold stayed on record over three held messages, a release row in 0.30.0's shape showing them, and one message held after it. After a restart, this head's cancelAllExcept() wrote one release that showed all four, the first three a second time. With 0f11975e48's src/tune-out/coordinator.ts, the same file as v0.30.0's, the probe printed the same line. For a channel's tune-out, cancel() at 4a45766f01 calls cancelTuneOut( (:621) before the dump's addMessage( (:729). The probe and its output are below.
The upgrade probe, test/zz-probe-old-release-row.test.ts

Put it in test/, run npm run build, then node --test dist/test/zz-probe-old-release-row.test.js. We ran it at clean checkouts of this head and of 0f11975e48, on Linux with Node 24.22.1 and npm 11.19.1, after npm install with package-lock.json and node_modules removed, with the packages Tests lists. Each run's one test passed and printed:

PROBE-RESULT {"epochIdSet":true,"result":{"ok":true},"holdAfter":"ended","releasesWritten":1,"earlierShownAgainOf3":3,"laterShown":true,"summaryLine":"4 messages held in 1 channel."}
/**
 * PROBE, uncommitted: an all-except hold still on record over a release row in
 * 0.30.0's shape (no `released`), met by this code after a restart. Prints how
 * many of the held messages that row already showed the new release shows
 * again, and whether it shows one held after it. Run at e6908ed473 and at
 * 0f11975e48 alike.
 */
import { describe, it } from 'node:test';
import assert from 'node:assert/strict';
import { TuneOutCoordinator, type TuneOutFrameworkHooks } from '../src/tune-out/coordinator.js';
import type { ChannelRegistry } from '../src/mcpl/channel-registry.js';
import type { McplServerRegistry } from '../src/mcpl/server-registry.js';

function harness(stored: Array<{ participant: string; content: Array<{ type: string; text?: string }>; metadata: Record<string, unknown> }> = []) {
  const state = new Map<string, unknown>();
  const wakes: string[] = [];
  const messages: Array<{ participant: string; text: string; forAgent?: string }> = [];
  const traces: string[] = [];
  const fail = { read: false, write: false, addMessage: false, commit: false, wake: false };
  const anchors: number[] = [];
  const registry = {
    getTuneOutState: () => null,
    // Each channel is shown by its id where no other server registers it,
    // and with its server where two do (general, on disc and slack).
    nameShown: (ref: { serverId: string; channelId: string }) => (ref.channelId === 'general' ? `${ref.channelId} (${ref.serverId})` : ref.channelId),
    listChannelsRaw: () => [
      { serverId: 'disc', descriptor: { id: 'work' } }, { serverId: 'disc', descriptor: { id: 'noisy' } },
      { serverId: 'disc', descriptor: { id: 'general' } }, { serverId: 'slack', descriptor: { id: 'general' } },
    ],
    readCoordinatorState: (id: string) => {
      if (fail.read) throw new Error('store unreadable');
      return state.get(id) ?? null;
    },
    writeCoordinatorState: (id: string, value: unknown) => {
      if (fail.write) throw new Error('store write failed');
      state.set(id, value);
    },
  } as unknown as ChannelRegistry;
  const servers = { getServer: () => undefined } as unknown as McplServerRegistry;
  const hooks = {
    addMessage: (participant: string, content: Array<{ type: string; text?: string }>, metadata: Record<string, unknown> | undefined, forAgent?: string) => {
      if (fail.addMessage) throw new Error('context write failed');
      messages.push({ participant, text: content.map((b) => b.text ?? '').join(''), forAgent });
      if (!forAgent && metadata) stored.push({ participant, content, metadata });
      return 'id';
    },
    commitDeliveries: () => !fail.commit,
    requestInference: (agentName: string) => {
      if (fail.wake) throw new Error('the wake request failed');
      wakes.push(agentName);
    },
    subconsciousName: () => 'Subconscious',
    primaryName: () => 'scout',
    getStoredMessages: () => stored,
    currentSequence: () => 1,
    setSubconsciousAnchor: (sequence: number) => { anchors.push(sequence); },
    isForkBound: () => false,
    isPrivilegedAuthor: () => false,
    allowChannelSpeech: () => false,
    emitTrace: (e: { type: string }) => { traces.push(e.type); },
  } as unknown as TuneOutFrameworkHooks;
  const make = () => new TuneOutCoordinator(registry, servers, hooks);
  return { make, state, wakes, messages, traces, fail, stored, anchors };
}

describe('PROBE old-shape release row', () => {
  it('PROBE old-release-row', () => {
    const h = harness();
    const first = h.make();
    assert.ok(first.enterAllExcept({ serverId: 'disc', channelId: 'work' }, { durationSeconds: 3600 }, 'agent-tool').ok);
    first.stop();
    const id = [...h.state.keys()].find((k) => k.endsWith('/all-except'))!;
    const epochId = (h.state.get(id) as { params: { epochId: string } }).params.epochId;
    const held = (n: number) => ({
      participant: 'u',
      content: [{ type: 'text', text: `PROBE-HELD-${n}` }],
      metadata: { author: { name: 'Bo' }, serverId: 'disc', channelId: 'noisy', tuneOut: { epochId } },
    });
    // Three held rows, then a release row as 0.30.0 writes it (its text shows
    // them; its metadata carries no `released`), then one held after it: the
    // hold's end write failed, or the process stopped before it.
    h.stored.push(held(1), held(2), held(3));
    h.stored.push({
      participant: 'user',
      content: [{ type: 'text', text: '[tune-out: all channels except work released — probe]\n\n3 messages held in 1 channel.\n\n<tuned-out-backlog channel="noisy" messages=3>\nBo: PROBE-HELD-1\nBo: PROBE-HELD-2\nBo: PROBE-HELD-3\n</tuned-out-backlog>' }],
      metadata: { system: true, kind: 'tune-out-cancel', scope: 'all-except', channelId: 'work', epochId },
    });
    h.stored.push(held(4));
    // The restart: a fresh coordinator over the same store and state.
    const second = h.make();
    const before = h.messages.length;
    const result = second.cancelAllExcept('agent-tool', 'probe');
    const releases = h.messages.slice(before).filter((m) => /all channels except work released — probe/.test(m.text));
    const text = releases.map((m) => m.text).join('\n');
    const shownAgain = [1, 2, 3].filter((n) => text.includes(`PROBE-HELD-${n}`)).length;
    console.log('PROBE-RESULT ' + JSON.stringify({
      epochIdSet: typeof epochId === 'string',
      result,
      holdAfter: second.getAllExcept() === null ? 'ended' : 'in force',
      releasesWritten: releases.length,
      earlierShownAgainOf3: shownAgain,
      laterShown: text.includes('PROBE-HELD-4'),
      summaryLine: (text.split('\n\n')[1] ?? '').slice(0, 80),
    }));
    second.stop();
  });
});

  • Changelog fragment added: changelog.d/tune-out-cancel-order.fixed.md.

Hilda-2681 made the change, wrote the probes and ran the whole suite. Crispin-2406 read its first commit word for word, and Esther-2584 the other three, the second before one clause of its message was reworded. Whether each part is needed was reviewed by Gladys-2413 for an earlier version of the first commit and then for what its redesign changed, by Gordon-2414 for what changed after that, and by Esther-2584 for the first commit as it is. The second commit follows Ada-1017's rulings on Esther-2584's review. Ada-1017 settled design questions during review. Four runs of the house's internal review found issues that later rounds of the change addressed. Daniel-2612 reworded the probes' comments, test titles and messages, ran the tune-out tests and the probes at clean checkouts, and wrote this description.

Hilda-2681 added 4 commits October 29, 2026 03:32
…cords the end

TuneOutCoordinator.cancel() recorded the tune-out's end first
(ChannelRegistry.cancelTuneOut appends it to the lifecycle log at once)
and only then read, rendered and wrote the backlog dump. A cancel that
failed between the two, because a write threw or the process died, left
the tune-out ended with no dump. Its held rows keep their tuneOut stamp,
which every resident's view filter leaves out for good, so nothing
delivered them: after a restart no tune-out is on record to dump, and the
boot's restore of deferred writes covers only a dump already queued.

cancel() now reads the tune-out's state, writes the dump, makes it
durable, and only then records the end, the order cancelAllExcept()
already used. A failure before the end record leaves the tune-out in
force, and a later cancel, in this process or after a restart, dumps the
stored held rows no earlier dump covered.

Each dump row records, in its metadata (`released`), the held rows it
covered, those its cap left out among them: rows that came through the
deferred queue by the queued write's id, which the stored row keeps as
deferredWriteId once it lands, and other stored rows by id. It also
records how many it covered, how many it left out, and which rows it
left for the subconscious's report. A cancel reads this epoch's earlier
dump rows from the stored and queued messages it already scans, and
dumps only the held rows none of them covered. It writes no dump when an
earlier one covered every held row. A dump the store keeps in shards
carries the same record on each shard, and counts once. Both cancels use
this rule, and it replaces the in-memory mark that kept the all-except
release from being written twice in one process. That mark lived apart
from the delivery, so a dump that was stored before its commit threw, or
taken away by a rollback, or written before a restart, could disagree
with it. The row can't, so a retry shows no held message twice, after a
restart too, and the restart's second dump that the old all-except order
accepted is gone. After a retry, a channel's report to the subconscious
says what each dump of the epoch left out, each dump's own oldest, and
names the dump when only one was cut.

The framework's commitDeliveries() now says whether the release is
durable: false when the deferred queue's recovery write fails or is over
its cap, or the store's sync fails. Both were only logged before. Both
cancels record the end only on true, and otherwise throw, leaving the
tune-out or hold in force for the next cancel.

Every cancel a timer makes is tried again when it throws. The end timers
already armed their retry first; the cancel of a tune-out past its wake
budget now does too, in the channel's expiry slot, so a quiet tune-out
with no end time still ends. At the start, a tune-out or hold that
reached its end time while the process was down and whose cancel throws
stays in force as one does at runtime, its cadence running and its retry
armed, one stderr line naming it, and the start goes on to arm the
others; before, the throw ended the framework's start. Whether it is
still in force is read from the registry, so one whose cancel ended it
and then failed a later step counts as ended: no retry, and it doesn't
anchor the subconscious. A tune-out found past its wake budget is
cancelled at the start, since its retry is a timer a restart loses. A
tool close of the excepted channel whose release fails or is refused
answers "The close wasn't recorded", with the release's reason: a
release that failed before the hold's end was recorded leaves the hold
in force, and one that failed after leaves it ended. docs/dendrite.md
says the same of a hold whose release fails at the start: before its
end, it stays in force and is tried again a minute later; after, it has
ended, with no retry. A tune-out past its wake budget whose cancel fails
at the start says so on stderr as the start's, with no timer-failed
trace, since no timer fired; one whose cancel ended it and then failed a
later step says it ended.

The reorder opened a window in which a hook called during the dump could
start a second cancel of the same channel. A second cancel of the same
channel, or of the all-except hold, made while one is running is now
refused, and a cancel records the end only if the registry still holds
the epoch it read.

The channel's timers and its count of messages diverted since the
reader's last look are cleared once the end is recorded, so a cancel
that fails leaves them running.

The doc comments of cancel(), cancelAllExcept() and both commitDeliveries
are removed rather than rewritten: they documented internals, and this
message carries what they said.

Four test files' stand-in registries said no channel was tuned out while
their cancelTuneOut ended one. cancel() now reads the tune-out's state
first, so each stand-in answers it for the channel being cancelled until
that channel's end, as the registry does. The end-time test counted the
registry's cancelTuneOut calls to show a retry doesn't happen; cancel()
no longer calls it for a tune-out already ended, so the test counts the
coordinator's cancel attempts instead. The test of an end time whose
cancel throws after the registry ended the tune-out made the dump throw,
which now comes before the end; it now makes the resident's wake throw,
which comes after it, and checks the tune-out ended. all-except-unit's
harness now stores the release rows its addMessage is given, as the
framework does, and answers commitDeliveries.

Evidence, built, on the published packages, each phase swapping only
src/tune-out/coordinator.ts, over the 44 test files whose source names
TuneOutCoordinator, tune_out, tuneOut, cancelTuneOut or all-except:
- with the coordinator as it was, 1031 of 1053 tests pass. The 22 that
  fail are the channel tests for a cancel after a failed dump (after a
  restart, and in the same process), a retry after a written dump whose
  end record failed (in the same process, with a message held since,
  after a restart, after a rollback, and with a queued row that lands),
  a dump that couldn't be made durable, a cancel made during its own
  dump, the report after a retry, the five tests of the start (an
  expired tune-out that can't be made durable, one that fails after its
  end, one past its wake budget, and one past its budget whose cancel
  fails there, before its end and after), and the wake budget's retry;
  and all-except-unit's tests for the start's failed release, a message
  held while the clear keeps failing, a restart between the release and
  a failed clear, the release record's counts, a cancel made during the
  release, and a release that couldn't be made durable;
- at this commit, 1053 of 1053 pass.
…onscious's report sums what the dumps left out

A cancel refused a second cancel of the same channel, or of the
all-except hold, made while one was running, and recorded the end only
if the registry still held the epoch it read. Nothing reaches a cancel
while one runs. A cancel is synchronous, and its callers are an agent's
tune_out or focus call, a tool close of the excepted channel, the
coordinator's own expiry and wake-budget timers, the start, and the
subconscious's judgment. None is reached from what a dump calls: the
store write, the durability commit, the room and history-tool reads and
the row reads. The one way left is a host calling the public
executeToolCall synchronously from a trace listener during a dump, since
handleTuneOutTool has no await before the cancel; connectome-host never
calls it. So the refusal, its two wrappers, the epoch
checks before the end and their two tests go.

The cancel report the subconscious gets, or the last reader fork, said
for each dump how many of its held messages it left out. Each dump's
own text already says that, for a reader that can see it. A dump still
queued behind the resident's turn is in no reader's view yet, and its
number in a per-dump count wouldn't let a reader find it either, so the
report gives the sum in one sentence.

The fragment drops its sentence on the refusal, and now says that a
tune-out whose queue of writes waiting to be stored is over its cap
stays in force past its end time, its cancel tried again every minute,
until the queue is back under it.

The rollback test's title now says what its removal stands for: a
rollback, an undo of the turn or a surgery, each of which can take a
stored dump row away.

With the parent's coordinator, the two changed test files fail only the
report test, on the summed wording.
…g cancels again, and the summed report counts a dump that left nothing out

The two tests of a start whose cancel ended a tune-out and then failed a
later step, one past its end time and one past its wake budget, counted
the registry's ends or the dumps. A retry of a tune-out that has ended
returns at cancel()'s "not tuned out" before it ends or dumps anything,
so neither count could see one. Both now count the coordinator's
cancel() calls, its timers' included, through ten minutes, and the end
time's test also checks the line that says the cancel ended it.

The all-except hold's start had no test for a release that ended the
hold and then failed a later step. It has one now: the hold is ended,
nothing anchors the subconscious, the line says the release ended it,
and nothing releases it again. The unit harness's wake can fail for
it, and it records the anchors the start sets.

The summed report's test had both dumps leave messages out, so a sum of
the dumps that left something out would have passed it. Its second dump
now holds two messages under a cap of two, so the report reads "1 of 5
diverted messages across 2 dumps".
…etry

Its doc named only an end time's retry. The wake budget's cancel, at a
wake or at the start, is tried again after the same interval.
@nissa-seru
nissa-seru merged commit 3b7191c into anima-research:main Oct 30, 2026
15 checks passed
tel-0s added a commit that referenced this pull request Oct 30, 2026
…p durable before its end, gate texts as written, awaited calls routed as the turn's, a stop that waits for a start

main at 48f26ce is four merges past d700dce, with no release among
them, so package.json stays 0.30.0 and this branch's floors stand
(chronicle ^0.6.0, context-manager ^0.25.0, membrane ^0.8.0), which
test/dependency-ranges pins. It brings #910 (the gate's refusals and file
errors name fields, rules and gate.json's path as written, and a change
says it is saved only when it brings gate.json into force), #909 (a
channel's tune-out cancel stores its dump and makes it durable before it
records the end; each dump row records the held rows it covered
(`released`), so a retried cancel shows nothing twice; commitDeliveries
answers whether the release is durable; every timer's cancel is tried
again when it throws, at the start too, and a tune-out past its wake
budget is cancelled at the start; a tool close of the excepted channel
whose release fails answers "The close wasn't recorded"), #913 (the
turn's dispatch and an awaited tool call take one route per tool,
toolRoute; the puppet refuses skip_reply) and #912 (a module's stop()
waits for its start() to settle; addModule() refused once a stop() is
admitted, after the add).

Three paths conflicted (17 regions):

- src/tune-out/coordinator.ts (12), the #909-into-#915 port. This branch
  has no deferred queue (#291), and its cancel waits while the framework
  keeps writes in memory (writesHeld: a surgery's hold, a manager
  refusing its work; deferCancel / runDeferredCancels, answered
  `deferred`, the tools' `waiting` + `note`). #909's rule goes into that
  cancel as it stands:
  - cancel(): reads the tune-out's state (getTuneOutState) first, and
    answers "not tuned out" without it; then, writes held, waits as
    before, nothing dumped or ended. Past the wait: the backlog is the
    epoch's rows (residentMessages: stored, then a surgery's kept writes
    where writesHeld threw) less those an earlier dump of the epoch
    covered (releaseRecordsOf over the same rows); the dump is written
    with its `released` record, and none when an earlier dump covered
    every row; commitDeliveries() === false throws, the tune-out in
    force; only then cancelTuneOut records the end, and the channel's
    timers and diverted count are cleared. The subconscious's report
    and the last reader fork's notice sum what the epoch's dumps left
    out (truncationNote). Main's heldForReport (queued rows handed to
    the subconscious as tune-out-cancel-held rows) stays out, as on this
    branch: every held message is stored, and the report keeps its
    unseenBySubconscious count; the record's `report` is always empty
    here, kept so the record has the shape main reads.
  - cancelAllExcept(): the same, over the hold's rows; main's `released`
    record replaces this branch's in-memory scopeReleased mark, and
    commitDeliveries() === false throws before writeScope(null).
  - Rows are keyed by id, and by `deferredWriteId` (writeIdOf, kept):
    a row an earlier version queued, stored by the boot's import, is
    named by its queued write's id in that version's records, so a
    cancel retried after an upgrade finds it covered. A surgery's kept
    write has no id, so a dump made during the hold (writesHeld having
    thrown) can't name it; a retry of that dump dumps it again.
  - resumeActiveEpochs(): main's try/catch around the start's cancels,
    their retries (armExpiry's and armScopeTimers' EXPIRY_RETRY_MS) and
    the start's wake-budget cancel (cancelPastBudget); main's
    `inForce`/oldestAnchor bookkeeping goes, since this branch anchors
    from the tune-outs the record holds after the cancels
    (pruneWindowStarts, anchorSubconscious('resume')), which leaves out
    one whose cancel ended it and counts one whose cancel waits.
  - The hooks' getDeferredMessages stays out (this branch's
    getSurgeryHeldMessages); commitDeliveries is `() => boolean`, its doc
    kept and rewritten. cancel()'s and cancelAllExcept()'s doc comments,
    which main removed, stay, rewritten to the new order: they carry this
    branch's `deferred` answer.
- src/framework.ts (4):
  - addModule(): main's refuseOnceStopped('addModule()') after the add,
    with this branch's comment on useHistory (its check leaves out an
    undone row).
  - the deferred queue's persistence (persistDeferredWrites and the
    queue's record): stays deleted, as on this branch; main's change
    there was persistDeferredWrites' boolean.
  - onToolClose: main's form (a release that throws or refuses answers
    "The close wasn't recorded: …"), with this branch's sentences for a
    release that waits and one that went through ("The focus on it
    ended, …", 6243858).
  - commitDeliveries(): answers false when the store's sync fails and
    true otherwise; there is no queue to persist.
- src/gate/event-gate.ts (1): a pending batch whose delivery throws:
  main's `Rule "<name>"` opening, with this branch's deliveredInPart for
  a batch that woke some recipients before it threw.

Read onto this branch past the conflicts:
- docs/dendrite.md: a cancel stores and syncs its release before it
  records the end; one whose release can't be made durable throws, the
  tune-out in force, and the next cancel delivers what no earlier
  release covered (`released`). Main's sentence on the all-except
  hold's start came in clean.
- test/all-except-unit: main's tests take this branch's harness
  ({ stored, kept, writesHeld }); its anchors hook reads the start's
  sequence (setSubconsciousAnchor takes a SubconsciousWindowStart here).
- test/tune-out-cancel-order: no getDeferredMessages hook; the anchors
  hook as above. Main's "a message queued at a dump whose end record
  failed, which lands before the retry" has no queue to stand on here;
  it becomes the upgrade case: a dump an earlier version wrote, naming a
  row by its queued write's id, and that row stored by the import with
  deferredWriteId, gets no second dump.

storeSees: no new site; main opens and derives no context manager.
agentSees: no new view decision. #913's awaited routes run the turn's
handlers with a `deliver` that answers the caller; none stores a row of
its own (the puppet's pair is stored as before, its agent's own row).
Notice tags: none new; #910's texts are tool results and stderr lines,
#909's are stderr lines and errors thrown to a tool or timer.

Changelog: no release in main's range, so no fragment was consumed;
main's seven new fragments come in as written. 291-deferral-queue's
tune-out hosts' bullet says that with no queue a release fails to be
made durable only where the store's sync fails, so no queue over its
cap keeps a tune-out in force (tune-out-cancel-order's sentence), and
how a dump's record names a row.

Tests: built with zero errors; no stale compiled test; fixtures copied.
main's 16 changed test files: 419 of 419. presentation*, window-*,
dendrite-*, stored-injections, audience-stamp*, dependency-ranges,
held-*, tune-out-*, admission-e2e, all-except*, mcpl-coalescing*,
operator-surgery, read-marks* and turn-inputs-pin (53 files): 964 of
964. With this branch's coordinator as it was (HEAD's), and the rest of
the merge, tune-out-cancel-order, all-except-unit, tune-out-timers and
tune-out-gate-composition fail 21 of 77; at this merge, 77 of 77.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant