[nissa-seru]

When an agent is stopped, or its job's activation cancelled, this clears its failure accounting, so a new agent under its name meets its first failed request as its first. Fixes #885, which FableDendrite filed.

Before, a new agent under a stopped agent's name started with the stopped agent's consecutive-failure streak, poison-history rewind count, refusal streak, refusal-rewind count and refusal rewind episode. By run at 0f11975e48, which was main until 07:26 UTC on October 29, a new agent under a stopped resident's name counted its first refused request as its third failure, so it was reported hard-down and the poison-history breaker set aside its own message. A job whose run ends on its own, and which is resumed under its name in the same process, keeps its accounting, as at 0f11975e48. For a job resumed in the same process, one thing changes: a running job an operator stops or cancels, then resumes, now starts its count clean, where at 0f11975e48 it carried. The change writes nothing to the store. main has since moved to 4a45766f01 (#869), and this head merges with it without conflicts.

Problem

releaseAgentRuntime (src/framework.ts:11804 at 0f11975e48) is the release of an agent that ends by a stop, by its parent's end or by a fork's dispose. The comment above the release test at 0f11975e48 says what the release is for: it "drops this per-name turn state, so a new agent under the name starts clean" (test/exhausted-turn-rearm.test.ts:929–931). At 0f11975e48 it doesn't drop five per-name maps of failure accounting: consecutiveInferenceFailures, exhaustionRewinds, refusalStreak, refusalRewinds and rewindEpisode (src/framework.ts:3776–3819). Nor does it drop freshCallsCut, the mark that a cut call's fresh call was cut too, which the next failure note reads (:3621, :31480), or forcedRewind, an open /unstick session (:3823).

By run, with the probe below added to a clean 0f11975e48 and to this head, on the platform under "Tests". The probe prints what it sees and asserts nothing.

• A: a resident, second, fails two turns with a 400 and is stopped. A new agent is made under its name, as the release test at 0f11975e48 makes one. At 0f11975e48, the new agent's first refused request logs consecutive=3 and [inference-hard-down], and raises a hard-down alert, and the poison-history breaker sets aside the new agent's own message. At this head it logs consecutive=1, raises no alert, and keeps its message.
• B: second fails until its poison-history rewind budget is spent, and is stopped. At 0f11975e48, the new agent's first failure logs consecutive=7, hard-down, and "rewind cap 3 reached — not shedding further history. Manual repair (/unstick or /undo) needed." At this head it logs consecutive=1 alone.
• C: the model refuses second once, and it's stopped. At 0f11975e48, the new agent's first refusal raises the alert "refusal streak 2, category=reasoning_extraction". At this head it raises none.
• D: a job made with createEphemeralAgent fails twice, is stopped while its third call is in flight, and is resumed. At 0f11975e48, the resume's first failure logs consecutive=3 and hard-down, and raises a hard-down alert, and the poison-history breaker sets aside the job's newest message. At this head it logs consecutive=1 and keeps the message.

#885's other stated consequence, that the failure marker may speak of earlier rounds the new agent never ran, doesn't hold at 0f11975e48. A note's sentence about earlier rounds, "Its earlier rounds ran…", comes from the exhausted-turn record (src/framework.ts:31551 at 0f11975e48). The test "a resident stopped mid-turn takes its turn state with it…" at 0f11975e48 (test/exhausted-turn-rearm.test.ts:932) sets that record, with its rounds run, before the stop, and checks that the release drops it. It passes with 0f11975e48's src/framework.ts in the red run under "Tests". In case A, the new agent's failure note has the same text at both commits, and only its metadata's consecutive differs: 3 at 0f11975e48, 1 at this head.

The probe, test/probe-885.test.ts, as it ran

// Probe for #885: failure accounting under an ended agent's name, in one process.
// Each case prints what it saw; it asserts nothing. Run after `npx tsc -p .`:
//   node --test dist/test/probe-885.test.js
import { test, type TestContext } from 'node:test';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { MembraneError } from '@animalabs/membrane';
import type { NormalizedResponse, StreamEvent, YieldingStream } from '@animalabs/membrane';
import type { EventResponse, Module, ProcessEvent, ToolResult } from '../src/index.js';
import { AgentFramework, workerSpec } from '../src/index.js';
import { MockMembrane, MockYieldingStream, createMockResponse } from './helpers/mock-membrane.js';

const log: string[] = [];
console.error = (...a: unknown[]) => { log.push(a.map(String).join(' ')); };
const show = (label: string, value: unknown) => process.stdout.write(`PROBE ${label}: ${JSON.stringify(value)}\n`);
const linesFor = (name: string) => log.filter((line) => line.includes(`agent=${name} `)).map((line) => line.slice(0, 160));

// Every request is refused with a 400, or (for 'refuse') answered with a model refusal; 'hold' keeps the call open until it's cancelled.
const refused = () => new MembraneError({ type: 'invalid_request', retryable: false, httpStatus: 400, message: 'probe: invalid request', rawError: undefined } as never);
class FailingStream implements YieldingStream {
  isWaitingForTools = false; pendingToolCallIds: string[] = []; toolDepth = 0; isCancelled = false;
  private release = () => {};
  constructor(private readonly heldOpen: boolean) {}
  provideToolResults(): void { throw new Error('not waiting'); }
  cancel(): void { this.isCancelled = true; this.release(); }
  async *[Symbol.asyncIterator](): AsyncIterator<StreamEvent> {
    if (this.heldOpen) await new Promise<void>((resolve) => { this.release = resolve; });
    if (this.isCancelled) return;
    yield { type: 'error', error: refused() } as StreamEvent;
  }
}
const refusal = () => ({
  ...createMockResponse([], 'refusal'), content: [], stopReason: 'refusal', toolCalls: [], rawAssistantText: '',
  usage: { inputTokens: 40, outputTokens: 0 },
  details: { stop: { reason: 'refusal', wasTruncated: false }, usage: { inputTokens: 40, outputTokens: 0 }, timing: { totalDurationMs: 100, attempts: 1 },
    model: { requested: 'probe-model', actual: 'probe-model', provider: 'mock' }, cache: { markersInRequest: 0, tokensCreated: 0, tokensRead: 0, hitRatio: 0 } },
  raw: { request: {}, response: { stop_details: { category: 'reasoning_extraction' } } },
}) as unknown as NormalizedResponse;
class ProbeMembrane extends MockMembrane {
  sends = 0;
  constructor(private readonly reply: (send: number) => 'fail' | 'refuse' | 'hold') { super(); }
  override streamYielding(): YieldingStream {
    const kind = this.reply(++this.sends);
    return kind === 'refuse' ? new MockYieldingStream([refusal()]) : new FailingStream(kind === 'hold');
  }
}

// An external message whose content names an agent wakes that agent.
class Wake implements Module {
  readonly name = 'wake';
  async start(): Promise<void> {}
  async stop(): Promise<void> {}
  getTools() { return []; }
  async handleToolCall(): Promise<ToolResult> { return { success: true }; }
  async onProcess(event: ProcessEvent): Promise<EventResponse> {
    return event.type === 'external-message' ? { requestInference: [String((event as { content: unknown }).content)] } : {};
  }
}

async function until(done: () => boolean, what: string): Promise<void> {
  const deadline = Date.now() + 10_000;
  while (!done()) {
    if (Date.now() > deadline) throw new Error(`probe: ${what} didn't happen within 10 s`);
    await new Promise((resolve) => setTimeout(resolve, 10));
  }
}

async function start(t: TestContext, membrane: ProbeMembrane) {
  log.length = 0;
  const dir = mkdtempSync(join(tmpdir(), 'probe-885-'));
  const fw = await AgentFramework.create({
    storePath: join(dir, 'store'), membrane: membrane.asMembrane(), modules: [new Wake()], syncIntervalMs: 0, maintenanceIntervalMs: 0,
    exhaustedTurnRearmMs: false,
    agents: [{ name: 'resident', model: 'probe-model', systemPrompt: 'system' }, { name: 'second', model: 'probe-model', systemPrompt: 'system' }],
  } as never);
  const traces: Array<{ type: string; kind?: string; message?: string; agentName?: string }> = [];
  fw.onTrace((e) => { traces.push(e as never); });
  t.after(async () => { await fw.stop().catch(() => {}); rmSync(dir, { recursive: true, force: true }); });
  fw.start();
  const alerts = () => traces.filter((tr) => tr.type === 'ops:alert').map((tr) => `${tr.kind}: ${String(tr.message).slice(0, 80)}`);
  // `second` runs `turns` turns, each ending exhausted or refused.
  const turns = async (n: number, ended: () => number) => {
    for (let i = 1; i <= n; i++) {
      const before = ended();
      fw.pushEvent({ type: 'external-message', source: 'probe', content: 'second', metadata: {} } as never);
      await until(() => ended() > before, `second's turn ${i} to end`);
    }
  };
  const exhausted = () => traces.filter((tr) => tr.type === 'inference:exhausted' && tr.agentName === 'second').length;
  const refusals = () => log.filter((line) => line.startsWith('[inference-refusal] agent=second')).length;
  return { fw, alerts, turns, exhausted, refusals };
}

// A new agent under `second`'s name, after `second` was stopped, as main's release test makes one.
async function newSecond(fw: AgentFramework, history: string[][] = []) {
  const spec = workerSpec('second');
  const { name: _name, ...body } = { ...spec, roles: { ...spec.roles, ownsProviderScheduling: true } };
  const made = await fw.createEphemeralAgent({ name: 'second', model: 'probe-model', systemPrompt: 'system' }, body);
  for (const [q, a] of history) {
    made.contextManager.addMessage('User', [{ type: 'text', text: q! }]);
    made.contextManager.addMessage('second', [{ type: 'text', text: a! }]);
  }
  made.contextManager.addMessage('User', [{ type: 'text', text: 'probe-new-message' }]);
  await fw.runEphemeralToCompletion(made.agent, made.contextManager).catch(() => {});
  const messages = made.contextManager.getAllMessages();
  const texts = messages.flatMap((m) => m.content).map((b) => (b as { text?: string }).text ?? '');
  const notes = messages.filter((m) => m.content.some((b) => ((b as { text?: string }).text ?? '').startsWith('[inference-failed]')))
    .map((m) => ({ text: m.content.map((b) => (b as { text?: string }).text ?? '').join(' '), metadata: m.metadata }));
  return { texts, notes };
}

test('A: second fails two turns and is stopped; a new agent under its name meets its first refused request', async (t) => {
  const r = await start(t, new ProbeMembrane(() => 'fail'));
  await r.turns(2, r.exhausted);
  r.fw.stopAgent('second', { by: 'probe-operator' });
  log.length = 0;
  const alertsBefore = r.alerts().length;
  const { texts, notes } = await newSecond(r.fw);
  show('A new agent logs', linesFor('second'));
  show('A alerts after the stop', r.alerts().slice(alertsBefore));
  show('A new agent keeps its own message', texts.includes('probe-new-message'));
  show("A new agent's failure note", notes);
});

test('B: second, with three exchanges, fails until its rewind budget is spent and is stopped; a new agent with one exchange meets its first refused request', async (t) => {
  const r = await start(t, new ProbeMembrane(() => 'fail'));
  const cm = r.fw.getAgent('second')!.getContextManager();
  for (const n of [1, 2, 3]) {
    cm.addMessage('User', [{ type: 'text', text: `probe-old-question-${n}` }]);
    cm.addMessage('second', [{ type: 'text', text: `probe-old-answer-${n}` }]);
  }
  // From its third failure on, each failure sets aside an exchange and retries, until the budget of three is spent.
  await r.turns(3, r.exhausted);
  await until(() => linesFor('second').some((line) => line.includes('rewind cap 3 reached')), "second's rewind budget to be spent");
  show('B second before the stop', linesFor('second').map((line) => line.slice(0, 70)));
  r.fw.stopAgent('second', { by: 'probe-operator' });
  log.length = 0;
  await newSecond(r.fw, [['probe-new-question', 'probe-new-answer']]);
  show('B new agent logs', linesFor('second'));
});

test('C: second is refused once and stopped; a new agent under its name is refused once', async (t) => {
  const r = await start(t, new ProbeMembrane(() => 'refuse'));
  await r.turns(1, r.refusals);
  show('C alerts before the stop', r.alerts());
  r.fw.stopAgent('second', { by: 'probe-operator' });
  const alertsBefore = r.alerts().length;
  await newSecond(r.fw);
  show('C alerts after the stop', r.alerts().slice(alertsBefore));
});

test('D: a job fails twice, is stopped while its third call is in flight, and is resumed', async (t) => {
  const membrane = new ProbeMembrane((send) => (send === 3 ? 'hold' : 'fail'));
  const r = await start(t, membrane);
  log.length = 0;
  const config = { model: 'probe-model', systemPrompt: 'system' };
  const run = async (made: { agent: Parameters<AgentFramework['runEphemeralToCompletion']>[0]; contextManager: Parameters<AgentFramework['runEphemeralToCompletion']>[1] }, text: string) => {
    made.contextManager.addMessage('User', [{ type: 'text', text }]);
    return r.fw.runEphemeralToCompletion(made.agent, made.contextManager).catch(() => {});
  };
  await run(await r.fw.createEphemeralAgent({ name: 'job', ...config }), 'probe-task');
  await run(await r.fw.resumeAgent('job', config), 'probe-again');
  const held = run(await r.fw.resumeAgent('job', config), 'probe-held');
  await until(() => membrane.sends === 3, "the job's third call to be in flight");
  r.fw.stopAgent('job', { by: 'probe-operator' });
  await held;
  const last = await r.fw.resumeAgent('job', config);
  const alertsBefore = r.alerts().length;
  await run(last, 'probe-after-the-stop');
  show('D job logs', linesFor('job'));
  show('D alerts after the stop', r.alerts().slice(alertsBefore));
  show('D resume keeps its newest message', last.contextManager.getAllMessages().flatMap((m) => m.content)
    .some((b) => (b as { text?: string }).text === 'probe-after-the-stop'));
});

Changes

Line numbers are this head's, 98f84cc00e.

• One helper, forgetFailureAccounting (src/framework.ts:11798), clears the five maps and ends an open /unstick session, with a stderr line saying it ended with its agent. The channel the /unstick came from isn't told.
• The release calls it and also clears freshCallsCut (:11851–11852).
• A running job ended from outside. stopAgent ends a running job's run through teardownEndedAgent (:11447), and so does a parent's end, for each agent it cascades to (:11921). A cancel of a job's activation ends its run in cancelActivation. None of these reaches the release. teardownEndedAgent and cancelActivation now mark the run endedFromOutside (:11789, :11410), and the run's end calls the helper when the run is so marked (:12621). endReason couldn't tell, since a run's own completion and its deadline set it too.
• A run's own end, by a failure, a completion or its deadline, keeps what accounting it has and an open /unstick session, and clears only freshCallsCut (:12620). abortInference and an operator's quiesce abandon don't mark the run, so a run they end keeps its accounting, as a run's own end does.
• disposeConversationAgent's branch with no live agent calls the helper and clears freshCallsCut (:17769–17770), as it already cleared deadlineCuts.
• The changelog fragment, changelog.d/885-release-clears-failure-accounting.fixed.md.

So a job that ended on its own keeps what accounting it has when it's resumed under its name with resumeAgent in the same process. A resume reopens the same history, and a history refused across resumes is what the streak and the breaker count.

What each part is for

• forgetFailureAccounting: one rule for what the end of an agent clears, called from the three places that end one, so they can't drift apart. Its stderr line tells an operator that the /unstick session they opened has ended.
• The call in the release: every end the release covers drops the accounting, beside the release's other per-name clears (deadlineCuts, thinkingLossIncidents).
• endedFromOutside: a running job's stop or cancel never reaches the release, so the run carries why it ended. Clearing at every end of a run would lose the history a resumed job's streak and breaker count.
• freshCallsCut at every end: the mark is about one run's last call, so a resume's first failure note shouldn't say a call was made again.
• Dispose's branch with no live agent: a fork disposed with no live agent drops the same accounting. It's left untested, as deadlineCuts there was in feat(inference)!: cut an AgentFramework model call that stops making progress, and give a deadline expiry one fresh call #378.
• The tests' refusal() and RefusingMembrane: a model refusal for the refusal tests, built as test/refusal-single-call.test.ts builds one.

Each part, taken out alone at this head and rebuilt, fails a named test of test/exhausted-turn-rearm.test.ts:

• each of the five deletes in forgetFailureAccounting, and its forcedRewind line: the resident-stopped test and both in-flight tests, and for refusalStreak, the refusal-stopped test too;
• the helper's call in the release: the resident-stopped and refusal-stopped tests;
• the release's freshCallsCut delete: the resident-stopped test;
• the run's end's freshCallsCut delete: the fresh-call-mark test;
• the run's end never calling the helper: the stopped and cancelled in-flight tests; always calling it: the four tests of a job's own end;
• the mark in cancelActivation: the cancelled in-flight test; the mark in teardownEndedAgent: the stopped in-flight test.

Tests

From a clean copy of this head, 98f84cc00e, on Linux 6.12.113 (amd64) with Node 24.22.1, installed with no lockfile. npm ls --depth=0 listed @animalabs/chronicle 0.5.0, @animalabs/context-manager 0.24.1, @animalabs/mcpl-core 0.4.0, @animalabs/membrane 0.8.0, @babel/parser 7.29.11, @modelcontextprotocol/client 2.4.0, @modelcontextprotocol/server 2.4.0, @types/node 22.20.5, @types/ws 8.18.2, chokidar 4.0.3, discord.js 14.27.0, tsx 4.23.15, typescript 7.0.2 and ws 8.22.0.

• tsc -p . exits 0.

• Green: exhausted-turn-rearm, inference-bounds, inference-exhaustion-rewind and refusal-single-call pass, 137 tests.

• Red first: this head's test/exhausted-turn-rearm.test.ts with 0f11975e48's src/framework.ts runs 64 tests, and 5 fail, each at the assertion named:

  • "a resident stopped with a failure streak…", at "the release drops each";
  • "a job stopped while its run is in flight…" and "a job cancelled while its run is in flight…", at "the run's end drops each, as the release does";
  • "a resident stopped after a refusal…", at "and its first refusal raises no streak alert";
  • "a job's own run's end takes its fresh-call mark…", at "which says nothing of a call made again".

  The other four new tests pass there too: at 0f11975e48 a job's own end already keeps its accounting.

• Time: exhausted-turn-rearm alone, by the test runner's duration, three rounds, each with 0f11975e48's src/framework.ts and test file (55 tests) and then this head's (64 tests), on 12 CPUs with a one-minute load of 26.85 to 28.25: 2,701, 2,719 and 2,934 ms for 0f11975e48's, and 3,074, 2,698 and 3,202 ms for this head's.

The tests this change adds to test/exhausted-turn-rearm.test.ts:

• a resident stopped with a failure streak of two and the other state set leaves none of it, says its /unstick session ended, and a new agent under its name meets its first refused request as its first: consecutive=1, not hard-down, no hard-down alert, its own message kept, and its failure note saying nothing of a call made again;
• a resident stopped after a refusal leaves a new agent's first refusal with no refusal-streak alert;
• a job stopped, and a job whose activation is cancelled, while its run is in flight, leaves none of its accounting or its /unstick session behind, and its resume meets its first refused request as its first;
• a job resumed after its own run ended on a failure, on a refusal or at its deadline counts on from that run, and in the failure case its resumed context holds the run's own message;
• a job whose resumed call is aborted counts on past the abort, which adds nothing to its streak;
• a resume after a run that left the made-again mark says nothing of a call made again.

Not verified

• refusalRewinds and rewindEpisode's effect on a new agent. The tests check only that they're cleared. No run had a new agent meet a refusal auto-rewind under autoRewind, or the consolidated rewind marker.
• A parent's end reaching a running job. No run ended a running job by its parent's end. A parent's end calls teardownEndedAgent for each agent it cascades to (src/framework.ts:11921), as stopAgent does (:11447).
• An operator's quiesce abandon. No run ended a job that way.
• freshCallsCut within one live agent. A failure note that reads the mark clears it (src/framework.ts:31498). No run had a turn set the mark and then end with no failure note.
• Hosts. No run used a host. In connectome-host's main at f8c45174, as of 07:14 UTC on October 29, the subagent module names each agent it spawns with a time stamp (nameStamp(), src/modules/subagent-module.ts:3207), and resumes only a job that a host restart, its deadline or a failure ended (resumeRefusal, :940–956). Other hosts weren't read.
• Other platforms, and the full suite. Every run here was on Linux, and none ran the whole suite. CI runs it on this pull request.

What this leaves

• A stop from a listener of an agent's own trace leaves its work acting under its name: A stop from a listener of an agent's own trace leaves its work acting under its name #894. The runs there reach more than the failure accounting, the re-arm state among it, so the fix is a change of its own.
• A cancel of a run that has already ended on its own. With this change, a cancelActivation from a job's own inference:exhausted listener, after the job's run has failed on its own, marks the run as ended from outside, so its end clears accounting a resume should keep. That's path 4 in A stop from a listener of an agent's own trace leaves its work acting under its name #894.

---

• Changelog fragment added: changelog.d/885-release-clears-failure-accounting.fixed.md.

Nigel-2405 made the change, and Ada-1017 made its design calls. Desmond-2579 found that freshCallsCut was left over. Stanley-2495 read the change and its commit message. Duncan-2410 checked by reading that each part is needed, against a smallest change he wrote from the purpose before reading the diff, and cut nothing. Our internal review read it in three rounds, and Ada chose which findings to fix. Fiona-2757 wrote this description.

[nissa-seru]

@antra-tess Flagging for your awareness this PR, which made the following changes to what residents see (in brief), for an agent stopped, ended with its parent or disposed as a fork:

• Its failure accounting now goes with it. A new agent created under its name, or a job resumed after such an end, meets its first failed request as its first, so the poison-history breaker no longer sets aside the new agent's own message on one refusal.
• The new agent's [inference-failed] note no longer says The call was made again, and it timed out again. from the old agent's last timed-out call.

— Lottie-2661