Repository navigation
fix(runtime): recover from a killed Host and stop rereading the store for previews - #6519
Conversation
Replacing the process Host in 9e2a192 dropped its recovery call, so `host serve`, candidate Hosts and `application serve` left a killed Host's Turns open. Sessions then stayed busy for good and every send returned `agent_admission_failed`. Move invocation and shell recovery into `RootStore::open`, beside the effect and auxiliary recovery it already ran, so no entry point can receive an execution store with work abandoned by the previous owner. This removes the second recovery pass and the host-runtime wrapper, and `maka code` now opens through `maka_transport::open_store` like the other entry points, which waits for unconfirmed cleanup. Log the store error behind `agent_admission_failed`, which hid the cause.
d1404ea moved streamed text to an in-memory preview but left its wake in the watch loop, which reread queue, interactions, computer apps, session and events on each one, plus once more after emitting. A viewed stream at 30 increments per second cost the Host about 26% of a core. Read the store only for the first pass, a commit notice, or committed events not delivered yet. A cut's dependency can still fence the event page below the committed head, so that remainder keeps being read without waiting for another notice.
Astro-Han
left a comment
There was a problem hiding this comment.
Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.
Reviewed the complete 13-file diff at 7df7d56b820baf50cb6dacc7e9721298a635f0cd and traced the Root startup recovery barrier and observation loop in source. I found no confirmed P0–P2 issue in this source review.
Invocation and shell recovery now occur while RootStore::open still owns startup, after the existing creation/checkpoint/effect recovery and before maintenance begins or the store is returned. The recovery APIs retain their original cleanup barriers, and opening errors still take the store shutdown path. The CLI code entry now uses maka_transport::open_store rather than a second recovery wrapper. The new crash test exercises kill/reopen/new admission in its source, but I did not execute it.
The watch loop reads durable state for its first pass, a commit notice or undelivered committed events. Preview-only wakes no longer force those reads. I checked the fenced-page through < committed condition, the preview-ready continuation and the biased selection that handles cancellation and commit notices before another immediate preview pass. An actual merge calculation against Rust dfc556ebf50ee5f6cfd71ddfa2d1c097faf27fa6 is clean.
This machine has no Rust toolchain. I did not execute recovery, kill/restart, concurrency or native Windows tests, force a failed startup cleanup, measure CPU or reproduce the reported 1,024-session results. In particular, source inspection alone does not establish preview/commit fairness under load or the performance improvement. Those remain validation limits; the contributor's test and measurement results are not my independent results. CI is still running.
P3: AI contribution is declared, but the tool is not named and the two reviewed commits lack the required Generated-by: <tool> trailer. Please complete that attribution and preserve it when squashing.
Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.
…h-recovery-watch-cost
#6517 changed two notice inputs under packages/plugin-sdk without regenerating the manifest, so the notices test fails on rust-preview and stops `just check` before the Rust suites run.
…x/host-crash-recovery-watch-cost # Conflicts: # products/maka/cli/notices.json
…x/host-crash-recovery-watch-cost
Astro-Han
left a comment
There was a problem hiding this comment.
Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.
I rechecked the current 13-file diff at 61f13397a701990cd8cd9337619262d6acc80ff7 against its updated Rust base d513a72e7ebdc11842442341bbf680deac886835.
The author patch has the same stable patch ID as reviewed 7df7d56b820baf50cb6dacc7e9721298a635f0cd (f07a7469c2ca7b8133b4052e29938bf22dcf0862). All 12 retained changed files are byte-identical; platform/host-runtime/src/recovery.rs is absent in both. I reread the current recovery/watch diff, keeping incoming storage, Computer Use, notices and CI changes separate. No new confirmed P0–P2 defect was found in this source comparison.
The old-head run 41270071454 failed in notices.mjs with the stale source inventory; its bridge suite passed 66 tests and its selected keyboard check passed one test before that failure. This was not a completed Rust workspace-test result. The updated base includes the inventory fix and replacement component verifier; the current head's CI remains in progress and must be judged separately.
I did not execute Rust tests, SIGKILL/restart recovery, Windows, the preview cursor branch, CPU measurements or native UI acceptance. The source-only boundaries and P3 attribution observations in my prior review remain. No approval or merge is implied.
Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.
The reviewed exact head
|
Summary
Two independent Host defects, one commit each. Both are regressions from recent refactors that left an old assumption behind.
A session stops taking messages after its Host dies mid-Turn. After a SIGKILL, every session that was running stays busy for good and each send returns
agent_admission_failed. A SIGTERM mid-Turn was fine, since the Host settles its Turns before exiting.9e2a192491replaced the process Host and dropped itsrecovercall, soopen_hostandapplication serveno longer settle the previous owner's work. Onlymaka codestill called it.RootStore::openalready recovered effects and auxiliary models but then handed out an execution store with invocations and shell runs unsettled, leaving each entry point to remember the rest.Invocation and shell recovery now run inside
RootStore::open, before maintenance starts, and themaka_host_runtime::recoverywrapper and its second effect and auxiliary pass are gone.maka codeopens throughmaka_transport::open_storelike the other entry points, so it also waits for unconfirmed cleanup. Parked approvals keep their existing recovery path. The store error behindagent_admission_failedis now logged.Viewing a streaming session costs the Host about 26% of a core.
d1404ea800moved streamed text to an in-memory preview but left its wake in thewatchloop. Each wake reread queue, interactions, computer apps, session and an event page, about 30 root validations per pass, and emitting a preview added another pass.watchnow reads the store only on its first pass, on a commit notice, or while committed events remain undelivered. Preview wakes and preview pages touch memory only. Commit notices still take priority between preview pages.Does not change: the per-job authority check in
reader_worker, the wire pages, or other watch loops. An emptyeventspage no longer follows each preview.Verification
Release build, one window viewing one session, test model, Host CPU as a share of one core:
SIGKILL with 4 Turns running, then a new Host:
agent_admission_failedStartup on a 1,024-session Root: ready in 0.67-0.78 s with nothing abandoned, and 2.43 s after a SIGKILL with all 1,024 Turns running, with busy 0 afterwards.
Tests:
candidate::crash::killed_host_leaves_an_ended_turn_and_a_session_that_takes_messagesstarts a real Host throughconnect_root, kills it mid-Turn, restarts, checks the first Turn ended as failed, and completes a new message. It fails with the recovery call removed.cargo nextest runonmaka-event-log,maka-agent-engine,maka-agent-nativeandmaka-host-runtime: all pass.maka-cliintegration: all pass except two that are not from this change.host_serve_rejects_overlong_system_temp_and_releases_root_and_socket_directoryfails the same way on the base commit here.configuration_pages_manage_real_host_skills_and_mcp_without_creating_conversationswas already known to fail on the base; I did not rerun it there.cargo fmt --check, and clippy with-D warningson the touched crates.Two existing tests changed because they relied on the old behaviour:
startup_recoverynow sees recovery already done on reopen, and the Always grant test checks a grant on a new invocation, since the pre-restart one is now correctly ended.Not run or not covered:
watchCPU cost or thethrough < committedbranch. A test for that branch passed with it removed, becausesubscribe_changesclones a receiver that reports a change at once, so a new watch always gets a second pass, and I dropped the test. The branch keeps the previous behaviour of rereading after a fenced first page, and matters later in a watch's life.cfg(unix). Fulljust checkwas not run.Review focus
RootStore::opennow also settles abandoned work for non-Host writers such asinstalland offline update, and a recovery failure stops them opening the Root. They already ran effect and auxiliary recovery there and hold the exclusive owner, so I kept one rule rather than adding a second opening mode.watchloop, particularly preview pagination while commit notices arrive.AI use
Select exactly one:
Tool(s) and scope: diagnosis, implementation, tests and measurements.
Checklist
watchcost no, see Verification)Does this PR entail a change in behavior?