Skip to content

fix(runtime): recover from a killed Host and stop rereading the store for previews - #6519

Merged
Astro-Han merged 6 commits into
rust-previewfrom
fix/host-crash-recovery-watch-cost
Nov 1, 2026
Merged

Astro-Han merged 6 commits into
rust-previewfrom
fix/host-crash-recovery-watch-cost

Conversation

@Astro-Han

@Astro-Han Astro-Han commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two independent Host defects, one commit each. Both are regressions from recent refactors that left an old assumption behind.

A session stops taking messages after its Host dies mid-Turn. After a SIGKILL, every session that was running stays busy for good and each send returns agent_admission_failed. A SIGTERM mid-Turn was fine, since the Host settles its Turns before exiting.

9e2a192491 replaced the process Host and dropped its recover call, so open_host and application serve no longer settle the previous owner's work. Only maka code still called it. RootStore::open already recovered effects and auxiliary models but then handed out an execution store with invocations and shell runs unsettled, leaving each entry point to remember the rest.

Invocation and shell recovery now run inside RootStore::open, before maintenance starts, and the maka_host_runtime::recovery wrapper and its second effect and auxiliary pass are gone. maka code opens through maka_transport::open_store like the other entry points, so it also waits for unconfirmed cleanup. Parked approvals keep their existing recovery path. The store error behind agent_admission_failed is now logged.

Viewing a streaming session costs the Host about 26% of a core. d1404ea800 moved streamed text to an in-memory preview but left its wake in the watch loop. Each wake reread queue, interactions, computer apps, session and an event page, about 30 root validations per pass, and emitting a preview added another pass.

watch now reads the store only on its first pass, on a commit notice, or while committed events remain undelivered. Preview wakes and preview pages touch memory only. Commit notices still take priority between preview pages.

Does not change: the per-job authority check in reader_worker, the wire pages, or other watch loops. An empty events page no longer follows each preview.

Verification

Release build, one window viewing one session, test model, Host CPU as a share of one core:

Scenario Before After
30 increments/s, window 25.6-27.1% 0.7-0.8%
1 increment/s, window 1.2-1.4% 0.1-0.2%
Idle session, window 0-0.2% 0-0.1%
30 increments/s, no window 0.3% 0.3-0.4%

SIGKILL with 4 Turns running, then a new Host:

Busy after restart Send
Before 4 agent_admission_failed
After 0 succeeds; busy 0 afterwards

Startup on a 1,024-session Root: ready in 0.67-0.78 s with nothing abandoned, and 2.43 s after a SIGKILL with all 1,024 Turns running, with busy 0 afterwards.

Tests:

  • New candidate::crash::killed_host_leaves_an_ended_turn_and_a_session_that_takes_messages starts a real Host through connect_root, kills it mid-Turn, restarts, checks the first Turn ended as failed, and completes a new message. It fails with the recovery call removed.
  • cargo nextest run on maka-event-log, maka-agent-engine, maka-agent-native and maka-host-runtime: all pass.
  • maka-cli integration: all pass except two that are not from this change. host_serve_rejects_overlong_system_temp_and_releases_root_and_socket_directory fails the same way on the base commit here. configuration_pages_manage_real_host_skills_and_mcp_without_creating_conversations was already known to fail on the base; I did not rerun it there.
  • cargo fmt --check, and clippy with -D warnings on the touched crates.

Two existing tests changed because they relied on the old behaviour: startup_recovery now sees recovery already done on reopen, and the Always grant test checks a grant on a new invocation, since the pre-restart one is now correctly ended.

Not run or not covered:

  • No automated test guards the watch CPU cost or the through < committed branch. A test for that branch passed with it removed, because subscribe_changes clones a receiver that reports a change at once, so a new watch always gets a second pass, and I dropped the test. The branch keeps the previous behaviour of rereading after a fenced first page, and matters later in a watch's life.
  • The streaming text path is covered by the real-Host terminal tests. I did not visually check a desktop window after the change.
  • Windows was not run, and the new test is cfg(unix). Full just check was not run.

Review focus

  • RootStore::open now also settles abandoned work for non-Host writers such as install and offline update, and a recovery failure stops them opening the Root. They already ran effect and auxiliary recovery there and hold the exclusive owner, so I kept one rule rather than adding a second opening mode.
  • Ordering between previews and commits in the watch loop, particularly preview pagination while commit notices arrive.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: diagnosis, implementation, tests and measurements.

Checklist

  • Tests cover the change and fail without it (recovery yes, watch cost no, see Verification)
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

Replacing the process Host in 9e2a192 dropped its recovery call, so
`host serve`, candidate Hosts and `application serve` left a killed
Host's Turns open. Sessions then stayed busy for good and every send
returned `agent_admission_failed`.

Move invocation and shell recovery into `RootStore::open`, beside the
effect and auxiliary recovery it already ran, so no entry point can
receive an execution store with work abandoned by the previous owner.
This removes the second recovery pass and the host-runtime wrapper, and
`maka code` now opens through `maka_transport::open_store` like the
other entry points, which waits for unconfirmed cleanup.

Log the store error behind `agent_admission_failed`, which hid the cause.
d1404ea moved streamed text to an in-memory preview but left its
wake in the watch loop, which reread queue, interactions, computer apps,
session and events on each one, plus once more after emitting. A viewed
stream at 30 increments per second cost the Host about 26% of a core.

Read the store only for the first pass, a commit notice, or committed
events not delivered yet. A cut's dependency can still fence the event
page below the committed head, so that remainder keeps being read
without waiting for another notice.

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.

Reviewed the complete 13-file diff at 7df7d56b820baf50cb6dacc7e9721298a635f0cd and traced the Root startup recovery barrier and observation loop in source. I found no confirmed P0–P2 issue in this source review.

Invocation and shell recovery now occur while RootStore::open still owns startup, after the existing creation/checkpoint/effect recovery and before maintenance begins or the store is returned. The recovery APIs retain their original cleanup barriers, and opening errors still take the store shutdown path. The CLI code entry now uses maka_transport::open_store rather than a second recovery wrapper. The new crash test exercises kill/reopen/new admission in its source, but I did not execute it.

The watch loop reads durable state for its first pass, a commit notice or undelivered committed events. Preview-only wakes no longer force those reads. I checked the fenced-page through < committed condition, the preview-ready continuation and the biased selection that handles cancellation and commit notices before another immediate preview pass. An actual merge calculation against Rust dfc556ebf50ee5f6cfd71ddfa2d1c097faf27fa6 is clean.

This machine has no Rust toolchain. I did not execute recovery, kill/restart, concurrency or native Windows tests, force a failed startup cleanup, measure CPU or reproduce the reported 1,024-session results. In particular, source inspection alone does not establish preview/commit fairness under load or the performance improvement. Those remain validation limits; the contributor's test and measurement results are not my independent results. CI is still running.

P3: AI contribution is declared, but the tool is not named and the two reviewed commits lack the required Generated-by: <tool> trailer. Please complete that attribution and preserve it when squashing.


Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.

@github-actions github-actions Bot added the effort/M Under 500 readable lines label Nov 1, 2026
#6517 changed two notice inputs under packages/plugin-sdk without
regenerating the manifest, so the notices test fails on rust-preview and
stops `just check` before the Rust suites run.
…x/host-crash-recovery-watch-cost

# Conflicts:
#	products/maka/cli/notices.json

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.

I rechecked the current 13-file diff at 61f13397a701990cd8cd9337619262d6acc80ff7 against its updated Rust base d513a72e7ebdc11842442341bbf680deac886835.

The author patch has the same stable patch ID as reviewed 7df7d56b820baf50cb6dacc7e9721298a635f0cd (f07a7469c2ca7b8133b4052e29938bf22dcf0862). All 12 retained changed files are byte-identical; platform/host-runtime/src/recovery.rs is absent in both. I reread the current recovery/watch diff, keeping incoming storage, Computer Use, notices and CI changes separate. No new confirmed P0–P2 defect was found in this source comparison.

The old-head run 41270071454 failed in notices.mjs with the stale source inventory; its bridge suite passed 66 tests and its selected keyboard check passed one test before that failure. This was not a completed Rust workspace-test result. The updated base includes the inventory fix and replacement component verifier; the current head's CI remains in progress and must be judged separately.

I did not execute Rust tests, SIGKILL/restart recovery, Windows, the preview cursor branch, CPU measurements or native UI acceptance. The source-only boundaries and P3 attribution observations in my prior review remain. No approval or merge is implied.


Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.

@Astro-Han
Astro-Han marked this pull request as ready for review November 1, 2026 07:54
@Astro-Han
Astro-Han merged commit 50687a7 into rust-preview Nov 1, 2026
9 checks passed
@Astro-Han
Astro-Han deleted the fix/host-crash-recovery-watch-cost branch November 1, 2026 07:54
@Astro-Han

Copy link
Copy Markdown
Contributor Author

Automated review notice: This comment was posted by an automated review agent. It is not an independent human review and does not replace one.

The reviewed exact head 61f13397a701990cd8cd9337619262d6acc80ff7 now has successful CI 41278967847 and was merged as 50687a74. This closes the pending-CI state in my preceding review. It is hosted evidence; I still have not independently executed SIGKILL/restart recovery, preview CPU measurements, Windows or native UI acceptance. The new merged-Rust CI runs are separate and remain in progress.


Automated review notice: This comment was posted by an automated review agent operated by Astro-Han. It is not an independent human review and does not replace one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/M Under 500 readable lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant