Skip to content

Port #3763 to main: fix the second snapshot-based update on an agent - #3849

Merged
kmatasfp merged 4 commits into
mainfrom
port-3763-snapshot-update-baseline
Sep 11, 2026
Merged

Port #3763 to main: fix the second snapshot-based update on an agent#3849
kmatasfp merged 4 commits into
mainfrom
port-3763-snapshot-update-baseline

Conversation

@kmatasfp

@kmatasfp kmatasfp commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Ports #3763 to main.

main restructured PrivateDurableWorkerState::new to take a ready ReplayState, so the set_override being deleted lives in DurableWorkerCtx::create here rather than in the state constructor. Same block, one call site up.

All three hot_update tests are ported.

manual_update_on_idle_twice and manual_update_on_idle_to_earlier_component run on main's agent-updates-v1/v2 pair using the SnapshotUpdateTest agent, which stamps the build that wrote the snapshot into it.

auto_update_on_idle_after_manual_update needs a recorded invocation that cannot replay under a later build, so it brings over the agent-counters-v2 test component that 1.5.x got with the chaos suite in #3743. It is agent-counters with SnapshotCounter::component_version returning 2 instead of 1, and the two release builds differ in exactly one byte. agent-counters gains that method returning 1, and the new app joins rust_test_apps in build-components.sh.

Verified by reintroducing the bug on this branch: manual_update_on_idle_twice then fails with Timeout waiting for agent .../SnapshotUpdateTest() to reach component revision 2, at exactly the second manual update.

auto_update_on_idle_after_manual_update survives that revert. That is expected and its doc comment says so: it guards the reducer folding the snapshot region into the skipped regions, not the set_override deletion, and it goes red when that fold is removed. Both directions were run.

The eight hot_update tests matching *update_on_idle* pass on this branch after the main merge. The earlier full-module run (20 passed, 1 failed, snapshot_after_auto_update_recovers_with_updated_component_context failing identically on unmodified main) predates that merge and has not been repeated.

🤖 Generated with Claude Code

@kmatasfp
kmatasfp requested a review from a team September 10, 2026 01:17
@netlify

netlify Bot commented Sep 10, 2026

Copy link
Copy Markdown

Deploy Preview for golemcloud canceled.

Name Link
🔨 Latest commit d542f71
🔍 Latest deploy log https://app.netlify.com/projects/golemcloud/deploys/6aa316246475310008c81529

@vigoo

vigoo commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Why "not having a test component" is an excuse for not porting one of the tests? We can add new test agents / test components if needed

Comment thread golem-worker-executor/src/model/mod.rs Outdated
#[derive(Clone, Debug)]
pub struct AgentConfig {
pub deleted_regions: DeletedRegions,
/// The regions of the oplog that replay must skip: jumps, reverts, and the prefix

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unnecessary over-explaining comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in d542f71.

@vigoo

vigoo commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Found one issue that we did not find in the original PR:

Manual-snapshot download failure can abort the executor.
At durable_host/mod.rs:941–946, removing the constructor override leaves only the permanent region layer after a successful manual update. On restart, a manual-snapshot payload download failure still calls restart_replay_without_snapshot(). Its unconditional drop_override() removes that sole layer; the following region lookup panics at regions.rs:226. Release builds use panic = "abort", so a transient blob-store error can terminate the entire executor.

Update the failure path for the non-overridden representation without discarding the authoritative manual baseline. Add a regression test that injects a snapshot-download failure after a successful manual update and restart; the existing fail_next_oplog_download helper supports this.

@kmatasfp
kmatasfp enabled auto-merge (squash) September 10, 2026 17:06
@kmatasfp
kmatasfp disabled auto-merge September 10, 2026 17:06
@kmatasfp

Copy link
Copy Markdown
Contributor Author

Fixed in d542f71.

A manual-update snapshot that cannot be read (wrong entry at the index, or a payload download failure) now returns a new SnapshotRecoveryResult::BaselineUnavailable and the start attempt fails with that error. The skipped regions are left as they are, so the next start (a resume, or recovery) reads the snapshot again. The automatic path is unchanged. restart_replay_without_snapshot and drop_override_and_restart had no other caller, so they are gone.

Two regression tests in hot_update.rs cover both arms: manual_snapshot_download_failure_fails_the_start_and_keeps_the_baseline and manual_snapshot_invalid_entry_fails_the_start_and_keeps_the_baseline. The download one needs a snapshot that is stored outside the oplog, so agent-updates-v1/v2 gained an ExternalSnapshotUpdateTest agent whose snapshot is 128 KiB, above the default max_payload_size. With the fix backed out both tests hit the regions.rs:226 unwrap.

One thing I left alone: a failed start is held on the worker until it is resumed or unloaded, like every other instance-creation failure, so the tests resume before the second invoke.

#3763 on 1.5.x has the same drop, so this needs porting back.

@kmatasfp
kmatasfp merged commit 8500461 into main Sep 11, 2026
153 of 156 checks passed
@kmatasfp
kmatasfp deleted the port-3763-snapshot-update-baseline branch September 11, 2026 06:45
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 11, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants