Conversation
Pin the emulator, the paired ctsi-2 kernel and guest tools 0.18.0, and rebuild the canonical images. v0.21 stores are not loadable by v0.20 and vice versa, so every image and watchdog checkpoint is regenerated. Vendor the cartesi-tools-rs crates (libcmt-sys, trolley, testsi, types) under cartesi-tools/ instead of pinning a git rev: testsi moves to the v0.21 Rust bindings, trolley follows the libcmt 0.18 renames, and libcmt-sys builds the vendored libcmt 0.18 sources without downloading kernel headers. CI installs the pinned emulator for canonical-test. The watchdog's Lua host loop does not run on v0.21 (it feeds inputs without a revert root hash); its rewrite follows in this branch.
docs/cartesi-machine.md owns the emulator behavior the canonical guest, watchdog and recovery depend on: all-or-nothing versions, the reference host semantics (revert on reject, sticky fixed points, inspect on a discarded snapshot), sharing modes and durable publication, clone fallback and measured costs, memory ranges and labels, hashing, and the CLI's traps as a test oracle. The watchdog v2 plan records the decisions: generic state sources (inspect or a labeled memory range), in-process execution with the reference snapshot semantics, SHA-256 digest comparison, a fetch-first tick, and a divergence latch with incident tooling, runbook and drills.
GET /finalized_state/digest hashes the file GET /finalized_state streams, off the async runtime and under the same finalized lease, so the block and digest describe one checkpoint. The watchdog compares digests and downloads the bytes only as divergence evidence: application states are expected to reach several GiB, and transferring them on every accepted checkpoint is waste.
A small Cartesi Machine guest keeps its state in a 64 KiB NVRAM labeled `state` and, by payload, accepts, rejects (after scribbling over its state), raises a CMIO exception, halts with exit code 7, or emits a report; the inspect query `state` returns the whole NVRAM. The watchdog's real-machine tests run it against the reference `cartesi-machine` CLI. The image carries musl stand-ins for two guest-tools binaries that are glibc-only: the `nvram` label helper the CLI's init lines call, and `xhalt`, without which a non-zero entrypoint exit halts with payload 0.
…arison Rewrite the watchdog around the design in docs/plans/2026-09-watchdog-v2.md. - Canonical execution runs in-process with the reference rollup host semantics: a per-input clone-backed snapshot of an in-place working machine, a restore (checked against the revert root hash) on RX_REJECTED, and a permanent fixed point on exception, halt, any other manual yield, or cycle overflow. v0.21 refuses the old loop's feeding after a reject, and the old loop inspected the machine it then stored. - The application state comes from an init-time state source: the inspect query `state`, or a whole user-labeled flash drive or NVRAM. - Ticks compare SHA-256 digests. The target block comes from the sequencer's digest before replay, so a long catch-up cannot chase a moving target. - The L1 reader proves completeness: contiguous InputBox indices from the head's input count, a getNumberOfInputs witness pinned at the target, the configured chain in every payload, and a safe head at the target. - Checkpoints are machines published with sync_stored/rename_stored and named by block and input count; the head is the newest, with no pointer file. - A divergence latches with evidence until `clear`; `status` and `replay` support the incident runbook. init checks a genesis bootstrap against the on-chain template hash. - Failures are classed (transient, operator, internal) instead of matched by string; there are no in-tick retries. status.prom adds a heartbeat and the checked block. - Directory work uses a vendored LuaFileSystem instead of shelling out. BREAKING CHANGE: config.json is version 2 and CM v0.21 stores are not loadable by v0.20; re-initialize every watchdog state directory. init requires CARTESI_WATCHDOG_STATE_SOURCE and an L1 RPC.
- A regression is relative to a block the sequencer agreed with. Before the first agreement the head is only the bootstrap machine, so a sequencer behind it (a genesis bootstrap above the sequencer's block-0 genesis) makes the tick idle instead of latching a false regression. - Derive the InputBox from the application's getDataAvailability(), as the sequencer does, and drop CARTESI_WATCHDOG_CONTRACTS_INPUT_BOX_ADDRESS: another release's InputBox would present an empty history as complete. - `replay` checks its --from machine like init's bootstrap (safe block, input count, template hash when no input precedes it), requires absolute paths and a fresh --out, and cleans up after a failure. - Prune superseded checkpoints with an idempotent tree removal; the emulator's remove_stored cannot resume an interrupted removal. - Latch as soon as the local evidence is durable and fetch the sequencer's bytes afterwards; treat a torn marker as latched; keep exit 2 while latched even when configuration or endpoints are broken; never let a failed record write change the exit code. - Keep JSON-RPC errors sent with an HTTP error status, so range limits still split, and record byte-identical evidence explicitly. - status.prom reports cartesi_watchdog_head_block: the bootstrap block is not a block the sequencer matched.
A mistyped CARTESI_WATCHDOG_STATE_DIR made `status` create an empty directory and report it uninitialized; `status` and `replay` now refuse a missing state directory. The failure-class comment names the classes as the code applies them: filesystem failures are internal, not transient.
…nt runbook - docs/watchdog/README.md owns the runtime contract: the tick, canonical execution, L1 completeness, state sources, commands, one configuration table, the state directory and its crash model, the latch, metrics and alerting (including the heartbeat), the detection boundary, and why it is built this way. - docs/watchdog/incident-runbook.md is the divergence playbook: what the latch records, stopping the sequencer, finding the faulty side with `replay`, resolving each kind, and the drills that practice it. - The operator and local guides link to those owners instead of copying them; design-notes, staging-drills, and the sepolia stub are folded in or removed. - The application contract owns the comparison bytes for both state sources; the C header and the Rust trait point at it. - docs/cartesi-machine.md records that a deployment is pinned to its emulator version and the guest-side findings from the test guest; cockroach recovery names the watchdog's evidence as a canonical source. - The v2 plan is reduced to its owners and remaining work.
The crates imported from cartesi-tools-rs are the canonical side of the SDK: an application author builds and tests the Cartesi Machine guest with them, as sdk/rust-client serves the client side. Move them from the root cartesi-tools/ to sdk/guest/, and rename the `types` crate to `rollups-types` so its name says what it holds. The guest binaries embed their source paths, so the machine images and their template hashes change; the e2e harness derives the hash from the built image.
A divergence wrote its evidence (the canonical machine and the whole comparison range) before the latch marker, so a full disk, an I/O error, or a kill in that window lost the latch: the tick exited 1, and a later agreeing tick advanced past the diverged block without operator clearance. Evidence errors after the marker also downgraded exit 2 to 1, and the synchronous download of the sequencer's file delayed the failed metric and the event. Latching now creates incident/ and writes the latch record, nothing else. The directory is the latch, so a record lost to a crash or a full disk still latches (as unreadable_marker), and discard_interrupted is gone. The tick returns its evidence as data; main writes the event, last_tick.json, and status.prom first, then collects the evidence, each item on its own, most perishable first: the sequencer's file, the comparison against the in-memory canonical bytes, the canonical machine, then canonical.bin. incident/evidence.json indexes each item as kept or missing with a reason, and its collection is running, finished, or, marked by the next latched tick, interrupted. A tick that cannot even create incident/ still exits 2 and says the divergence is not latched. Atomic writes and the evidence download now remove their temporary file on failure, and a relative metrics path is logged instead of ignored silently. The README, runbook, deployment guide, and cockroach recovery describe the new order, the evidence index, and what a graceful sequencer stop does to a download in flight.
…ality From the PR review's design remarks. The tick's peak of about twice the range comes from read_memory (a native buffer copied into a Lua string), not from hashing, and read_memory can read a range in chunks. So the fallback if memory binds is to stream the same flat SHA-256, which keeps the protocol, the sequencer (it already streams its file), and sha256sum on the evidence files; the two-level digest is dropped. It needs an incremental hasher, since the emulator exports only one-shot hashes. The application contract now asks for byte equality after the same inputs, including after from_dump or recovery partway, not only for equivalent logical state: for a range source, layout-affecting state belongs to the checkpoint. The C header and the Rust trait now say the same. The plan adds restart and recovery comparison tests for a range-source application, the matching DEX question, and the reflink requirement behind the per-input snapshot cost.
GCdePaula
force-pushed
the
feature/watchdog-state-inspect
branch
from
September 28, 2026 21:50
3a207a9 to
d31c86d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Reworks the watchdog so it can check any application, not only one with an inspect
statehandler. A deployment picks its state source atinit: the inspect querystate(the wallet) or a whole user-labeled flash drive or NVRAM (the DEX). Each tick replays the canonical machine in-process to the block of the sequencer's newGET /finalized_state/digestand compares SHA-256 digests; a mismatch latches until an operator clears it. Also bumps the Cartesi Machine to v0.21.0 and brings the guest crates into the repo assdk/guest/.What changed
--revert-mode=storedloop (a snapshot per input, restored on reject; a permanent stop on exception or halt). The CLI is the test oracle, driven by a new test guest.GET /finalized_state/digesthashes the comparison file under the same lease as its block. The watchdog downloads the file only as divergence evidence.getNumberOfInputswitness, the chain id in every payload, and a safe head at the target. The InputBox comes from the app'sgetDataAvailability().status,clear, andreplaycommands, an incident runbook, and drills in the test suites.libcmt-sys,trolley,testsi, androllups-typesfrom cartesi-tools-rs, updated for v0.21 (provenance insdk/guest/README.md).docs/watchdog/README.mdowns the contract,incident-runbook.mdthe playbook, anddocs/cartesi-machine.mdthe emulator facts; the plan keeps the remaining work.Compatibility
config.jsonv2 and v0.21 stores, so state directories must be re-initialized.initrequiresCARTESI_WATCHDOG_STATE_SOURCEand an L1 RPC;CARTESI_WATCHDOG_CONTRACTS_INPUT_BOX_ADDRESSis gone. There are no live deployments./finalized_state/digestis operator-internal.docs/plans/2026-09-watchdog-v2.md.Follow-ups
Staging drill, cost and memory measurements at DEX scale, restart and recovery comparison tests for a range-source app,
xhaltin the canonical image, and atrolleyexception call. See the plan.Testing
fmt, clippy, workspace tests, both canonical images,
canonical-test-guest, and the watchdog unit, real-machine, and rollups e2e suites (including the divergence drill) pass locally. The CI workflow changes run on GitHub for the first time here.