Storage locker and execution boundary - #71
Conversation
A store now writes every M.2 slot and acks only when all of them hold the record. A load adopts only when the slots agree, restores from the survivor when one is missing (an M.2 swap), and otherwise discards, with one exception: every write carries a nonce drawn once per process, and slots that disagree under one nonce were torn by a single process, so the higher sequence number is that process's newest write. Adopting it is safe because a store that never returned had no effects. The locker is tenant-generic; the bookmark tenant maps a discard to a fresh identity rather than risk resuming a stale one. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Before each spawn, the launcher writes the boundary: the job this sled is about to run, its session, and the causal frontier of everything the sled had seen when it committed. When the job ends, its terminal status joins the record. Launches drain in release order, so the record never regresses, and a job whose record cannot be written is refused, with the refusal gossiped. At rejoin, the sled compares the recorded frontier with the join frontier. Covered means every request it had processed was witnessed: replay refuses old jobs, the session chain never releases a resubmitted one, and zombie reaping reports what the sled left running, so the session keeps working here. Not covered means a suffix of its history died with it: jobs may have run that no sled can name, and their signed artifacts could be resubmitted and run twice. The sled refuses jobs of the recorded session until a new session supersedes it, and adjudicates the recorded job itself: the recorded ending if the job got one, and interrupted if it did not. A recorded job still running across a universe swap is skipped, since its events land in the new universe when it finishes. A start event no longer displaces a terminal status, so an adjudicated interrupted survives a concurrent replayed start in either arrival order. State's relationship to its own past (join frontier, boundary, lost session, zombies) now lives in one struct, Past. The wiring layer owns storage: JobManager takes the Locker, and Seed::grow makes the seed together with the locker's one bookmark source, which spawn_gossip consumes as a pair. BookmarkSource::probe is gone; the seed probes the locker itself. Accepted for now: jobs in a lost suffix other than the recorded one get no verdict of their own, only a refusal when retried; and a crash between a job's end and the outcome write falls back to interrupted for that job. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
plaidfinch
left a comment
plaidfinch
left a comment
There was a problem hiding this comment.
I think this is definitely the right direction, but there are some devils in the details. I've tried to scrawl my thoughts in the margins, mostly intended as note-taking in advance to chatting with each other about it.
| shared: self.shared.clone(), | ||
| generation, | ||
| ratchet: self.ratchet.clone(), | ||
| generation: self.ratchet.generation.fetch_add(1, Ordering::SeqCst) + 1, |
There was a problem hiding this comment.
I don't understand why we need a bookmark generation number under the new approach in this PR. It seems like we shouldn't? What state transition relies on it?
| //! After a restart, compare the recorded frontier with the join | ||
| //! frontier. If the join frontier covers it, every request we had | ||
| //! processed was witnessed: replay refuses old jobs, the session | ||
| //! chain never releases a resubmitted one, and zombie reaping reports | ||
| //! what we left running. If not, a suffix of our history died with | ||
| //! us: jobs may have run here that no sled can name, and their signed | ||
| //! artifacts could be resubmitted and run again. The state machine | ||
| //! then refuses jobs of the recorded session until a new session | ||
| //! supersedes it, and adjudicates the recorded job itself: the | ||
| //! recorded ending if the job got one, and interrupted if it did not. |
There was a problem hiding this comment.
I think this comment is wrong, or, if it's accurately describing the implementation, I think the implementation is wrong. Specifically:
If not, a suffix of our history died with us: jobs may have run here that no sled can name, and their signed artifacts could be resubmitted and run again. The state machine then refuses jobs of the recorded session until a new session supersedes it.
We can learn about the history of messages gradually — it's not a guarantee that the first peer we connect to will tell us everything that every peer happens to know. So just because our first sync doesn't catch us up to our stored frontier, means nothing about whether more gossip will get us there. I also don't understand why this claims that "their signed artifacts could be resubmitted and run again" — no, they can't, because they'd be rejected? A resubmission of a signed job can't chain onto any existing job, so can't be re-executed. And if we're learning about an only-once-submitted job, but just late, we know whether or not to execute it, based on this very log.
It feels like this implies a lot more special-casing than is necessary.
| pub network: Network, | ||
| pub session: SessionId, | ||
| pub job: JobId, | ||
| #[serde(with = "version_bytes")] |
There was a problem hiding this comment.
Upcoming changes in Rumors should obviate this by making Version serialize itself as bytes meaningfully in human-readable and non-human-readable formats alike.
| pub session: SessionId, | ||
| pub job: JobId, | ||
| #[serde(with = "version_bytes")] | ||
| pub frontier: Version, |
There was a problem hiding this comment.
But why do we need this? I am skeptical. I will see whether I agree by the end of reading.
| /// Whether the rack already knows everything we knew at | ||
| /// commitment. Covered means no committed job can be lost. |
There was a problem hiding this comment.
I think this is false. There is no "the rack" as a singular entity that knows or doesn't know things. Just because you managed to gossip with another sled, does not mean that they will necessarily manage to propagate that information before dying.
| let bookmarks = BookmarkSource::new(log, locker); | ||
| let handle = match locker.probe().await { | ||
| Ok(()) => bookmarks.next_handle(), | ||
| Err(_) => bookmarks.shed_handle(), | ||
| }; | ||
| let rumors = match Peer::seed().bookmark(handle).await { | ||
| Ok(peer) => peer.into_rumors(), | ||
| Err(unbookmarked) => match unbookmarked.peer.bookmark(bookmarks.shed_handle()).await { | ||
| Ok(peer) => peer.into_rumors(), | ||
| Err(_) => unreachable!("a shed bookmark never touches storage"), | ||
| }, | ||
| }; |
There was a problem hiding this comment.
Okay, cool, I think this pattern makes sense. What I'm reading here is that we automatically discard bookmarks we fail to load, then try again, which means we don't permafail if a bookmark is torn or corrupted, we just (safely) leak identity space.
| job_id, | ||
| ProcessError::Io { | ||
| what: "consulting the execution boundary".to_string(), | ||
| error: "the store is untrusted; this sled needs service".to_string(), |
There was a problem hiding this comment.
NICE! Though, maybe we can make the error a little more specifically instructive, perhaps indicating potential failure of the M.2 directly, so the operator knows what this means.
| what: "consulting the execution boundary".to_string(), | ||
| error: "a restart lost part of this session's history on this sled; start a new session" | ||
| .to_string(), |
There was a problem hiding this comment.
I don't understand why or how this is relevant. We can figure out what ran based on the boundary, and we can conclude what got interrupted based on the output / exit log files. We shouldn't have to restart the session entirely just because one sled restarted and interrupted a job before gossiping; that is going to be a pain, and I think it's evitable.
| pub struct Past { | ||
| /// The causal frontier we joined this universe at, if we joined | ||
| /// rather than seeded it. | ||
| join_frontier: Option<Version>, |
There was a problem hiding this comment.
I am very concerned about the correctness of this. Is this just the causal version of whichever peer we gossip with first when we bootstrap? That peer could be arbitrarily causally related (ahead, behind, concurrent) to the operator's notion of the "current" state of the session at the time the sled connects, it just depends on which peer you talk to and how caught-up they are, right? If the intent is to prevent the operator from seeing past jobs in the session execute on newly installed sleds, isn't this already prevented by binding execution at job submission time to the sled baseboard ID? If a sled hasn't been adopted by the rack, it will never execute any jobs submitted prior to its adoption. Once it has been adopted, we want it to participate in "catch-up" behavior. So I think this is a red herring, and will lead to unexpected behavior.
| /// Sleds that witnessed a real terminal event keep it; the ruling | ||
| /// convinces only sleds that knew nothing, so verdicts can differ | ||
| /// across sleds. |
There was a problem hiding this comment.
I don't know what this means. One only adjudicates one's own local verdicts?
The set may never drop an entry, and an attacker must not be able to grow the file by inserting garbage, which rules out an exact set. The bit layout and the hash algorithm are on-disk format, pinned by tests. The filter errs only by calling a network burned when it is not; that error refuses a session, and never runs a job. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
A skip is a decision, not a failure. Without it, a job the boundary machinery has ruled out sits Queued forever on that sled; that hangs client waits and records no outcome. The status is terminal and carries a reason: the job's session sits below the sled's execution floor, or a previous life of the sled already handled it. The client renders a skip with its own row and reason. Nothing emits the status yet; the boundary rework that follows will emit it. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
U+26A0 (warning sign) is East Asian Width neutral, so wcwidth counts one column while the variation selector makes most emulators draw two, and aligned output drifts on those rows. U+2757 (exclamation mark) means the same thing and carries the wide property. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
The old record stored the causal frontier at commitment and compared it against the join frontier to decide whether history was lost. That comparison does not survive a crash: a restarted sled re-issues its dead predecessor's untransmitted versions as soon as it sends, so the recorded frontier can be re-covered by different messages and the lost-history verdict silently flips. This change removes the frontier from the record; the chain itself becomes the watermark. The record now stores the network, a burned set, the join of executed session starts, and the committed job. The commit also computes the successor's chain id from the signed bytes and stores it. The sled screens each session's start version against this record. The committed session resumes at its stored successor when it activates, so jobs at earlier chain positions drain without executing and a lost suffix no longer strands the session. A session whose start lies strictly above the executed join has never run here and is admitted. A start the record cannot order makes the sled hop: the sled reports a SessionHop error to the gossip set and raises its floor at the frontier that includes the report. The record joins every executed start rather than keeping only the last one, because a record that remembers only its last session cannot rank the session before it, and a third session could then replay the first session's jobs. The burned set holds every network whose watermark this record overwrote. It is a Bloom filter because entries may never be dropped, and a flood of forged networks must not grow the file. The launcher writes the burn and the advance together, so no crash window separates them. A sled that re-enters a burned universe reports a UniverseFlipFlop error and raises its floor the same way. Without the burn, returning to an earlier universe would overwrite the watermark that remembers the jobs it ran there, and those jobs could be resubmitted and re-executed. A sled that enters a universe with history while holding no record raises its floor at the frontier that includes its own announce, since it cannot order any session started before it arrived. Floors live in memory only. A floor's version is created by sending a message, and if the sled dies before that message reaches anyone, no other copy of it ever exists and no future session can dominate the floor; a persisted floor would then refuse every session in that universe until a cold boot. Every restart re-detects the burn, the missing record, or the unordered session and raises the floor again, so persisting the floor gains nothing. Jobs below the floor end as JobStatus::Skipped, and a new JobEvent carries the skip to the gossip set, so client waits resolve instead of hanging on a job the sled will never run. Arrival and birth marks replace the join frontier. The zombie check classified this sled's own job starts against the join frontier, which is one peer's view of history: a start from a previous life could sit above a lagging peer's frontier, be classified as this life's, and leave the rack believing the job still runs. The birth mark, the frontier just past this life's first send, separates the lives exactly: nothing a previous life sent dominates it, and everything this life sends does. The arrival mark, the frontier just before that first send, keeps the general replay line for foreign traffic, which can reach this sled before its sender has seen that first send. Both marks are computed locally and never persisted. They replace every use of Universe::frontier, so that field is deleted. This change also deletes the bookmark generation numbers. The record keeps every universe's identities, so a straggling write costs at most a stranded identity, and the locker's sequence guard already refuses stale writers. The gossip manager stops every session before handing a new peer its handle. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Both tenant records are now stored in a two-element CBOR envelope: the format's version number, then the body. A trait chains each version to the one it superseded, and the chain's Previous bound requires the conversion, so declaring a new version does not compile until the upgrade from the old one exists. Decoding matches the stored version against the chain, parses the body at that version, and converts the result up to the latest. The chain follows the ledger versioning in omicron's config-reconciler (see omicron#11249), with three changes. The version is stored explicitly instead of inferred from the body's shape, because two versions that differ only by an optional field encode identically. A record that fails to parse or convert is reported and left to the caller, so the boundary store stays untrusted and the bookmark assumes a fresh identity; omicron panics instead. Nothing is written back at load time; the first ordinary write persists the latest format. A Versioned ancestor carries the version number, and two policy traits carry the chain and its conversion. Record covers tenant records, and its conversion may fail: the caller can quarantine a bad record. Wire covers gossip messages, and its conversion is infallible: delivery is prefix-closed, so the state machine could neither skip a message that failed to convert nor stop at it. The VersionedMessage enum tag already serves as the wire envelope, so the message chain converts up when the state machine unwraps it; v0::Message is wire version 0. Version 0 is the shipped baseline for both tenants: the boundary record, and the bookmark record wrapping the bytes rumors writes. The latest format of each tenant is the live type itself, with no frozen copy. A pinned snapshot freezes its bytes instead, so a change to the live type fails the pin rather than silently changing version 0. The format module documents the steps a format change requires. A version newer than the software recognizes gets its own error, so a downgraded sled reports what happened instead of calling the record corrupt. An adversarial review of this machinery drove five hardening changes. Chain version numbers are checked at compile time: walk carries a const assertion that each version exceeds its predecessor, so a duplicate number cannot silently decode old records as the new format, and a decreasing one cannot refuse a known version as Future. The envelope refuses trailing elements instead of silently ignoring them. Schema snapshots freeze the shapes of the two signed types every sled rebuilds from the wire (the job request and the challenge response), because a field added under the house serde extension idiom would otherwise change what verifies, silently and only in a mixed rack. And the certificate and login-key wire fields now carry raw DER and OpenSSH bytes, parsed where used, so an artifact a future dependency refuses to parse costs one import or one login, never the gossip sessions that replay it. The Unknown message variant carries the whole received message re-encoded, so its Serialize cannot fail; rumors treats a message that fails to serialize as a panic. The wire bytes of all of these are unchanged; every pinned snapshot passes as before. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Fixes #76. A session can end while jobs still wait in its queue: stopped by its starter, superseded by a causally later start, or annihilated by a concurrent one. Until now the sled dropped the queue silently, and each waiting job stayed Queued forever. Now every queued job this sled would have run gets the terminal skip status with a new reason, SessionEnded, and jobs already started finish and report as usual. A job request arriving after its session ended gets the same skip in place of the old error status, so a job's fate does not depend on whether its request arrived before or after the end. Replayed requests report nothing, as with floor skips: re-reporting history on every rejoin would grow the message set. The new reason joins the pinned wire snapshots. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
We'll use a `sush` directory on the Omicron side. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
Match current omicron/main. Co-Authored-By: Claude Mythos 5 <noreply@anthropic.com>
2 participants
A store for sled state that must never be stale, and an execution boundary built on it.
The locker writes every M.2 slot and acks only when both hold the record. A load applies the pair rules:
A discarded bookmark means a fresh identity. A discarded boundary means the sled refuses to run jobs.
The boundary records, before each spawn, the job, its session, and the sled's causal frontier. When the job ends, the sled adds its terminal status to the record. After a crash, if the join frontier covers the recorded frontier, nothing was lost and the session keeps working here. If not, some jobs died unwitnessed: the sled refuses that session's jobs until a new session supersedes it, and adjudicates the recorded job, announcing its real ending if known and interrupted if not. A resubmitted signed artifact can never run twice on one sled.
Upgrade: the envelope format changed, so existing
SUSHBOOKMARKfiles will read as corrupt once, then the sled will assume a fresh identity. No manual cleanup.Unit tests cover the pair rules, torn writes, fencing, and outcomes. Integration tests cover lost and witnessed restarts and universe swaps.
Residue: only the recorded frontie gets a verdict; earlier lost jobs get refusals when retried. If the sled crashes between a job's end and the record write, the verdict falls back to interrupted.