[nissa-seru]

A workspace file whose stored contents the store has lost, or can't read, is now read from disk's copy when that copy has exactly those bytes. The contents are stored again, and whoever runs the host is told. Where nothing can restore them, the answer says so and what the agent can do instead.

Problem

Each file a chronicle mount holds is a tree entry that names its contents by their SHA-256. Chronicle keeps the contents as a blob file under that hash. Measured on main (0a4a2c69) with the blob's file deleted and the store opened again, while disk still held exactly the same bytes:

• read, edit and readBinary, which the framework's own read_image route calls, answered "Blob not found for: work/a.txt". edit_many answered "…stored none of its edits: 1 problem. edit 1: Blob not found for: work/a.txt".
• grep found nothing in the file and listed it as "the workspace copy is missing from the store, so it was not searched".
• sync synced nothing, and materialize, plain or forced, materialized nothing. Disk's copy agrees with the entry by hash, so neither stores anything, and a read after the sync still failed. ls showed the file synced.
• read_image answered from disk's file. With a different image on disk, it served that one with nothing saying it wasn't the workspace's copy.
• Nothing told whoever runs the host.

A blob file chronicle can't read whole, or whose format, checksum or hash doesn't check out, makes getBlob throw (get in chronicle's src/blobs/storage.rs). By reading, chronicle 0.4.0's storeBlob writes to the blob's final path and takes any file already there as stored, so a write cut short leaves such a file. With one cut short by 6 bytes, read answered "IO error: failed to fill whole buffer". grep over the mount failed the whole call, though another file matched. Writing the same text again changed nothing. By reading, the same throw fails a whole materialize that pushes the file, and a listing that reads a branch intent whose blob is cut short.

By reading, nothing in chronicle 0.4.0 or agent-framework deletes a blob file: chronicle's BlobStorage::delete has no caller and no binding. A blob file goes missing when something outside them removes or loses it. A blob deleted while the process runs stays readable from chronicle's cache of recently used blobs (1,000 unless the store is opened with another size), so the loss shows after a restart or an eviction.

Changes

• One restore (contentsOf). A read of a version's contents reads the store first (storedBytes, which tells chronicle's three answers apart). Where the store has no file for them, or one chronicle can't read, it reads disk's copy within the mount's boundary: openContainedFile, with the mount's own rule for symbolic links. It takes the copy only when its size and SHA-256 are the version's, stores it again with storeBlob, reads the store again to see whether that took, and answers with the bytes. Where disk can't supply them, it reads the store once more before reporting them lost, since another read may have restored them meanwhile. Disk has no copy only where sync would take its deletion: a disk read that finds nothing (ENOENT or ENOTDIR) asks confirmAbsent, the proof of absence sync uses, and anything it doesn't prove, such as a path beneath a link that leads nowhere, is a copy disk can't read, with that proof's reason. A restore or a loss sends an ops alert through the module context's notifyOps, with agent workspace, a message that opens with the blob's hash, and data mount, path and blobHash, plus storeError where chronicle couldn't read the blob:

  • workspace-contents-restored: stored again.
  • workspace-contents-unstored: answered from disk's copy, but the store still doesn't give the contents, with unstored saying why. Chronicle keeps a blob file it can't read in place, so where the store can't read the file, the alert says that if the file is damaged, removing it lets the next read store them.
  • workspace-contents-lost: no copy with their hash could be read, saying what disk holds and that putting back the store's blob file for that hash, from a copy of the store, brings them back.

  failures.log and the ops:alert trace get every alert, so each read that answers from disk because the store won't keep the contents, or finds them lost, adds one. The webhook posts at most once per kind every 15 minutes, as for every framework alert.

• read, readBinary and read_image's read of the workspace's copy restore in place. The await comes after the entry is read, and the answer is that entry's bytes.

• Several files at once (restoreEach, for edit, edit_many and grep, which restores each mount's files in that mount's turn). Lost versions are restored by hash: each hash from the first of its files whose disk copy has it, and every file holding it then takes those bytes. Whoever runs the host is told a hash is lost, once for each of its files, only when none of the files the call reads in that mount restored it and the store still lacks it. So a file whose own disk copy differs raises no workspace-contents-lost when another file restores the contents, and a hash the store won't keep is read and alerted once.

• edit and edit_many read each named file's stored contents after their pass. Where the store has lost some, they restore them, then run the pass again, and the plan reads that second pass and both passes' changes taken in from disk. Each file's entry is read in the plan and set with nothing awaited between, as before, and the bytes the plan uses are that entry's own, by hash: those read before planning, restored or stored, so the edits plan from disk's copy even where the store couldn't keep it. The plan reads the store again only for a version it has no bytes for. A file whose contents turn out lost after the second pass, because its entry changed meanwhile, is refused, and the call can be made again.

• grep searches each readable text inside its synchronous loop and keeps only the answer. A lost version is held in its place: the workspace's version, or a conflict's recorded disk version. After the loop the lost versions are restored by hash, and each is searched as soon as its bytes are in hand, keeping only the answer. The answers are then recorded in path order, results, skips and the CRLF and BOM notes alike, so contents two files share are searched in both, whichever file restored them. Each restored version has the version's own hash, so it's exactly what the pass labelled. One blob chronicle can't read no longer fails the whole call.

• materialize's push (pushPaths) reads the workspace's copy through storedBytes. A copy chronicle can't read is listed as skipped, "the workspace copy is unreadable in the store (…)", and the other paths go ahead. A missing one is skipped as on main. A push restores nothing from disk.

• readIntent reads a branch intent through storedBytes, so an intent whose blob chronicle can't read gives its "unreadable" reason, as a missing one does, and the listing answers.

• Where nothing can restore them, the answer says what the store and disk hold, what didn't happen, and what the agent can do. Each state's words below were checked by a test that takes the remedy and reads the file after:

  The store	Disk	read answers
  missing	other contents (a lost draft, or a conflict)	"The workspace's copy of work/a.txt is missing from the store, and disk's copy holds other contents, so it can't be restored or read. Sync the path to keep disk's copy and give up the workspace's, or write the file again."
  missing	no copy	"…and disk has no copy, so it can't be restored or read. Write the file again, or delete it."
  missing	a copy it can't read	"…and disk's copy can't be read (Symlinks are not allowed: work/a.txt), so it can't be restored or read. Write the file again, or delete it."
  can't read	other contents	"The workspace's copy of work/a.txt is unreadable in the store (IO error: failed to fill whole buffer), and disk's copy holds other contents, so it can't be restored or read. Sync the path to keep disk's copy and give up the workspace's."
  can't read	no copy	"…and disk has no copy, so it can't be restored or read. Delete the file to clear it."
  missing, on a read-only mount	a copy it can't read	"…so it can't be restored or read. Once disk's copy can be read, read the file again, which reads it from disk's copy if that copy has the same contents, or sync the path to keep disk's copy and give up the workspace's."
  missing, on a mount configured direct that runs as chronicle	no copy	"…so it can't be restored or read. Sync the path to take disk's deletion, which removes the workspace's copy."

  The last two are for a mount that takes no writes or deletions (writeRefusal), so they name a sync. edit says "…so it can't be restored, and the edit changed nothing." before the remedy, and edit_many names the edits. grep lists the version as not searched, saying what the store and disk hold. readBinary answers as read does, and so does read_image where disk has no copy. Where disk's copy holds other contents that can't be shown, read_image says why, in disk's own words. A copy behind a link that leads nowhere, at the path or on its way, is one disk can't read, not one disk lacks, since sync can't take a deletion there. Where the store can't read the blob, "write the file again" is left out, since writing the same bytes leaves chronicle's file as it is.

• read_image serving disk's copy of a file whose workspace copy is lost adds a line to its text: "The workspace's copy of work/p.png is missing from the store, so this is disk's copy, which holds other contents."

What each part is for

• storedBytes, in sync.ts so that reconcile.ts and branch-intent.ts can use it, tells chronicle's three answers apart: the bytes, null (no file), and a throw (a file it can't read whole, or whose format, checksum or hash doesn't check out). Before, a throw failed the whole call. A closed store's throw is passed on, since it isn't about the blob.
• LostContents and its text helpers (lostText, lostInStore, lostOnDisk, nothingToRead, lostRemedy) put each lost answer's words in one place for the five sites that give them. Its writable picks a remedy the mount can take.
• contentsOf:
  • its first read of the store answers a version that's there, or that a sibling read restored a moment before, and its read before reporting a loss catches one restored meanwhile and gives the loss the store's latest state;
  • diskLacks asks confirmAbsent before calling disk's copy absent, since the remedy for "no copy" is sync's deletion, which sync takes only where that proof holds. It asks in the view a disk read checks its root against (rootIdentityForRead's identity), since it's part of a read and records nothing, and a root the read couldn't reach is a copy disk can't read, in the read's own words. CHANGED is exported from observe.ts so that a link at the path itself, which the proof finds present, keeps disk's own error as its reason;
  • its size check skips reading a copy of another size, and its hash check is what makes disk's copy the contents;
  • the store read after storeBlob is how it knows whether the contents took, since storeBlob reports success for a file it kept and can throw after writing one;
  • the store read just before storeBlob keeps a second restore of the same contents, such as read_image's outside the mount's turn, from storing and alerting twice;
  • what names a conflict's recorded disk version in the alert, which grep restores too;
  • the three alerts tell whoever runs the host the three outcomes, which need different responses. Each message opens with the blob's hash, because the webhook carries no data and keeps only the head and tail of a message over 500 characters;
  • tell lets a caller restoring several files hold the loss alert until every file holding the hash has been tried. tellLost sends it, and contentsSubject and inTheStore word all three alerts.
• restoreEach restores lost versions by hash for edit, edit_many and grep, so the loss alert is true of every file the call reads in a mount, and a hash is read from disk and stored once, and told restored or unstored once, however many files hold it.
• readImageFromTree is tryReadImageFromTree made async, since it restores, and returning the lost contents where it threw blob_missing. That code is removed from WorkspaceImageReadError: nothing throws it now.
• edit's reads before planning: the contents map holds each version's bytes, so the plan reads none twice. The second pass is what the plan then reads, so a restore that let disk or the branch move can't leave the plan reading a stale pass. ops keeps the first pass's changes for the receipt's notes. The lost map is keyed by path, since two files can share lost contents and hold different things on disk.
• grep's search, split from its recording, keeps memory to each file's matches, restored files' included, while the recordings wait for the lost versions in path order.
• unstoredCopy words pushPaths' two skips.
• imageResult's note carries read_image's line about disk's copy. read_image's lost answer for any refusal of disk's copy tells the agent the workspace's copy is lost, which disk's refusal alone didn't, and gives disk's reason once.
• edit's which-copy and conflict notes go in only once a file's bytes are read: a lost file's conflict note offered to force a copy that can't be read onto disk.
• grep's answers map holds each lost version's answer, its matches or a skip, until the recording reaches it in path order.
• The changelog fragment says what changed for a host's operators and agents.
• BranchIntent.unreadable's doc says a put replaces a damaged record, except where it puts the same contents over a blob file chronicle can't read, which chronicle keeps.
• The test file's Env opens a module on a real store, and its restartAfter, lose and damage stop the module, change blob files as a restart would find them, and open both again. synced and draft build the two states a file is lost from. In workspace-three-way.test.ts, conflictWithBothVersionsLost builds a conflict whose two stored versions are both lost.

Tests

Requirements, each with the test that pins it, in test/workspace-lost-contents.test.ts unless named. Each test in that file loses or damages a blob for real: it stops the module, closes the store, deletes or truncates the blob's file, and opens both again.

• Disk holds the same bytes:
  • read answers with them, the store holds them again, one workspace-contents-restored alert names the mount, path and hash, and a second read raises none ("read answers with them…").
  • edit and edit_many make their edits to them, and materialize writes the edited text ("edit and edit_many make their edits…").
  • grep searches them in path order among other files ("grep searches them…"), and searches contents two files share once either restores them, in both path orders, with no workspace-contents-lost ("grep searches contents two files share…"). edit_many edits two such files, with no loss alert either ("edit_many edits files that share contents…").
  • read_image and readBinary answer with them ("read_image and readBinary…").
  • Two reads restoring the same contents at once both answer, with one alert ("two reads restoring the same contents at once…").
  • Contents the store took though its write then threw are told as restored ("contents the store took…"), and a write that fails is told as unstored, with the store's error, while the read answers from disk ("a store write that fails…"). A write that leaves a file chronicle can't read and then throws is told with the store's error, the store's state after the write and the damaged-file advice, and removing that file lets the next read store the contents ("a store write that leaves a file it can't read…"). Each alert's message opens with the blob's hash.
  • edit_many that restores one file still says what it took in from disk for another ("edit_many that restores one file…").
• The boundary holds: a copy disk holds only through a link the mount doesn't follow isn't taken or stored, and writing the file again replaces it ("a copy disk holds only through a link…"). Disk bytes of the version's size but another hash aren't taken ("disk bytes of the same size…"). On a mount that follows links, a link leading nowhere, at the path or on its way, is a copy disk can't read, not one it lacks ("on a mount that follows links…"). On a read-only mount the answer says what a read or a sync does once disk's copy can be read, and a read then restores the contents ("on a read-only mount…").
• Nothing can restore them: each state's words and remedy, as in the table ("a lost draft…", twice; "a conflict whose workspace copy is lost…", with edit's refusal; "disk has no copy…"; "edit_many refuses…"; "edit_many tells each file's own state…" and "grep tells each file's own state…", with one alert for each file naming its own disk state; "grep lists it as not searched…"; "on a mount configured direct…"). read_image serves disk's other image with its line, says the copy is lost with no file on disk ("read_image serves disk's other image…"), and where disk's file isn't an image, with disk's reason ("read_image says the workspace's copy is lost…"). Where disk's copy can't be read, it gives disk's reason once ("read_image gives disk's reason once…"), and while the disk-agreement evidence awaits reopening after a failed append, it still tells the loss, since it checks the root as the read does ("read_image tells a loss while…"). The lost alert's advice works: putting back the blob file brings the contents back ("whoever runs the host is told that putting back…").
• The store can't read the blob:
  • A read is answered from disk's matching copy, the alert is workspace-contents-unstored, and grep finds both files' matches ("a read is answered from disk's matching copy…").
  • edit and edit_many plan from disk's copy and store their change ("edit and edit_many plan from disk's matching copy…").
  • The unstored alert's advice works: removing the blob file lets the next read store the contents ("whoever runs the host is told that removing…").
  • Files sharing such contents, one with other contents on disk and one with the matching copy, are all answered from that copy by grep and edit_many, in both path orders, with one workspace-contents-unstored and no loss ("files sharing contents whose blob the store can't read are all answered…"). Where both hold the matching copy, it's read from disk and told once ("…each with the matching copy on disk…").
  • With other contents on disk, and with none, the words and remedies ("with other contents on disk…", "with no copy on disk…").
  • materialize skips that path, saying why, and writes the rest ("materialize lists a draft whose blob it can't read…"). A forced materialize over disk's newer copy skips it too, leaving disk's copy ("a forced materialize over disk's newer copy…").
  • A closed store's error is passed on, not read as a blob the store can't read ("a store closed while a read restores contents…").
  • A branch intent whose blob can't be read leaves its path unverified with the reason, and the listing answers ("a branch intent whose blob the store can't read…").
• grep, a conflict's versions (test/workspace-three-way.test.ts): a conflict's recorded disk version, lost while disk still holds it, is searched, and with disk's file gone, both versions are listed as not searched with what the store and disk hold. These replace one test that pinned the old skip.
• edit_many (test/workspace-edits.test.ts): the test of a file whose stored copy is missing, which stands in for getBlob, now gives that file a draft disk doesn't hold, so nothing can restore it, and pins the new words.

Runs, all on Linux:

• npm test at this head, Node 24.22.1: 7,613 tests, 7,569 pass, 0 fail, 44 skipped. CI's run of main (0a4a2c69) on ubuntu and Node 24 counted 7,572 tests, 7,527 pass and 45 skipped. The 41 more tests are this change's 40 new ones and the grep test split in two. The one fewer skip is "under Bun, a failed dial's error leaves no credential…", which CI's job skips because Bun isn't on its PATH.
• The three test files this change touches, unforced, on Node 24.22.1 and 22.12.0: 497 of 497 pass on each.
• On main's code with this head's tests, all 40 new tests fail, each at main's answer: "Blob not found for:", the old skip, chronicle's error failing the call, no alert, or no line about disk's copy. The three changed tests fail there too, at main's answers.

Other open pull requests

Of the 46 open pull requests, one conflicts with this head where it merges cleanly with main: #624, in one line, index.ts's import from observe.js, to which both add names. #271 already conflicts with main in src/modules/workspace/index.ts and src/modules/workspace/sync.ts, and this change alters main's side of four of those conflicts: index.ts's imports from sync.js and observe.js, handleReadImage's read of the workspace's copy, handleGrep's search, and the end of sync.ts, where both add code.

Not verified

• Every run is on Linux, with Node 24.22.1 and 22.12.0.
• A blob file that is whole but fails chronicle's format, checksum or hash check takes the same path as a truncated one, by reading chronicle's get; the tests only cut one short.
• The ops webhook's post itself: the tests stand in for notifyOps, and its posting and cooldown are the framework's, unchanged.
• A blob file lost to a power loss: chronicle doesn't fsync a blob's directory after creating its file, which by reading could let a power loss keep the entry and lose the file. No run here tested that.

Out of scope

• sync and materialize of a file whose disk copy agrees with its lost contents by hash do nothing, as on main: there's nothing to take in or push. A read then restores it.
• materialize of a file whose workspace copy is lost and disk holds other contents: disk's copy isn't the contents, so there's nothing to push, and it's listed as skipped as on main.
• ls and status, which don't read contents: once reads restore them, a file's state is true of disk and agreement as before.
• Chronicle's storeBlob, which keeps a file it can't read in place: that is chronicle's code.

Who did what

I built the change. Ada-1017 (Claude Opus 5.5) made the design calls: a restore checked by hash on every read of stored contents, edit's restore before planning, grep's after its loop, the words where nothing can restore them, a blob chronicle can't read treated as a lost one, a loss told only when no file holding the contents restores them, and "no copy" judged by sync's own proof of absence. Naomi-2422 (Claude Opus 5.5), who built none of it, read each head and approved this one. Three internal reviews read it, the last at the head it opened with, and this head answers their findings and those of the automated review at opening.

---

• Changelog fragment added: changelog.d/workspace-restore-lost-contents.fixed.md

Written by Pippa-2571 (Claude Opus 5.5).

Description checked and posted by Lottie-2661 (Claude Opus 5.5).

🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared nissa-seru account.

[github-actions]

Automated review

Recommendation: merge and fast-follow. 3 medium severity findings can be fixed after the merge.

Reviewed commit 9c5f4f9 with claude-opus-5-5, by reading the code. It did not run the code or look at the checks.

Does this change belong in the project?

It fits the project. This bug fix belongs in the workspace module, which already owns workspace storage and syncing with disk. When the store loses a file's contents, or can't read them, the module now restores them from disk's copy if that copy has exactly the same bytes, and it reports the restore or loss through the existing ops-alert hook.

5 reasons with sources

• It fixes existing workspace tools that failed when the store lost a file's contents; it adds no new feature. The README describes WorkspaceModule as a mountable filesystem abstraction with Chronicle tree state, filesystem watching and materialization. Before this change, read, edit, edit_many, grep, readBinary, read_image and materialize answered "Blob not found", or failed the whole call, when a blob was missing or unreadable. The change keeps these tools working on top of what the module already does. (README.md (WorkspaceModule entry))
• Every change stays inside src/modules/workspace, and one shared helper replaces the scattered store reads. The new storedBytes helper sits in sync.ts next to hashContent and replaces the direct store.getBlob calls in index.ts, reconcile.ts and branch-intent.ts. Restoring from disk goes through the module's existing openContainedFile, so it follows the mount's own boundary and symbolic-link rules. (src/modules/workspace/sync.ts, reconcile.ts, branch-intent.ts)
• Alerts go through the framework's existing ops-alert hook, so the change adds no new channel. The workspace module already calls ctx.notifyOps (for example 'workspace-mount-not-direct'). The three new alert kinds use the same optional call, so maintainers take on no new interface or dependency. (src/types/module.ts (ModuleContext.notifyOps); src/modules/workspace/index.ts:1718)
• The PR follows the contributing rules: it adds a changelog fragment and tests. It adds changelog.d/workspace-restore-lost-contents.fixed.md, a new test file test/workspace-lost-contents.test.ts, and updates to existing tests whose expected messages changed. The other review's notes on missing test cases and false 'lost' alerts are code defects for that review and don't affect whether the change belongs. (CONTRIBUTING.md (What a PR should contain; Changelog))
• It is one coherent change: every part handles blobs the store has lost or can't read. The changes to materialize's skip reason and to reading branch intents use the same storedBytes helper to handle unreadable blobs. They are part of the same fix, not unrelated work. (CONTRIBUTING.md (How changes land: keep commits coherent))

Findings

3 findings of medium or high severity, see the inline comments:

• Medium severity. grep sends a false 'workspace-contents-lost' alert when a later file with the same hash restores the contents.
• Medium severity. edit_many sends a false 'lost' alert for shared contents that a later target restores.
• Medium severity, test coverage. No test for the edit/edit_many refusal when an entry changed during the restore.

2 more findings, less severe

• Low severity, test coverage, src/modules/workspace/index.ts:4025-4028. No test where disk's copy has the right size but a different hash. contentsOf takes disk's copy only when its size matches and hashContent(read.bytes) === version.blobHash. In every 'other contents' test the disk text has a different length, so the otherSize early throw rejects it and the hash comparison never decides anything. If the hash check were dropped or compared the wrong value, the different bytes would be stored under the old hash and served, and no test would fail. Add a case where disk holds different bytes of the same length.
• Low severity, test coverage, src/modules/workspace/reconcile.ts:1390-1404. Forced push of a stale copy whose blob is unreadable has no test. pushPaths now reads through storedBytes in two places. The new materialize test only reaches the ordinary write branch (around line 1555), using a draft whose blob is damaged. The forced 'stale copy' branch (v.adopt with opts.force) also changed: it used to throw on an unreadable blob and now skips the path with unstoredCopy's reason. No test covers that branch, so a regression there, such as going back to plain getBlob, would not be caught.

For a deeper review by several readers and a judge, which takes longer, a maintainer can comment /review deep.

[github-actions]

Medium severity. grep sends a false 'workspace-contents-lost' alert when a later file with the same hash restores the contents.

The restore loop calls contentsOf for each lost item in path order, and contentsOf sends workspace-contents-lost as soon as one path's disk copy doesn't match. If a later path holds the same blob hash and its disk copy matches, that path restores the contents and workspace-contents-restored follows. The operator has already been told the contents are gone and to put the blob file back from a backup copy of the store, which isn't true. The new test 'grep searches contents two files share once either restores them' asserts this lost-then-restored sequence. Because the webhook posts at most once per kind every 15 minutes, the false 'lost' post also takes that kind's slot. Run every restore attempt for a hash before deciding whether to alert that it is lost, or group the lost items by hash first.

[github-actions]

Medium severity. edit_many sends a false 'lost' alert for shared contents that a later target restores.

The toRecover loop calls contentsOf once for each target, in order. When target A's disk copy differs and target B shares A's blob hash and has a matching disk copy, contentsOf for A sends workspace-contents-lost, then B restores the contents and they go into contents. A is then planned from those bytes and the edit succeeds. So the call works, but the host operator has been told the contents are unrecoverable and that the blob must be restored from a backup. Try restoring each hash from every target that holds it before sending a loss alert.

[github-actions]

Medium severity, test coverage. No test for the edit/edit_many refusal when an entry changed during the restore.

The new else branch handles an entry whose blob hash changed between the restore and the second pass and is again not in the store. It answers "…changed while … restored lost contents from disk… Make the same call again." No test reaches it: every new edit test has found.blobHash === entry.blobHash. The template text also passes lostInStore({ store: stored }) where stored is a narrowed StoredBytes. A test that changes the entry during the await (for example, a concurrent write or a disk change the second pass takes in, onto an unstored blob) should cover it.