Repository navigation
Stop reporting stale compressed hashes as damage - #2
Merged
Merged
Conversation
git runs `git maintenance run --auto --detach` after every commit. Once the loose-object count passes gc.auto, each commit starts its own detached full repack, and nothing stops a second one from starting while the first is still running. An archive's pack is mostly zstd or git-crypt blobs, so each repack holds gigabytes. One ingest tick makes one commit per changed session, which stacked repacks until a 16 GB Mac with a 7.3 GiB archive ran out of memory and had to be force-restarted. Set maintenance.auto=false and gc.auto=0 in the archive's local git config. init.py sets them after `git init`. ingest.py sets them under the ingest lock before the first commit, which migrates existing archives on their next tick. Loose objects now wait for a manual `git repack -d`. The selftest checks both the init path and the migration path.
On a real archive 1,832 compressed deposits failed the sweep fsck, yet every one decompressed to its canonical sha256. The v2.3.0 pure migration copied sha256_compressed from the prior manifest instead of hashing the file, so stale values survived, and the fsck compares only the compressed hash. About 40 false failures per sweep slice buried any real one. Hash the file during a pure migration. In the fsck, fall back to the canonical hash when the compressed one disagrees and count a match as stale, not failed. Add scripts/repair_compressed_hashes.py to rewrite the stale values in one commit, touching only that field and only after the canonical hash matches.
This was referenced Sep 17, 2026
PEEKPerformer
changed the base branch from
fix/disable-auto-maintenance
to
main
September 17, 2026 12:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1. Review that one first; this PR's diff is only the second commit.
What this PR does
The rolling-sweep fsck no longer calls a deposit damaged when only its recorded
sha256_compressedis stale and the content still matches the canonicalsha256. A newscripts/repair_compressed_hashes.pyrewrites the stale values, and pure migrations stop carrying them forward.Why
On a real archive, 1,832 of 5,315 compressed deposits fail the fsck with
ARCHIVE INTEGRITY FAILURE. Every one decompresses to exactly its canonicalsha256, so the content is intact. All 1,832 were deposited on 2026-05-23 by v2.1.0 and migrated by v2.3.0.The v2.3.0 pure-migration path copies
sha256_compressedfrom the prior manifest instead of hashing the file, so a stale value survives. The fsck compares only the compressed hash. Each sweep slice therefore reported about 40 false failures, enough to hide a real one..zstfile on disk.stale-hash, not a failure. Real damage is reported as before.sha256_compressedstring, only after the canonical hash matches, in onerepair:commit. It takes the ingest lock.--dry-runreports without writing. Damaged deposits are listed, never rewritten, and make it exit 1.Type of change
Non-negotiables check
Append-only, for discussion: the repair script rewrites a manifest in place without a
manifest_versionbump. It changes one derived field (the hash of the file as stored) and only when the canonical hash proves the content is unchanged. Transcripts and the ledger, which chains canonicalsha256values only, are untouched. The history keeps both values in arepair:commit. The alternative is bumpingMANIFEST_VERSION, which migrates every manifest and has to read every source transcript, including ones iCloud has evicted.Selftest
python scripts/selftest.pypasses locallyNew step: a migration over a stale hash records the real one, the fsck reports a stale hash as stale, the repair commits the real value, and a truly damaged
.zstis still a failure that the repair refuses to touch.Manifest / archive-format impact
manifest_versionbumped from N to N+1index_schema_versionbumped from N to N+1Documentation
CHANGELOG.mdupdated (under the next-release section, 2.4.1)README.mdupdated if user-facing behavior changedSKILL.mdupdated if wizard flow changeddocs/updated as appropriateCross-platform impact
AI assistance
Diagnosis, fix, and draft written by Claude Code, from the sweep logs and a full pass over the real archive's manifests. Selftest run locally.
Anything else reviewers should know
I haven't found the exact event that made the v5 manifests stale in the first place. The archive is a shallow clone that starts after May, so git history can't show it. This PR fixes the symptom and stops migrations from spreading it.