Repository navigation
Skip iCloud-evicted source files instead of failing - #3
Merged
Merged
Conversation
git runs `git maintenance run --auto --detach` after every commit. Once the loose-object count passes gc.auto, each commit starts its own detached full repack, and nothing stops a second one from starting while the first is still running. An archive's pack is mostly zstd or git-crypt blobs, so each repack holds gigabytes. One ingest tick makes one commit per changed session, which stacked repacks until a 16 GB Mac with a 7.3 GiB archive ran out of memory and had to be force-restarted. Set maintenance.auto=false and gc.auto=0 in the archive's local git config. init.py sets them after `git init`. ingest.py sets them under the ingest lock before the first commit, which migrates existing archives on their next tick. Loose objects now wait for a manual `git repack -d`. The selftest checks both the init path and the migration path.
On a real archive 1,832 compressed deposits failed the sweep fsck, yet every one decompressed to its canonical sha256. The v2.3.0 pure migration copied sha256_compressed from the prior manifest instead of hashing the file, so stale values survived, and the fsck compares only the compressed hash. About 40 false failures per sweep slice buried any real one. Hash the file during a pure migration. In the fsck, fall back to the canonical hash when the compressed one disagrees and count a match as stale, not failed. Add scripts/repair_compressed_hashes.py to rewrite the stale values in one commit, touching only that field and only after the canonical hash matches.
The preferred Claude Code source on the maintainer's machine is a backup mirror in ~/Documents, which iCloud syncs. iCloud had evicted 3,556 of its 11,311 transcripts to the cloud. The launchd tick may not download an evicted file, so every read failed with EDEADLK and each sweep logged dozens of "Resource deadlock avoided" worker failures. Dedup now prefers a later path whose copy is on disk over an evicted first path. A deposit whose source is still evicted returns skipped-offline and leaves the prior deposit in place.
This was referenced Sep 17, 2026
PEEKPerformer
changed the base branch from
fix/stale-compressed-hash
to
main
September 17, 2026 12:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2 (which is stacked on #1). This PR's diff is only its own commit.
What this PR does
When a source transcript has been evicted to iCloud, ingest reads a mirrored copy that is still on disk. If there isn't one, it skips the session as
skipped-offlineinstead of logging a worker failure.Why
Every sweep logged dozens of
holotype: worker failed on ...: OSError: [Errno 11] Resource deadlock avoided, over 14,000 lines in the error log so far.The preferred Claude Code source was a backup mirror under
~/Documents, which iCloud syncs. With "Optimize Mac Storage" on, iCloud had evicted 3,556 of its 11,311 transcripts. Only a placeholder (SF_DATALESS) stays on disk. The launchd tick runs in the background and isn't allowed to download the file, so reading it fails withEDEADLK.~/.claude/projects(0 evicted candidates remain after this change).SF_DATALESSbefore reading and also catchesEDEADLK, returningskipped-offline. The prior deposit stands, and the session is read again once the file is local.offline=Nwhen any sessions were skipped.Type of change
Non-negotiables check
Holotype still never triggers an iCloud download itself.
Selftest
python scripts/selftest.pypasses locallyCI can't create an evicted file, so the new assertions patch
is_dataless. They check that dedup picks the mirror's on-disk copy over an evicted first path, and thatdeposit_onereturnsskipped-offlinefor an evicted source.Manifest / archive-format impact
manifest_versionbumped from N to N+1index_schema_versionbumped from N to N+1Documentation
CHANGELOG.mdupdated (under the next-release section, 2.4.1)README.mdupdated if user-facing behavior changedSKILL.mdupdated if wizard flow changeddocs/updated as appropriateCross-platform impact
st_flagsis absent there, sois_datalessis always false)AI assistance
Diagnosis, fix, and draft written by Claude Code, from the error log,
ls -lOon the failing sources, and a discovery run against the real config. Selftest run locally.Anything else reviewers should know
Out of scope: a session whose only copy is evicted and that was never deposited stays unarchived until the file is local again. It is counted in
offline=Nbut not named. No such session exists on the real archive today.