Repository navigation
Add prune_local.py to drop old local history - #6
Merged
Merged
Conversation
git runs `git maintenance run --auto --detach` after every commit. Once the loose-object count passes gc.auto, each commit starts its own detached full repack, and nothing stops a second one from starting while the first is still running. An archive's pack is mostly zstd or git-crypt blobs, so each repack holds gigabytes. One ingest tick makes one commit per changed session, which stacked repacks until a 16 GB Mac with a 7.3 GiB archive ran out of memory and had to be force-restarted. Set maintenance.auto=false and gc.auto=0 in the archive's local git config. init.py sets them after `git init`. ingest.py sets them under the ingest lock before the first commit, which migrates existing archives on their next tick. Loose objects now wait for a manual `git repack -d`. The selftest checks both the init path and the migration path.
On a real archive 1,832 compressed deposits failed the sweep fsck, yet every one decompressed to its canonical sha256. The v2.3.0 pure migration copied sha256_compressed from the prior manifest instead of hashing the file, so stale values survived, and the fsck compares only the compressed hash. About 40 false failures per sweep slice buried any real one. Hash the file during a pure migration. In the fsck, fall back to the canonical hash when the compressed one disagrees and count a match as stale, not failed. Add scripts/repair_compressed_hashes.py to rewrite the stale values in one commit, touching only that field and only after the canonical hash matches.
The preferred Claude Code source on the maintainer's machine is a backup mirror in ~/Documents, which iCloud syncs. iCloud had evicted 3,556 of its 11,311 transcripts to the cloud. The launchd tick may not download an evicted file, so every read failed with EDEADLK and each sweep logged dozens of "Resource deadlock avoided" worker failures. Dedup now prefers a later path whose copy is on disk over an evicted first path. A deposit whose source is still evicted returns skipped-offline and leaves the prior deposit in place.
git push ran with no timeout. A push that stalls rather than fails holds the ingest lock, so every later tick exits early and nothing is deposited until someone kills it by hand. Run pushes in their own process group with a one-hour limit and kill the whole group on timeout, so ssh and pack-objects exit too. A timed out push is reported like any other push failure.
A session that is still being written changes on almost every tick, and each update stores the whole transcript again: an encrypted, zstd blob gets no delta against its last version. On a real archive one 26 MB session was stored 129 times in two weeks, and history grew by 4.7 GB in 15 days. Defer a content update to an existing deposit until its source has been quiet for deposit.settle_hours (default 6), capped at 24 h after the last deposit. New sessions and pure migrations never wait. --settle-hours overrides the config; 0 deposits every change now. Replaying the archive's last 15 days, a 6 h window cuts history growth from 4.7 GB to about 0.9 GB.
The remote holds the archive's full history, but every local clone keeps every superseded transcript version too, and encrypted zstd blobs get no deltas. On the maintainer's machine .git was 13 GiB. prune_local.py makes the local clone shallow, keeping the last --keep-days days (default 7), then expires reflogs, repacks without a delta search, and prunes. It deletes nothing unless a fresh fetch shows every local commit on origin, no other branch or stash holds unpushed work, and no ingest is running. Without --yes it only prints the plan. It is never run automatically. On an APFS clone of the real archive, keeping 2 days took 13.2 GiB of objects down to 8.1 GiB in 56 s, peaking at 2.6 GiB resident.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #5. This PR's diff is only its own commit.
What this PR does
Adds
scripts/prune_local.py, which makes the local archive clone shallow. It keeps commits from the last--keep-daysdays (default 7) and deletes older objects from disk. The remote keeps the full history. Without--yesit only prints the plan, and it is never run automatically.Why
Every local clone keeps every superseded transcript version, and encrypted zstd blobs get no deltas. On the maintainer's machine
.gitwas 13 GiB, next to 5.2 GiB of working tree and a 2.6 GiB index. The history is already on the remote. The maintainer shrank the clone by hand on 2026-09-02; this makes that safe and repeatable.Deletion only happens if every check passes:
originremote exists with the current branch on it;The steps are
fetch --shallow-since(falling back to--depth=1when the window holds no commits),reflog expire,repack -a -dwithpack.window=0,prune --expire=now, andfsck --connectivity-only.pack.window=0skips a delta search that can't find anything in encrypted, compressed blobs. Every git step has a one-hour timeout.Measured on an APFS clone of the real archive (850 local commits over 15 days, 13.2 GiB of objects):
--keep-daysWith 14 days the saving was small because almost all local history was younger than that. The default is 7.
Type of change
Non-negotiables check
The remote is never rewritten. Local history is dropped only when the remote already holds it. The fetch is explicit: the user runs the script.
Selftest
python scripts/selftest.pypasses locallyNew step, using the existing bare-repo remote:
--yesit only plans;--yesit removes a blob reachable only from 30- and 40-day-old commits;Manifest / archive-format impact
manifest_versionbumped from N to N+1index_schema_versionbumped from N to N+1Documentation
CHANGELOG.mdupdated (2.5.0)README.mdupdated if user-facing behavior changedSKILL.mdupdated (operations table)docs/updated as appropriateCross-platform impact
fcntl, likeingest.py)AI assistance
Design agreed with the maintainer. Implementation, measurements, and this draft by Claude Code. Selftest run locally.
Anything else reviewers should know
A local-only archive (no remote) is refused, because its history is the only copy. The working tree and the search index are out of scope:
cite,verify, and search depend on them.