Repository navigation
Wait for a session to settle before updating it - #5
Merged
Merged
Conversation
git runs `git maintenance run --auto --detach` after every commit. Once the loose-object count passes gc.auto, each commit starts its own detached full repack, and nothing stops a second one from starting while the first is still running. An archive's pack is mostly zstd or git-crypt blobs, so each repack holds gigabytes. One ingest tick makes one commit per changed session, which stacked repacks until a 16 GB Mac with a 7.3 GiB archive ran out of memory and had to be force-restarted. Set maintenance.auto=false and gc.auto=0 in the archive's local git config. init.py sets them after `git init`. ingest.py sets them under the ingest lock before the first commit, which migrates existing archives on their next tick. Loose objects now wait for a manual `git repack -d`. The selftest checks both the init path and the migration path.
On a real archive 1,832 compressed deposits failed the sweep fsck, yet every one decompressed to its canonical sha256. The v2.3.0 pure migration copied sha256_compressed from the prior manifest instead of hashing the file, so stale values survived, and the fsck compares only the compressed hash. About 40 false failures per sweep slice buried any real one. Hash the file during a pure migration. In the fsck, fall back to the canonical hash when the compressed one disagrees and count a match as stale, not failed. Add scripts/repair_compressed_hashes.py to rewrite the stale values in one commit, touching only that field and only after the canonical hash matches.
The preferred Claude Code source on the maintainer's machine is a backup mirror in ~/Documents, which iCloud syncs. iCloud had evicted 3,556 of its 11,311 transcripts to the cloud. The launchd tick may not download an evicted file, so every read failed with EDEADLK and each sweep logged dozens of "Resource deadlock avoided" worker failures. Dedup now prefers a later path whose copy is on disk over an evicted first path. A deposit whose source is still evicted returns skipped-offline and leaves the prior deposit in place.
git push ran with no timeout. A push that stalls rather than fails holds the ingest lock, so every later tick exits early and nothing is deposited until someone kills it by hand. Run pushes in their own process group with a one-hour limit and kill the whole group on timeout, so ssh and pack-objects exit too. A timed out push is reported like any other push failure.
A session that is still being written changes on almost every tick, and each update stores the whole transcript again: an encrypted, zstd blob gets no delta against its last version. On a real archive one 26 MB session was stored 129 times in two weeks, and history grew by 4.7 GB in 15 days. Defer a content update to an existing deposit until its source has been quiet for deposit.settle_hours (default 6), capped at 24 h after the last deposit. New sessions and pure migrations never wait. --settle-hours overrides the config; 0 deposits every change now. Replaying the archive's last 15 days, a 6 h window cuts history growth from 4.7 GB to about 0.9 GB.
13 of 28 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #4 (#1, #2, #3, #4). This PR's diff is only its own commit.
What this PR does
A content update to an already-deposited session now waits until the source transcript has been quiet for
deposit.settle_hours(default 6). The wait is capped at 24 hours after the last deposit. New sessions and pure manifest migrations never wait.--settle-hoursoverrides the config for one run, and--settle-hours 0deposits every change now.Why
A session that is still being written changes on almost every 30-minute tick, and each update stores the whole transcript again. A git-crypt-encrypted zstd blob gets no delta against its previous version, so every update is a full copy.
On a real archive:
Replaying those 15 days of commit timestamps and blob sizes:
6 hours captures most of the saving and still records a working session several times a day. The 24-hour cap means a session that never goes quiet is still updated daily.
Type of change
Behavior change for existing archives: updates are deferred by default. Version bumped to 2.5.0.
Non-negotiables check
Forensic completeness: every byte of the final transcript is still deposited. What changes is how many intermediate snapshots of a running session are recorded.
Selftest
python scripts/selftest.pypasses locallyThe update step now checks that a one-minute-old change reports
settling=1, and that--settle-hours 0deposits it. Unit assertions onshould_defer_updatecover the window, the 24-hour cap, and 0 disabling it.Manifest / archive-format impact
manifest_versionbumped from N to N+1index_schema_versionbumped from N to N+1config.jsongainsdeposit.settle_hours. Archives without it use the default.Documentation
CHANGELOG.mdupdated (2.5.0)README.mdupdated if user-facing behavior changedSKILL.mdupdated if wizard flow changed (non-negotiable 1 now explains the window and--settle-hours 0)docs/updated as appropriateCross-platform impact
AI assistance
Design agreed with the maintainer. Implementation, the history replay, and this draft by Claude Code. Selftest run locally.
Anything else reviewers should know
Sessions that are still settling are read and hashed each tick, as before. Only the write and commit are deferred.