Stop reporting stale compressed hashes as damage

CHANGELOG.md

@@ -2,6 +2,41 @@
 
 All notable changes to `holotype`. Format adapted from [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.4.1] - 2026-09-17
+
+Holotype now turns off git's automatic maintenance in the archive. On a large archive it could exhaust the machine's memory.
+
+### The failure
+
+git runs `git maintenance run --auto --detach` after every commit. When the loose-object count passes `gc.auto`, that starts a detached full repack (`git repack --cruft` and `git pack-objects --all`). Nothing stops a second repack from starting while the first is still running: each commit starts its own. One ingest tick makes one commit per changed session, so a busy tick starts several repacks at once.
+
+An archive's pack is mostly zstd or git-crypt blobs, which do not delta or compress. Every repack therefore reads the whole pack and holds gigabytes while it runs. On a real archive (7.3 GiB pack, 16 GB Mac) the repacks stacked up over one tick to 63 `git` processes and about 44 GB of memory. They never finished, the pack was never rewritten, the loose-object count stayed over the threshold, and every later commit started another repack. The machine ran out of memory and had to be force-restarted.
+
+Reproduced on git 2.54 with a scratch repository: four commits in a row left five concurrent `pack-objects --all` processes.
+
+### The fix
+
+- New `holotype.archive.disable_auto_maintenance()` sets `maintenance.auto=false` and `gc.auto=0` in the archive's local `.git/config`. It writes only when a value differs.
+- `init.py` calls it right after `git init`.
+- `ingest.py` calls it under the ingest lock, before the tick's first commit. Existing archives are migrated on their next tick with no action from you.
+- No archive-format change. Nothing tracked in the archive changes.
+
+Loose objects now accumulate until you pack them. To pack only the loose objects, without rewriting the existing large pack, run `git -C <archive> repack -d`.
+
+### Stale compressed hashes no longer read as integrity failures
+
+On a real archive, 1,832 of 5,315 compressed deposits failed the rolling-sweep fsck with `ARCHIVE INTEGRITY FAILURE` ("stored bytes do not match manifest hash"). Every one of them decompressed to exactly the canonical `sha256`. The content was intact; only `sha256_compressed` was wrong. All 1,832 were deposited on the same day by v2.1.0 and migrated by v2.3.0.
+
+The v2.3.0 pure-migration path copied `sha256_compressed` from the prior manifest instead of hashing the file, so a stale value survived the migration. The sweep checks only the compressed hash, so each sweep slice reported about 40 false failures, and the real signal was lost in them.
+
+- The pure-migration path now hashes the `.zst` file on disk.
+- The fsck decompresses a deposit whose compressed hash disagrees. If the canonical `sha256` matches, it counts the deposit as `stale-hash`, not as an integrity failure. Real damage is still reported as before.
+- New `scripts/repair_compressed_hashes.py` rewrites stale values in one `repair:` commit. It changes only the `sha256_compressed` string, and only after the canonical hash matches, so transcripts and the ledger are untouched. `--dry-run` reports without writing. Damaged deposits are listed and never rewritten.
+
+### Tests
+
+The selftest checks that `init.py` sets both keys, and that `ingest.py` restores them on an archive where they are unset. A new step covers stale compressed hashes: the migration rehashes, the fsck reports stale instead of failed, the repair commits, and real damage is still caught.
+
 ## [2.4.0] — 2026-05-28
 
 The archive's "hash chain" is now a real chain. Until this release, "hash chain" named an aspiration: each `manifest.json` carried the SHA-256 of its own transcript, and `verify.py` checked each file against its *own* manifest. That proves per-file content integrity but binds nothing across deposits — a coordinated edit of a transcript *and* its manifest is internally consistent and passed `verify.py` clean. The only thing recording the set and order of deposits was git's commit DAG, which a self-contained `paper_bundle` `.tar.gz` (shipped with no `.git`) doesn't carry. So a reviewer holding a bundle could confirm self-consistency but not that nothing had been inserted, deleted, or substituted.

holotype/__init__.py

@@ -1,3 +1,3 @@
 """holotype: forensic-grade archival of agent-CLI sessions for scientific reproducibility."""
 
-__version__ = "2.4.0"
+__version__ = "2.4.1"

holotype/archive.py

@@ -12,6 +12,7 @@
 from __future__ import annotations
 
 import json
+import subprocess
 from pathlib import Path
 from typing import Iterator
 
@@ -62,3 +63,41 @@ def resolve_session_by_prefix(archive: Path, prefix: str) -> Path | None:
             sys.stderr.write(f"  {m.name}\n")
         return None
     return matches[0]
+
+
+# git runs `git maintenance run --auto --detach` after every commit. Once
+# the loose-object count crosses gc.auto, each commit starts its own
+# detached full repack (`pack-objects --all`), and nothing stops a second
+# one from starting while the first is still running. An archive's pack
+# is mostly incompressible blobs (zstd or git-crypt), so each repack holds
+# gigabytes and runs for a long time. One ingest tick makes one commit
+# per changed session, which is enough to stack repacks until the machine
+# runs out of memory. Holotype turns automatic maintenance off in the
+# archive; repack by hand with `git repack -d` when loose objects pile up.
+AUTO_MAINTENANCE_OFF = (
+    ("maintenance.auto", "false"),
+    ("gc.auto", "0"),
+)
+
+
+def disable_auto_maintenance(archive: Path) -> None:
+    """Turn off git's automatic background maintenance for ``archive``.
+
+    Writes to the archive's local ``.git/config`` only, and only when a
+    value differs, so calling this every ingest tick is cheap. Failures
+    are ignored: a missing setting costs memory, not correctness.
+    """
+    for key, value in AUTO_MAINTENANCE_OFF:
+        try:
+            current = subprocess.run(
+                ["git", "-C", str(archive), "config", "--local", "--get", key],
+                capture_output=True, text=True, timeout=10,
+            )
+            if current.stdout.strip() == value:
+                continue
+            subprocess.run(
+                ["git", "-C", str(archive), "config", "--local", key, value],
+                capture_output=True, text=True, timeout=10,
+            )
+        except (subprocess.SubprocessError, OSError):
+            pass

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "holotype"
-version = "2.4.0"
+version = "2.4.1"
 description = "Forensic-grade archival of LLM-driven agent CLI sessions for scientific reproducibility"
 readme = "README.md"
 license = { file = "LICENSE" }

scripts/ingest.py

@@ -42,8 +42,15 @@
 sys.path.insert(0, str(REPO_ROOT))
 
 from holotype import __version__
+from holotype.archive import disable_auto_maintenance
 from holotype.chunking import bin_pack_paths, dir_size_bytes
-from holotype.compression import compress_bytes, find_transcript, transcript_filename
+from holotype.compression import (
+    compress_bytes,
+    decompress_bytes,
+    find_transcript,
+    transcript_filename,
+    zstd_available,
+)
 from holotype.env import claude_code_version, git_state_for_path, platform_info
 from holotype.hashing import sha256_bytes
 from holotype.index import (
@@ -264,15 +271,22 @@ def needs_processing(
 
 
 def verify_deposit_integrity(
-    archive: Path, candidate: DepositCandidate
+    archive: Path,
+    candidate: DepositCandidate,
+    stale_out: list[str] | None = None,
 ) -> str | None:
     """fsck one deposited transcript against its manifest hash.
 
     Hashes the on-disk deposit bytes as they sit (compressed bytes vs
-    ``sha256_compressed`` when zstd, raw bytes vs ``sha256`` otherwise) —
-    pure bit-rot detection that needs no zstd and never false-alarms.
+    ``sha256_compressed`` when zstd, raw bytes vs ``sha256`` otherwise).
     Returns an error string on mismatch/missing, else None. Run on the
     rolling-sweep slice so the whole archive is re-checked every ~24h.
+
+    When only the compressed hash disagrees, the file is decompressed and
+    checked against the canonical ``sha256``. If that matches, the content
+    is intact and only the recorded compressed hash is stale: the session
+    is appended to ``stale_out`` instead of being reported as a failure.
+    ``scripts/repair_compressed_hashes.py`` rewrites the stale values.
     """
     m = existing_manifest(archive, candidate)
     if not m:
@@ -290,6 +304,15 @@ def verify_deposit_integrity(
     except OSError as e:
         return f"{candidate.archive_subpath}: deposited transcript unreadable ({e})"
     if sha256_bytes(raw) != expected:
+        if is_compressed and m.get("sha256") and zstd_available():
+            try:
+                canonical = sha256_bytes(decompress_bytes(raw))
+            except Exception:
+                canonical = None
+            if canonical == m.get("sha256"):
+                if stale_out is not None:
+                    stale_out.append(candidate.archive_subpath)
+                return None
         return (
             f"{candidate.archive_subpath}: ARCHIVE INTEGRITY FAILURE — "
             f"stored bytes do not match manifest hash"
@@ -397,9 +420,11 @@ def deposit_one(
 
     if is_pure_migration and dest_transcript.exists():
         if compression == "zstd":
-            sha_compressed = prior_manifest.get("sha256_compressed") or sha256_bytes(
-                dest_transcript.read_bytes()
-            )
+            # Hash the file as it sits instead of copying the prior
+            # manifest's value. A copied value can be stale: the v2.3.0
+            # migration carried 1,832 stale compressed hashes forward on a
+            # real archive, and each one read as an integrity failure.
+            sha_compressed = sha256_bytes(dest_transcript.read_bytes())
         else:
             sha_compressed = None
     else:
@@ -902,6 +927,11 @@ def main(argv: list[str] | None = None) -> int:
             print("holotype: another ingest is running, exiting")
         return 0
 
+    # Before the first commit of this tick, so no commit can start a
+    # detached repack. Also migrates archives created before this setting.
+    if not args.dry_run:
+        disable_auto_maintenance(archive)
+
     try:
         counts: dict[str, int] = {"new": 0, "updated": 0, "skipped-live": 0,
                                   "skipped-unchanged": 0, "skipped-empty": 0}
@@ -1244,9 +1274,10 @@ def main(argv: list[str] | None = None) -> int:
         # Failures are reported loudly and sent to stderr — they indicate
         # the deposited artifact no longer matches what was recorded.
         integrity_failures: list[str] = []
+        stale_compressed: list[str] = []
         if not args.dry_run:
             for _cls, cand in sweep_candidates:
-                err = verify_deposit_integrity(archive, cand)
+                err = verify_deposit_integrity(archive, cand, stale_compressed)
                 if err:
                     integrity_failures.append(err)
             advance_sweep_cursor(archive, sweep_bucket)
@@ -1257,6 +1288,12 @@ def main(argv: list[str] | None = None) -> int:
             )
             for f in integrity_failures:
                 sys.stderr.write(f"  {f}\n")
+        if stale_compressed and not args.quiet:
+            print(
+                f"holotype: {len(stale_compressed)} deposit(s) in this sweep have a "
+                f"stale sha256_compressed (content verified intact). Fix with "
+                f"`python scripts/repair_compressed_hashes.py`."
+            )
 
         any_changes = counts["new"] + counts["updated"] > 0
         if not args.quiet or any_changes or integrity_failures:
@@ -1267,6 +1304,7 @@ def main(argv: list[str] | None = None) -> int:
                 f" empty={counts['skipped-empty']}"
                 f" swept={len(sweep_candidates)}"
                 + (f" INTEGRITY-FAIL={len(integrity_failures)}" if integrity_failures else "")
+                + (f" stale-hash={len(stale_compressed)}" if stale_compressed else "")
             )
             if committed and not args.quiet:
                 print("\n".join(committed))

scripts/init.py

@@ -35,6 +35,7 @@
 
 sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
 from holotype import ledger  # noqa: E402
+from holotype.archive import disable_auto_maintenance  # noqa: E402
 
 ARCHIVE_FORMAT_VERSION = 1
 HOLOTYPE_SUBDIR = ".holotype"
@@ -525,6 +526,7 @@ def init_archive(args: argparse.Namespace) -> int:
     is_new_repo = not (archive / ".git").exists()
     if is_new_repo:
         run(["git", "init", "-b", "main"], cwd=archive)
+    disable_auto_maintenance(archive)
 
     if args.sign_commits:
         # Probe for a signing key. Don't fail init outright — the user