[Hafeok]

Milestone Extent maintenance, for #110. The descriptor proposes v0.1.0-preview.11: SHACL (#123) took preview.9 and the Management plane (#134) preview.10, and main is merged into this branch. ADRs 0158–0162, with dated amendments to 0134, 0138 and 0139, were accepted by the maintainer in 1dcbe2e; ADR 0164 gains a dated amendment for the wired hook. The record is docs/traceability/2026-10-19-issue-110-extent-maintenance.md.

What this changes, and why

Object storage landed extent sets without merges, so every read of the present merged one run per checkpoint, and read cost grew without bound.

• Background merges (ADR 0158).
  • Only the writer merges.
  • A merge writes the merged set, then swaps the catalog once under the epoch fence. It never blocks a commit or a checkpoint.
  • It rebases onto a catalog that a checkpoint or drop swapped meanwhile, or is abandoned if an input went.
  • A taken name means taking the next sequence.
  • Uploads are staged through a file in the cache and streamed in one create; the file becomes the cached copy. A merge whose output would pass the store's single-upload limit is refused before it starts (Azure 5,000 MiB, S3 5 GiB).
• The policy (ADR 0159).
  • Size-tiered: FanOut (4) adjacent sets of similar size merge, and past the bound the smallest adjacent run merges.
  • MaxSetBytes (256 MiB) caps a set. At the defaults a chain holds at most 15 sets below half the cap.
  • Configured by DatasetOptions.ExtentMerges / Varve:Store:ExtentMerges.
• Deletion by reader announcements (ADR 0160).
  • Every instance on an object store, the writer included, announces the interval [oldest, newest] of catalog versions it holds in readers/<id>, announcing then confirming.
  • The writer swaps, then reads the records, then deletes a set only when no record's interval overlaps the versions that named it.
  • The catalog version is the set sequence. Maintenance may poll.
  • Announcing is required. A 403 on the record fails the open and names the role. --unannounced reads without a record and fails a read of a gone extent with read-unannounced-extent-gone.
  • A forgotten reader fails with reader-forgotten and reattaches.
  • A record of an unknown format holds everything.
  • The reader record is specified as its own versioned format (storage format §8). A follower's write access is scoped to readers/ (ADR 0134 amended; Azure ABAC and an S3 prefix policy in the operator guide).
• The shared commit index (ADR 0161). Each set carries its window's commit index as an optional ninth blob in derived format 4, so a cold start reads no log below the catalog.
• Operations (ADR 0162).
  • varve storage, varve storage readers list and forget --reader --oldest --newest, guarded by what list showed.
  • The read commands attach to an object store.
  • /health/ready reports retention, and never turns unready because of it.
  • The operator guide's checkpoint budget is removed.

The reader contract with #105 is wired. #134 landed first, so this PR wires its hook (ADR 0164 point 6, amended 2026-10-20).

• A tier-1 follower opens through Dataset.OpenAsync, which announces the catalog version it loads in its reader record and confirms it.
• The no-op Hold call is removed, and Management plane, part 1: the fleet system dataset, grants as code, varve apply (v0.1.0-preview.10) #134's test is now a_follower_announces_the_catalog_version_it_opened.
• A writer and a follower of one dataset on one server keep separate records. With Varve:ObjectStorage:ReaderId set, the follower's record is that id with -follower; the setting is now 1 to 119 characters.
• A follower re-reads no catalog while it tails, so it retains what it opened from until it is opened again. retention reports those bytes.

What was measured

All figures are from this session's Linux container. Azurite and RustFS ran on loopback.

REC M on files, one run (--extent-maintenance M, fan-out 4, 256 MiB). The gate is relative within the run:

	10 checkpoints, no merges (10 sets)	100 checkpoints, merges (5 sets)	gate
lookup p50 / p99	59.0 / 123.8 µs	12.1 / 37.8 µs	≤ 1.5×: within
a subject's quads p50 / p99	9.0 / 40.9 µs	4.6 / 13.2 µs	within
exact count p50 / p99	37.7 / 86.7 µs	14.9 / 31.3 µs	within
as-of inside the largest set	2,358 ms (183 MiB)	2,736 ms (211 MiB)	≤ 2×: 1.16×

• The as-of read is compared like for like, the largest set of each chain. In absolute terms it is 2,736 ms at REC M, and it grows with MaxSetBytes, because narrowing is O(set size). O(matches) narrowing is a separate issue; MaxSetBytes is unchanged for it.
• Write amplification is 1.77 (746 MB rewritten for 420 MB checkpointed).
• Chain length: the chain held 3 to 10 sets across the run.
• For reference, Milestone Object storage — the log on any object store, Azure first #92's figures: lookup p50 42.0 / 98.7 µs at 10 / 100 checkpoints without merges.

First read below the catalog (--objects, M):

• Azurite 6.0 s and RustFS 3.5 s, with 233 reads each.
• Milestone Object storage — the log on any object store, Azure first #92's figures were 27.7 s and 10.1 s, from 18,977 chunk reads.
• Cold start is unchanged at 4.2 s and 6.4 s. It is still dominated by listing the open segment's chunks.

Tests on the rebased head (Linux):

• Totals:

  • Varve.Store.Tests: 320, 303 passed, 17 skipped (the real Azure account and the deep fault exploration, both env-gated), 24 min 42 s.
  • Varve.Server.Tests: 87 passed, 6 skipped (the identity providers), 11.7 s.
  • Varve.Protocol.Tests: 108 passed, 30.6 s.
  • Also run, as the projects that build on the store: Varve.Sparql.Store.Tests 25, Varve.Protocol.Client.Tests 117, Varve.Events.Tests 64 and Varve.Sparql.Evaluation.Tests 28, all passed.

• Iteration counts and durations:

  • Policy bound: 500 sequences, 0.2 s.
  • Merged projection against the model: 40 schedules (VARVE_EXTENT_SCHEDULES), 0.9 s.
  • Reader schedules: 60 (VARVE_READER_SCHEDULES), 5.2 s. They include readers holding several versions at once and a superseded writer that keeps creating, checkpointing and merging, none of whose sets is ever named or fails the new writer. 1,200 ran clean on demand.
  • Faults: 117 writes and 234 injection points, full comparison every 8th point (VARVE_EXTENT_FAULTS_FULL=1 for all), 37.7 s.
  • A follower across merges and deletions on Azurite (12.2 s) and RustFS (3.1 s).

• Added time: the new store tests add about 70 s on Linux. CI's Test Varve.Store step, in minutes:

  Run	Linux	Windows
  main at f7d25da (the base)	16.1	28.6
  this PR at 7b40629	16.5	35.5

  Windows grew by about 7 minutes against main, more than the Linux 70 s predicts; Linux is unchanged. Per-test attribution is CI: time limits, hang dumps and per-test durations for the store-test steps #126's: it adds a TRX upload for every test step, with each OS's twenty slowest tests in the job summary, and re-measures these tests once this has landed. For reference, CI: time limits, hang dumps and per-test durations for the store-test steps #126's figures are 8 min before Milestone Object storage: the log, the projection and the deployment in an object store #116, 29 min on Milestone Object storage: the log, the projection and the deployment in an object store #116's PR run and 53 min at its land.

• After merging Management plane, part 1: the fleet system dataset, grants as code, varve apply (v0.1.0-preview.10) #134 (Linux, this session): Varve.Store.Tests 316 passed, 17 skipped; Varve.Server.Tests 110 passed, 6 skipped; Varve.Protocol.Tests 113; Varve.Protocol.Client.Tests 117. The Release build has 0 warnings and 0 errors.

What was found

• The plan's tier rule stranded sizes. It needed a run of F sets in one tier. The policy property found chains where nothing ever merged, with sizes alternating either side of a tier's edge. It became the similar-size rule plus a pressure rule (ADR 0159).
• An abandoned merge's output waited for a later round before deletion. It now goes straight to deletion.
• A split's index kept the replaced set, holding retention high. The index rebase is generalised over merge, split and drop-merge.
• Maintenance failed permanently when an announce failed with an IOException. It now tolerates the failure and retries next round.
• DefaultMaxSetBytes read as 0, because of static initialisation order.
• A test-harness artifact: two Datasets wrote one MemoryStorage while background merges ran. Tests that open a second dataset now run with merges off.

What is left

• Release number: v0.1.0-preview.11, after Management plane, part 1: the fleet system dataset, grants as code, varve apply (v0.1.0-preview.10) #134's preview.10 (cut once ci(release): the gates job grants id-token: write, so v0.1.0-preview.10 can be cut #139 let the Release workflow start).
• Per-test store durations per OS: CI: time limits, hang dumps and per-test durations for the store-test steps #126.
• varve storage verify waits for Object store verification: varve storage verify, provider result records and a generated compatibility page #117. Multipart uploads stay a later decision. Both are stated in ADR 0162's consequences.

---

Checklist

• Issue referenced: every commit carries Refs #110; the descriptor commit carries the closing trailer (eng/issue-refs.cs passes).
• ADR cited: 0158–0162 are new; 0134, 0138, 0139 and 0164 are amended by dated amendment. No accepted decision is contradicted.
• Specification updated: docs/spec/storage-format.md §7.5, §7.6 and §8.
• API baseline updated: PublicAPI.Unshipped.txt in Varve.Store, .ObjectStorage, .Azure, .S3 and Varve.Protocol. Nothing is removed from Shipped.
• Conformance baseline: n/a.
• Tests: real components through public contracts; Azurite and RustFS for the services.
• A new gate's failure path was proven: the benchmark's latency gate printed OVER on the first run's mismatched comparison before it was made like for like (quoted in the record).
• Signed off: DCO passes, eng/dco.cs.
• A traceability record exists.
• dotnet run eng/ci.cs passes locally: not run as a whole. Its jobs were run separately on the merged head: the Release build (0 warnings, 0 errors); the store, server, protocol and protocol client suites; and the licence-header, CI-image, DCO, issue-reference, decision-set, release and status checks.

🤖 Generated with Claude Code

https://claude.ai/code/session_014BgePW3C3ECWERkiQMVdPb

[Hafeok]

CI on b41564b is red only on CS0618, as expected for an unaccepted milestone.

Every failed job stops in its build step. The errors are the 16 citations of the not-yet-accepted decisions in background-extent-merges, extent-deletion-by-reader-announcements and the-shared-commit-index (ADRs 0158, 0160, 0161). The analyzer reads them as "Not accepted … a build with warnings as errors will not ship it". There are no other compiler, analyzer or test errors.

The failed jobs are build (ubuntu-latest), native aot, browser wasm, browser storage in Chromium, storage (services), samples, samples (eventhubs), both container legs, conformance, both auth legs and decision report. In auth (zitadel) the log tail is the Zitadel dump that the failure step prints after the build.

• Fix: accepting ADRs 0158–0162 clears this. Nothing in the code changes. I'm not re-running anything, because a re-run gives the same result until then.
• Windows store-step duration: the build step fails first, so CI won't report the store step until the decisions are accepted. I'll quote the Windows and Linux store-step times here and in the traceability record from the first run after acceptance, against CI: time limits, hang dumps and per-test durations for the store-test steps #126's 8 min, 29 min and 53 min.
• agent review: this waits on the responsible human's approve <head> comment.

---

Generated by Claude Code

[Hafeok]

approve 7b40629

[Hafeok]

approve 0b02803