feat(capture): bound sync's download by max_cases; fill slots with leading turns

CHANGELOG.md

@@ -17,15 +17,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Filter.** With either limit set, `capture sync` promotes only captures
   recorded on `defaults.source_model`, and with `max_age` set, only captures
   recorded within it. Model ids match on bare names: lower-cased, provider
-  prefix and trailing snapshot date dropped, no alias lookup. `max_age` alone
-  caps nothing. `capture promote` ignores the limits.
+  prefix and trailing snapshot date dropped, no alias lookup. A conversation
+  turn that recorded no messages list of its own is promoted only when every
+  earlier turn of its conversation is local, since its history is rebuilt
+  from them (`not eligible: N missing earlier turns`, with a warning naming
+  the likely causes: the fetch window or budget, `capture clean`, the SDK's
+  per-turn sample rate, or a `turn_index` numbered from 1). `max_age` alone caps
+  nothing. `capture promote` ignores the limits.
 - **Plan.** With `max_cases` set, sync never grows a suite past the cap and
   swaps stale cases (too old, or recorded on another model) for fresh
   captures as they arrive. A stale case leaves at once when a fresh capture of
   the same question is added in its place; if its fresh copy is not added,
   the stale case leaves only to make room, like any other stale case. So a
   suite never shrinks because its cases went stale (an over-cap suite is still
-  trimmed to `max_cases`). A conversation enters whole or not at all. Cases a
+  trimmed to `max_cases`). A conversation enters whole when it fits; once no
+  whole conversation fits, the slots left take the next one's leading turns,
+  so one long conversation cannot leave a suite nearly empty. Cases a
   human marked `provenance: reviewed` are never retired; in a captured suite
   the mark goes in the case file (`.evalshift/suites/<suite>/<case>.json`),
   not in `golden.jsonl`, which sync regenerates. Sync knows a case by its
@@ -43,13 +50,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   recorded on <model> …; not filtering by model — check
   defaults.source_model`) and does not filter that suite by model rather than
   empty it.
-- **Fetch.** `max_age` also bounds which captures `capture sync`'s automatic
-  fetch downloads from `captures.store` (toolset sidecars are always
-  fetched); `--since` still overrides it, and `capture list` and
-  `capture fetch` are unchanged.
+- **Fetch.** The limits also bound what the automatic fetch of
+  `capture sync` and `capture list` downloads from `captures.store`:
+  `max_age` skips older captures, and `max_cases` takes only each suite's
+  newest `3 × max_cases` captures inside that window (`skipped 2400 beyond
+  each suite's newest 600`). Captures already local or promoted count toward
+  that budget, so each sync downloads at most that many per suite, however
+  many captures the bucket holds, and a re-sync with nothing new downloads
+  nothing. Toolset sidecars are always fetched; `--since` replaces the age
+  cutoff, not the budget; `capture fetch` is unchanged.
 - **Output.** Sync prints one line per suite: `alpha: 200 cases — kept 150,
   retired 50 (32 older than 14d, 18 on another model), added 50 of 2940
-  eligible; not eligible: …`.
+  eligible; not eligible: …`. With `max_cases` set, a capture's build
+  warnings print only when it becomes a case.
 - `doctor` gains `suite freshness` rows for each `source: captured` suite when
   either limit is set: `ok` (`alpha: 200 cases, captured 2d–12d ago`), or one
   `warn` per finding — the newest case is older than `max_age`, cases were

DOCS.md

@@ -177,7 +177,7 @@ captures:
 
 and `capture sync` and `capture list` fetch what is new before they read the local directory — captures are immutable and toolsets content-addressed, so only objects missing locally are downloaded, captures already promoted are skipped (so `capture clean` never causes a re-download; `capture sync --repair` alone re-fetches the promoted captures its rebuilds need, past `--since` too), and each file is written atomically. Bucket keys are untrusted: one not shaped like the SDK's layout — a suite directory containing `:`, `/`, `\`, NUL or `..`, or starting or ending with `.` or a space, included — is counted as malformed and never written (a key with a trailing newline included), and no key can place a file outside `.evalshift/`; folder-marker keys ending in `/` (the S3 console's "Create folder", ADLS directories) are skipped without being counted. One summary line reports it (`fetched 12 capture(s) and 1 toolset(s) from s3://…`; with `capture list --json` it goes to stderr, so stdout stays pure JSON). `--since 24h` limits a fetch to recently written captures, `--offline` skips it, and `capture fetch` does only that step. `promote`, `diff` and `clean` stay local. Install the matching client library — `pip install boto3`, `google-cloud-storage` or `azure-storage-blob azure-identity` — and authenticate with the provider's normal chain (an OIDC role in CI, `aws sso login` / `gcloud auth application-default login` / `az login` locally); credentials never go in the URI, and a value containing `@` or `?` fails to load; no rejected value is echoed back, at most its scheme. A missing client library exits 1 naming the `pip install <package>` to run; a list or download failure exits 1, so `sync` promotes nothing from a partial fetch, with a hint naming the provider's credential chain (`aws sso login` / `AWS_PROFILE` / the CI OIDC role, `gcloud auth application-default login`, `az login`); a local write failure names the file and asks you to check free disk space and permissions instead. If `evalshift.yaml` cannot be read or does not validate, `sync` and `list` skip the fetch and work on the local directory as before, warning only when the file has a top-level `captures:` key. Full field reference: [`captures`](docs/configuration.md#captures).
 
-**Bounded suites.** Together, `captures.max_age` (a duration: `30m`, `24h`, `14d`, at most `36500d`; omit it for no age limit) and `captures.max_cases` (an int ≥ 1), both optional, keep each captured suite a capped, route-balanced sample of recent production on `defaults.source_model`; `evalshift init` scaffolds `max_age: 14d` and `max_cases: 200`. Setting either one turns the feature on: `max_age` filters what enters a suite and marks old cases stale, and `max_cases` caps and route-balances each suite. Without them `capture sync` promotes and keeps exactly what it did before, and `defaults.source_model` alone changes nothing. Case files record `captured_at` and `source_model` (the first `model_call`'s `model_id`) either way, and sync backfills both on existing cases whose capture is still local (`backfilled capture time and model on N case(s)`). With the limits set, sync promotes only captures recorded within `max_age` on `defaults.source_model`. Models match on bare names: lower-cased, provider prefix and one trailing snapshot date (`-20260301`, `-2026-08-07`) dropped, with no alias lookup, so set `defaults.source_model` to the id your app sends. If none of a suite's cases and captures was recorded on it, sync warns (`alpha: none of 212 case(s) and capture(s) were recorded on claude-sonnet-5-5 (seen: …); not filtering by model — check defaults.source_model`) and does not filter that suite by model, so a typo or a rollout whose first new-model capture has not arrived can neither empty nor freeze a suite. `max_cases` caps every suite under `.evalshift/suites/`, including ones promoted by hand with `capture promote` and ones whose entry is `managed: false` (that flag freezes the YAML entry, not the case files): new conversations enter whole or not at all, least-represented tool route first, in a fixed hash order, so the same captures give the same suite on any machine. A stale case (too old, or recorded on another model) is retired outright when a fresh capture of the same question is added in its place; if its fresh copy is not added, the stale case leaves only to make room, like any other stale case. Stale cases that leave to make room go other-model first, then oldest, so a suite never shrinks because it went stale. A suite still over the cap loses fresh cases from its most-represented route, oldest first. Cases marked `provenance: reviewed` are never retired; in a captured suite, mark them in the case file `.evalshift/suites/<suite>/<case>.json` (on the calls under `example.expected_tools`, and `example.expected_tool_rounds` when present), not in `golden.jsonl`, which sync regenerates from the case files on every run. `max_age` also bounds `capture sync`'s automatic fetch (`skipped 11 older than captures.max_age (14d)`); `--since` overrides it, and `capture list` and `capture fetch` do not apply it. On a fresh checkout or after `capture clean`, a conversation that straddles the window loses its earlier turns to it, and a turn that does not record its own messages list then rebuilds a truncated history: pass the full messages list to `capture.model_call`. `doctor` and `compare` report stale suites (below). Expire the bucket's `captures/` prefix with a lifecycle rule after a period ≥ `max_age`; the CLI never deletes from the bucket. Full rules, output and lifecycle examples: [Bounded suites](docs/configuration.md#bounded-suites).
+**Bounded suites.** Together, `captures.max_age` (a duration: `30m`, `24h`, `14d`, at most `36500d`; omit it for no age limit) and `captures.max_cases` (an int ≥ 1), both optional, keep each captured suite a capped, route-balanced sample of recent production on `defaults.source_model`; `evalshift init` scaffolds `max_age: 14d` and `max_cases: 200`. Setting either one turns the feature on: `max_age` filters what enters a suite and marks old cases stale, and `max_cases` caps and route-balances each suite. Without them `capture sync` promotes and keeps exactly what it did before, and `defaults.source_model` alone changes nothing. Case files record `captured_at` and `source_model` (the first `model_call`'s `model_id`) either way, and sync backfills both on existing cases whose capture is still local (`backfilled capture time and model on N case(s)`). With the limits set, sync promotes only captures recorded within `max_age` on `defaults.source_model`. Models match on bare names: lower-cased, provider prefix and one trailing snapshot date (`-20260301`, `-2026-08-07`) dropped, with no alias lookup, so set `defaults.source_model` to the id your app sends. If none of a suite's cases and captures was recorded on it, sync warns (`alpha: none of 212 case(s) and capture(s) were recorded on claude-sonnet-5-5 (seen: …); not filtering by model — check defaults.source_model`) and does not filter that suite by model, so a typo or a rollout whose first new-model capture has not arrived can neither empty nor freeze a suite. `max_cases` caps every suite under `.evalshift/suites/`, including ones promoted by hand with `capture promote` and ones whose entry is `managed: false` (that flag freezes the YAML entry, not the case files): new conversations enter whole, least-represented tool route first, in a fixed hash order, so the same captures give the same suite on any machine; once no whole conversation fits, the slots left take the next one's leading turns. A conversation turn that recorded no messages list of its own rebuilds its history from the turns before it, so under the limits it is promoted only when every earlier turn is local (`not eligible: 3 missing earlier turns`). A stale case (too old, or recorded on another model) is retired outright when a fresh capture of the same question is added in its place; if its fresh copy is not added, the stale case leaves only to make room, like any other stale case. Stale cases that leave to make room go other-model first, then oldest, so a suite never shrinks because it went stale. A suite still over the cap loses fresh cases from its most-represented route, oldest first. Cases marked `provenance: reviewed` are never retired; in a captured suite, mark them in the case file `.evalshift/suites/<suite>/<case>.json` (on the calls under `example.expected_tools`, and `example.expected_tool_rounds` when present), not in `golden.jsonl`, which sync regenerates from the case files on every run. The limits also bound the automatic fetch of `capture sync` and `capture list`: `max_age` skips older captures (`skipped 11 older than captures.max_age (14d)`), and `max_cases` downloads only each suite's newest `3 × max_cases` captures inside that window (`2400 beyond each suite's newest 600`), counting captures already local or promoted, so each sync downloads at most that many per suite and a re-sync with nothing new downloads nothing. `--since` replaces the age cutoff, not the budget, and `capture fetch` applies neither. A turn without its own messages list is also missing earlier turns after `capture clean`, on a fresh checkout (promoted turns are never fetched), when `EVALSHIFT_SAMPLE_RATE` (drawn per turn) dropped them, or when `turn_index` starts at 1 instead of 0; sync warns naming these causes. Pass the full messages list to `capture.model_call` so each turn carries its own history. With `max_cases` set, build warnings print only for captures that become cases. `doctor` and `compare` report stale suites (below). Expire the bucket's `captures/` prefix with a lifecycle rule after a period ≥ `max_age`; the CLI never deletes from the bucket. Full rules, output and lifecycle examples: [Bounded suites](docs/configuration.md#bounded-suites).
 
 **`capture sync`** is the workhorse. Per suite it: