feat(capture): bounded capture suites (captures.max_age / captures.max_cases)

[babaliauskas]

What this changes

Adds two optional settings under captures: in evalshift.yaml, max_age (e.g. 14d) and max_cases (e.g. 200). Together they keep each captured suite a capped, route-balanced sample of recent production traffic on defaults.source_model.

• capture sync filters. It promotes only captures recorded within max_age and on the source model. Model ids match on bare names: provider prefix, case and snapshot date are ignored. A per-suite guardrail turns model filtering off, with a warning, when no case or capture was recorded on defaults.source_model, so a typo or alias can never empty or freeze a suite.
• capture sync plans and retires. With max_cases set, the pure plan_suite fills free slots route-balanced, in sha256 order. Conversations enter whole. Stale cases (too old, or recorded on another model) leave only to make room, or when a fresh copy of the same question is added. Over-cap suites are trimmed from the largest route, oldest first. Cases marked provenance: reviewed (in the case file) are never retired. A retirement deletes the exact file the case was read from. The same captures give the same suite on any machine, and a re-run with nothing new touches no file.
• Fetch window. max_age bounds what sync downloads from captures.store. --since still overrides it.
• Provenance. Case files record captured_at / source_model, and sync backfills them on existing cases whose capture is still local.
• Freshness checks. doctor gains a suite freshness row per captured suite and a captures limits nudge (store set without max_cases). compare prints the same findings above the verdict, including for the auto-selected suite, so a CI-only team sees its suite went stale.
• init and the example. init scaffolds max_age: 14d, max_cases: 200. examples/capture-first was regenerated with this CLI.
• Opt-in. With neither limit set, sync keeps and promotes exactly what it did before. defaults.source_model alone changes nothing.

Why

The SDK's 200-per-suite cap applies to local disk only, and the bucket keeps everything. A team that captures thousands of records a week, deletes its local mirror and runs capture sync got every record ever captured, old models included, and suites only grew. This bounds suite size and keeps the suites fresh.

Design spec: docs/superpowers/specs/2026-10-16-bounded-capture-suites-design.md. The plan's "Execution notes" section lists where the shipped behaviour deviates from the plan: docs/superpowers/plans/2026-10-16-bounded-capture-suites.md.

Compatibility. Older CLIs reject the new config keys and the new case-file fields. The first capture sync on this CLI writes the fields even without limits, so everyone running capture commands, including a pinned CI action's evalshift-version, should upgrade together. CI's compare reads only golden.jsonl, whose format is unchanged. CHANGELOG has the details.

Test plan

• pytest passes: make ci shows 3267 passed, 1 skipped (the existing evalshift-server skip), coverage 95.15%.
• ruff check, ruff format --check clean; pre-commit run --all-files clean.
• mypy --strict src/evalshift_cli clean.
• CHANGELOG.md updated under ## [Unreleased].
• Docs updated: DOCS.md, llms-full.txt, docs/configuration.md (new "Bounded suites" section), getting-started, sdk, conversations, agents, and examples/capture-first.
• Integration test tests/integration/test_capture_lifecycle.py: weeks of bucket captures stay one bounded, fresh suite; the fresh-checkout fetch is pinned; production model migration; the wrong-source-model guardrail on new and established suites.
• Offline smoke on a scratch copy of a real project (3 suites, 31 cases):
  • First sync with 14d/200: backfill only.
  • Re-sync: nothing changed.
  • max_cases: 5: each suite trimmed to 5.
  • Re-sync after the trim: byte-identical.
  • doctor: ok / warn freshness rows as expected.
• Online fetch against a real bucket (not run: the AWS session had expired). The fake-bucket integration test covers it.

Screenshots

✓ promoted 0 capture(s) into 3 suite(s), skipped 31 already-promoted (use --force to overwrite).
  converter: 5 cases — kept 5, retired 6 (6 over max_cases), added 0 of 0 eligible
  planner: 5 cases — kept 5, retired 6 (6 over max_cases), added 0 of 0 eligible
  triage: 5 cases — kept 5, retired 4 (4 over max_cases), added 0 of 0 eligible

 ✓  suite freshness     converter: 5 cases, captured 1d–1d ago
 ✗  suite freshness     planner: newest case captured 1d ago (captures.max_age 1h); refresh with evalshift capture sync

🤖 Generated with Claude Code