[babaliauskas]

What this changes

This is the CLI half of conversation capture folders. evalshift-sdk 0.7.0 (a separate, later PR) will file a capture.agent_session conversation's turns in one folder, captures/<suite>/conv_<16 hex of sha256(conversation_id)>/cap_<hex>.json, on disk and in a bucket. This PR teaches the CLI that layout first. It ships in 1.6.0 with the unreleased #57 and #58.

It is safe on its own. With today's SDK (0.6, flat files) a flat capture is a unit of one, so capture sync, list, fetch and clean behave as before. The exceptions are the two fixes listed under capture clean and suite names below.

• Reader. iter_captures reads captures/<suite>/*.json, then each captures/<suite>/conv_<16 hex>/*.json folder. It goes one level deep and matches folder names with fullmatch, so nested/, uppercase hex and wrong lengths are ignored. A suite argument maps to the directory the SDK writes, so team/support reads captures/team_support/. That fixes capture list <suite>, sync --suite, fetch --suite, clean SUITE and promote --suite for suite names containing a slash. A suite directory that can't be listed is skipped, as in 1.5.x.
• Fetch. A bucket key is valid when it is captures/<suite>/<name>.json or captures/<suite>/conv_<16 hex>/<name>.json; anything else counts as malformed. A conversation folder moves as a unit:
  • A folder with an unpromoted turn inside the window is fetched with every turn inside the window, plus its promoted turns at any age while the bucket still holds them. A fresh checkout therefore rebuilds a new turn's history instead of reporting "missing earlier turns".
  • Unpromoted turns outside the window stay behind.
  • The max_cases budget (3 × max_cases per suite) counts captures but ranks and takes whole conversations, newest first.
  • Once no whole conversation fits, the leftover goes to the first skipped conversation whose leading turns include one not yet promoted. A conversation whose leading turns are all promoted already is passed over, so the budget is never spent on turns the suite already holds.
• capture clean (default --promoted) keeps a conversation's promoted turns while any of its turns is unpromoted, says how many it kept, and removes the conversation folders it empties.
• Docs, CHANGELOG and the "missing earlier turns" warning name the new layout and causes:
  • turns written flat (SDK < 0.7.0, EVALSHIFT_CAPTURE_LAYOUT=flat, or the @capture.agent decorator, whose conversation_id is static);
  • captures evicted by the SDK's max_captures / capture_ttl;
  • captures expired by the bucket's lifecycle rule.

Compatibility

SDK writes	CLI reads	Result
≤ 0.6.0 (flat)	this PR	Works. Flat files are units of one, and conversations group by envelope as today.
≥ 0.7.0 (folders)	this PR	Works. This is the target state.
≥ 0.7.0 (folders)	≤ 1.5.x	Broken, silently: nested captures are invisible to an old CLI. That is why this ships first, and SDK 0.7.0 will require CLI ≥ 1.6.0.
Mixed (SDK upgraded mid-conversation)	this PR	Works locally: sync groups by conversation_id across both layouts. On a fresh checkout, the conversation's promoted flat turns are not fetched (documented).

Release order:

1. This PR merges.
2. CLI 1.6.0 is released, covering feat(capture): bounded capture suites (captures.max_age / captures.max_cases) #57, feat(capture): bound sync's download by max_cases; fill slots with leading turns #58 and this PR.
3. The SDK PR is merged and released as 0.7.0, with "requires evalshift CLI 1.6.0".
4. Website PRs follow each release.

Test plan

• make ci: 3331 passed, 1 skipped (the existing evalshift-server skip), coverage 95.23%. ruff, ruff format, mypy --strict and pre-commit run --all-files are clean.
• New tests:
  • reading conversation folders, and ignoring look-alike folders (uppercase, 8 or 17 hex digits, two levels deep);
  • suite-name mapping that matches the SDK's _safe_segment;
  • a conversation split across both layouts syncs as one;
  • fetch units and malformed nested keys;
  • a new turn brings its promoted turns of any age, also under the budget;
  • an old unpromoted turn stays behind;
  • whole conversations newest first, the leftover leading turns, the leftover passing over promoted-only prefixes, and a rerun fetching nothing;
  • clean keeping a half-promoted conversation in both layouts, removing a fully promoted folder, and never removing a folder with other files;
  • an unreadable suite directory.
• Integration test: a fresh checkout (committed cases, no local captures) promotes a new turn with its full history. It fails when the wanted-conversation rule is reverted.
• Offline smoke on a scratch copy of a real app's captures (testApp):
  • main and this branch produce byte-identical suites;
  • a re-sync is byte-identical;
  • after moving converter's 11 conversation captures into their conv_* folders, the suite is still byte-identical, and capture list shows them.
• Per-task reviews, a whole-branch review, and a scoped re-review of its fixes. The whole-branch review found a flaw in the plan's leftover rule, now fixed: the leftover could go entirely to already-promoted turns, so a long conversation never progressed, and capture clean made sync re-download the same turns every cycle.

Known trade-offs (documented)

• A conversation with more turns than the budget is fetched from its start, not its middle, because later turns rebuild their history from earlier ones.
• After a sync promotes a passed-over conversation's leading turns, the next sync can give the leftover to the next conversation. The suite fills over a few syncs (bounded by max_cases, and every machine computes the same step from the committed suites), so a re-sync downloads nothing only while the promoted set is unchanged.

🤖 Generated with Claude Code

[babaliauskas]

The SDK half is open as a draft: evalshift/evalshift-sdk#18. It must not merge before CLI 1.6.0 is on PyPI.

Two commits since the PR opened: b5a26ec and 157ab21 bring docs/sdk.md and llms-full.txt in line with that PR's final behaviour:

• local GC counts what the TTL leaves; the conversation at the boundary keeps its newest turns;
• if an evalshift CLI older than 1.6 is installed beside SDK 0.7.0, the SDK writes flat and logs a warning.

Cross-repo smoke. The SDK branch wrote and this branch read, in a scratch venv with both editable installs:

• Folder mode (EVALSHIFT_CAPTURE_LAYOUT=conversation):
  • three agent_session turns landed in captures/planner/conv_e1961a13f5bd6642/, while standalone and @capture.agent captures stayed flat;
  • capture sync --offline promoted all 5, and the last turn's history was rebuilt from the two earlier turns;
  • the SDK's ObjectStoreSink keys, fetched by fetch_captures, rebuilt exactly the SDK's FileSink tree, with team/support landing in team_support/conv_d0f631ca1ddba8db/ and 0 keys malformed.
• Guard mode (env unset; this branch still declares 1.5.0 until the release bump): every capture was flat with one warning; sync still promoted all 5 with their history.

That last point is why the CLI 1.6.0 version bump has to land before SDK 0.7.0 ships.

🤖 Generated with Claude Code