[babaliauskas]

Requires evalshift CLI 1.6.0 or later. Don't merge this until CLI 1.6.0 is on PyPI. The CLI half is evalshift/evalshift-cli#60, which reads both layouts. This PR is a draft until then.

What this changes

Ships as evalshift-sdk 0.7.0. A conversation, not a turn, becomes the unit of capture storage.

• Conversation folders. A capture.agent_session / agent_session_async turn with a conversation_id is written to captures/<suite>/conv_<16 hex of sha256(conversation_id)>/cap_<hex>.json. The relative path is the same on disk (FileSink) and in a bucket (ObjectStoreSink). Captures stay flat at captures/<suite>/cap_<hex>.json in these cases:
  • standalone captures;
  • @capture.agent decorator captures, even with a conversation_id. That id is fixed when the function is decorated, so it names no conversation. Without this, a static-id agent would put its whole suite in one never-ending folder. The envelope still carries the id.
• Every write is still one immutable file. No read-modify-write, and the SDK never reads the bucket.
• Escape hatch. EVALSHIFT_CAPTURE_LAYOUT=flat, or layout="flat" on either sink, restores the 0.6 layout. This is also the rollback lever.
• Co-install guard. If an evalshift CLI older than 1.6 is installed in the same environment, the SDK writes flat and logs one warning. Released CLI 1.5.x declares evalshift-sdk>=0.4.0 with no upper bound, so a fresh install of a pinned 1.5 would otherwise pull in 0.7.0 without anyone choosing it. EVALSHIFT_CAPTURE_LAYOUT=conversation overrides the guard. An older CLI in a different environment, such as CI pinned to 1.5, can't be detected; the CHANGELOG says to upgrade it first.
• Sampling keeps conversations whole. With EVALSHIFT_SAMPLE_RATE set, an agent_session that has a conversation_id is sampled by a hash of that id, so every turn is kept or dropped together in every process. The decorator, runs without an id, and the LangChain handler still draw at random.
• Local GC counts conversations (max_captures / capture_ttl, per suite directory):
  • The TTL runs first.
  • The count then runs on the captures the TTL leaves, newest conversation first. The newest conversations are kept whole while they fit, the one at the boundary keeps its newest turns, and older ones are evicted whole. So one newer capture evicts at most one older capture.
  • A conversation that never ends keeps its newest max_captures turns.
  • Emptied folders are removed, and a folder or file that can't be read is skipped rather than stopping the pass.
  • Suites with only flat captures behave exactly as in 0.6.
• Dedup is unchanged. Its hash already folds in conversation_id and turn_index.
• Docs: CHANGELOG (Breaking), README, DOCS.md, llms-full.txt, llms.txt, SCHEMA.md, the example README, and a new DECISIONS.md entry, D-conv-layout. It records the problem, the decision and why we rejected one mutable file per conversation, the trade-offs, and the compatibility rule. D-6, D-6a, D-toolset and D-stores are amended in place.

Compatibility

SDK writes	CLI reads	Result
0.7.0 (folders)	≥ 1.6.0	Works. This is the target state.
0.7.0, CLI < 1.6 installed alongside	any	Flat plus a warning, which every CLI reads.
0.7.0 (folders)	≤ 1.5.x elsewhere	Broken, silently: an old CLI can't see folder captures, and its capture clean can sweep their toolset sidecars. Hence "upgrade the CLI first".
0.7.0 with EVALSHIFT_CAPTURE_LAYOUT=flat	any	Same as 0.6, except sampling is per conversation.

Release order:

1. CLI #60 merges.
2. CLI 1.6.0 is released. The CLI's version bump must land first: the guard compares against it, and the CLI branch still says 1.5.0.
3. This PR merges.
4. The chore/release-0.7.0 PR bumps pyproject.toml and __version__.
5. Tag v0.7.0, after checking the PyPI trusted publisher.
6. A website PR follows.

Test plan

• ruff check, ruff format --check and mypy (107 files) are clean; pytest: 888 passed.

• New tests:

  • folder names pinned to the same golden values as the CLI's tests (c1 → conv_d0f631ca1ddba8db, planner:session-1 → conv_e1961a13f5bd6642);
  • look-alike names rejected;
  • both sinks, both layouts, and env versus argument precedence;
  • decorator captures flat (sync, async, store) while a later session write still goes into its folder;
  • every guard branch, including odd versions, lookup faults and exactly one warning;
  • per-conversation sampling (sync and async) with golden draws, and the decorator's static id still random;
  • GC: the boundary rule, TTL then count, the never-ending conversation, unreadable and vanishing folders, the collapse case, and a seeded flat-only parity test against keep-newest-N.

• Cross-repo smoke, with this branch writing and the CLI #60 branch reading, from disk and through a bucket, in a scratch venv with both editable installs.

  • Folder mode (EVALSHIFT_CAPTURE_LAYOUT=conversation):
    • three agent_session turns landed in planner/conv_e1961a13f5bd6642/, while the standalone and @capture.agent captures stayed flat;
    • capture sync promoted all 5, and the last turn's history was rebuilt from the two earlier ones;
    • the bucket mirror fetched by the CLI equals the FileSink tree, with team/support landing in team_support/conv_d0f631ca1ddba8db and 0 keys malformed.
  • Guard mode (env unset; the CLI branch still declares 1.5.0): every capture was written flat, with exactly one warning naming CLI 1.5.0. Sync still promoted all 5 with history, and the bucket mirror still equals the tree.

• Per-task reviews, a whole-branch review, and a scoped re-review of its fixes. The whole-branch review found:

  • TTL plus count over-evicting;
  • one unreadable folder disabling GC;
  • the unbounded CLI 1.5 dependency.

  All three are fixed above. A probe compared the new GC rule with an independent reference on 3,000 random suites and found no mismatch.

Known trade-offs (documented in D-conv-layout)

• Turns lost locally: a conversation longer than max_captures, at the GC boundary, or older than capture_ttl loses its oldest turns locally.
• The rmdir race: if GC or capture clean empties a folder just as another process writes into it, that one capture is dropped, fail-open.
• What gets sampled: the conversation_id itself. A user id passed as conversation_id samples users.

🤖 Generated with Claude Code