feat: conversation folders, per-conversation sampling and GC

CHANGELOG.md

@@ -9,6 +9,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
+- **Breaking: conversation folders.** A `capture.agent_session` /
+  `agent_session_async` turn with a `conversation_id` is written to
+  `captures/<suite>/conv_<16 hex>/cap_<hex>.json`, on disk and in an object
+  store. The hex is the start of `sha256(conversation_id)`. Standalone
+  captures stay at `captures/<suite>/cap_<hex>.json`, and so does every
+  capture from the `@capture.agent` decorator, whose `conversation_id` is
+  fixed when it is applied (its envelope still carries the id). **Requires
+  evalshift CLI 1.6.0 or later, so upgrade the CLI first**: older CLIs do not
+  see the folders, and their `capture clean` can sweep toolset sidecars that
+  only folder captures use. If an evalshift CLI older than 1.6 is installed
+  in the same environment, the SDK writes flat and logs a warning (set
+  `EVALSHIFT_CAPTURE_LAYOUT=conversation` to override); an older CLI in a
+  *different* environment (e.g. CI pinned to 1.5) can't be detected, so
+  upgrade it first. Code that globs `captures/<suite>/*.json` must also read
+  `captures/<suite>/conv_*/*.json`. Set `EVALSHIFT_CAPTURE_LAYOUT=flat` (or
+  pass `layout="flat"` to `FileSink` / `ObjectStoreSink`) to keep the 0.6
+  layout.
+- **Sampling keeps conversations whole.** With `EVALSHIFT_SAMPLE_RATE` set,
+  a `capture.agent_session` (or `agent_session_async`) given a
+  `conversation_id` is sampled by a hash of that id, so every turn of a
+  conversation is captured or none is, in every process. The
+  `conversation_id` is what gets sampled: a user id passed as
+  `conversation_id` samples users, and a constant id in `agent_session`
+  keeps every run or none. The `@capture.agent` decorator (whose
+  `conversation_id` is fixed when it is applied), runs without an id, and
+  the LangChain handler still draw at random.
+- **`max_captures` / `capture_ttl` count conversations.** Each suite
+  directory is capped by unit, a flat capture or a whole conversation
+  folder. `capture_ttl` first evicts every capture older than it;
+  `max_captures` then counts the captures the TTL leaves, newest unit first:
+  the newest conversations are kept whole while they fit, the one at the
+  boundary keeps its newest turns, and older ones are evicted whole. So one
+  newer capture evicts at most one older capture, not a whole live
+  conversation, and a conversation that never ends (a user id passed as
+  `conversation_id`) keeps its newest `max_captures` turns and stays bounded.
+  Folders left empty are removed.
+  Suites with only flat captures behave as before. Dedup is unchanged: its
+  hash already includes the conversation and turn.
 - The PyPI project page's Homepage link goes to <https://www.evalshift.dev> instead of the
   GitHub repo, and a Documentation link to <https://www.evalshift.dev/docs/sdk> is added.
 

README.md

@@ -8,6 +8,15 @@ own when the host's disk does not outlive it. The `evalshift` CLI reads those ca
 disk, or mirrors them down from the bucket; the layout is the interface, and the SDK and CLI
 never call each other.
 
+A `capture.agent_session` / `agent_session_async` turn with a `conversation_id` is written to
+`captures/<suite>/conv_<16 hex>/cap_<hex>.json`, where the hex is the start of
+`sha256(conversation_id)`; every other capture, including every `@capture.agent` decorator
+capture, is written to `captures/<suite>/cap_<hex>.json`. Reading the folders needs evalshift CLI
+1.6.0 or later: if an older evalshift CLI is installed in the same environment, the SDK writes
+flat and logs a warning (`EVALSHIFT_CAPTURE_LAYOUT=conversation` overrides it); an older CLI in a
+different environment can't be detected, so upgrade it first. `EVALSHIFT_CAPTURE_LAYOUT=flat` (or
+`layout="flat"` on `FileSink` / `ObjectStoreSink`) writes every capture flat, as before 0.7.0.
+
 - **Distribution:** `evalshift-sdk` · **import name:** `evalshift`
 - **Runtime deps:** none (stdlib-only)
 - **Python:** >= 3.10
@@ -86,15 +95,18 @@ client = wrap_openai(OpenAI())                          # OpenAI(base_url=...) c
 
 Capture writes one JSON file per sampled invocation, so `.evalshift/captures/` is kept bounded by
 default (no configuration needed): identical-input re-runs are de-duplicated, and each suite
-directory is capped at the **200 newest** captures (oldest evicted). Tune it with env vars — no code
-change required (precedence: an explicit `configure(...)` call > env var > built-in default):
+directory is capped at the **200 newest** captures the TTL leaves, counted by unit: a capture or
+a whole conversation folder. The newest conversations are kept whole while they fit, the one at
+the boundary keeps its newest turns, and older ones are evicted whole. Tune it with env vars — no
+code change required (precedence: an explicit `configure(...)` call > env var > built-in default):
 
 | Env var | Default | Meaning |
 | --- | --- | --- |
-| `EVALSHIFT_MAX_CAPTURES` | `200` | Max captures kept per suite dir. `0` / `none` / `unlimited` = uncapped. |
+| `EVALSHIFT_MAX_CAPTURES` | `200` | Max captures kept per suite dir, by unit: a capture or a whole conversation folder. `0` / `none` / `unlimited` = uncapped. |
 | `EVALSHIFT_DEDUP` | `on` | Collapse identical-input captures (per-process). `off` to disable. |
 | `EVALSHIFT_CAPTURE_TTL` | off | Evict captures older than N **seconds**. |
-| `EVALSHIFT_SAMPLE_RATE` | off | Capture only this fraction of runs, e.g. `0.25`. |
+| `EVALSHIFT_SAMPLE_RATE` | off | Capture only this fraction of runs, e.g. `0.25`; decided once per agent invocation; per conversation when an `agent_session` is given a `conversation_id`. |
+| `EVALSHIFT_CAPTURE_LAYOUT` | conversation folders (flat if an evalshift CLI older than 1.6 is installed alongside) | `flat` writes every capture flat, as before 0.7.0; `conversation` writes conversation folders even beside an older CLI. |
 | `EVALSHIFT_DIR` | `.evalshift` | Capture root directory. |
 
 A malformed value falls back to the default (these knobs never crash capture). To restore fully unbounded

docs/DECISIONS.md

@@ -180,6 +180,9 @@ exactly as in Phases 0-5; `active_sink()` returns the bare sink with identity pr
 - **Sampling decides at agent entry**, not write time. `should_capture_now()` is read at the top of
   `_run_agent`/`_run_agent_async` and both `agent_session*` context managers; an unsampled run is a
   near-pure pass-through (no span tree, no redaction, no serialize). Decision made once per run.
+  *Amended by D-conv-layout:* an `agent_session` / `agent_session_async` given a
+  `conversation_id` draws by a hash of it, so the decision is per conversation, the same in every
+  process; the decorator, runs without an id and the LangChain handler still draw at random.
 - **Dedup + GC compose as a `HygieneSink`** returned by `active_sink()` only when a knob is set —
   the single write seam in `capture/api.py` is unchanged. Fail-open is **granular inside the
   wrapper**: the outer `fail_open("sink write")` would drop the whole capture, so the dedup check
@@ -200,6 +203,11 @@ exactly as in Phases 0-5; `active_sink()` returns the bare sink with identity pr
     back, so one `os.scandir` stat pass (no JSON parse) gives both count-ordering and TTL-age
     cheaply, even at large `max_captures`. Limitation: tools that rewrite mtimes (`cp -p`, rsync,
     tar-extract) can perturb eviction order. `capture_ttl` is in **seconds**.
+    *Amended by D-conv-layout:* GC works per suite dir, by unit: a flat capture, or a whole
+    `conv_<16 hex>/` conversation folder timed by its newest file's mtime (one more `os.scandir`
+    per folder, still no JSON parse). The TTL goes first; the count then keeps the newest units
+    whole while they fit, the unit at the boundary keeps its newest turns, older units are evicted
+    whole, and a folder left empty is removed.
 - **Verified by:** `tests/test_hygiene_{sample,dedup,gc,sink}.py`, `tests/test_capture_sampling.py`,
   `tests/test_capture_hygiene_e2e.py`, and the hygiene fault matrix added to
   `tests/test_capture_failopen.py` + `tests/test_capture_async_failopen.py`.
@@ -215,7 +223,10 @@ from the environment (no code change required):
   `sample_rate` stay `None` (off). 200 is larger than any realistic single golden suite, so normal
   use never loses recent data, but runaway accumulation is capped. Dedup-on only collapses
   *identical-input* re-runs (the same best-effort per-process key as D-6) — a free win against the
-  re-run-the-same-eval churn pattern.
+  re-run-the-same-eval churn pattern. *Amended by D-conv-layout:* the 200 are counted per suite
+  dir, by unit: a capture or a whole conversation folder, among the captures the TTL leaves. The
+  newest conversations are kept whole while they fit, the one at the boundary keeps its newest
+  turns, and older ones are evicted whole.
 - **Env vars set the default** (read at `_Config` construction and re-read on `reset_config()`):
   `EVALSHIFT_MAX_CAPTURES`, `EVALSHIFT_CAPTURE_TTL`, `EVALSHIFT_DEDUP`, `EVALSHIFT_SAMPLE_RATE`.
   For the numeric knobs, `0` / `none` / `unlimited` means **uncapped** (`None`); a malformed value
@@ -366,6 +377,11 @@ policy in `docs/SCHEMA.md`.
   of, `<base>/toolsets/`. Distinct toolsets are few and captures number in the thousands, so an
   evicted toolset would orphan every capture that still references it, not just the one just
   evicted; orphan sweeping (if ever needed) is a CLI-side concern, not this SDK's.
+  *Amended by D-conv-layout:* `evict` now enters `conv_<16 hex>` conversation folders, one level,
+  and never any other subdirectory (its docstring says so), and its caller still passes
+  `<base>/captures/<suite>/`, so `toolsets/` stays out of reach. An evalshift CLI older than 1.6.0
+  does not see folder captures, so its `capture clean` can sweep sidecars that only they use:
+  upgrade the CLI first.
 - **No migration.** Schema 2.0.0 (D-8) already registers no edge from the 1.x major for exactly this
   reason — there is no honest value `tools_offered` can take for a capture written before per-call
   toolset capture existed. This decision does not reopen that question; it is the feature D-8's
@@ -523,7 +539,10 @@ D-stores-c) or `configure(sink=...)`.
 
 - **Keys are the local layout** (`captures/<safe_suite>/cap_<hex>.json`, `toolsets/<hex>.json`),
   so the CLI mirrors a bucket into `.evalshift/` and reads it unchanged. The documented contract
-  moves from "disk is the only interface" to "the layout is the interface".
+  moves from "disk is the only interface" to "the layout is the interface". *Amended by
+  D-conv-layout:* an `agent_session` turn with a `conversation_id` is keyed
+  `captures/<safe_suite>/conv_<16 hex>/cap_<hex>.json`, the same relative key `FileSink` writes,
+  so the bucket stays a mirror of `.evalshift/`.
 - **One URI grammar** shared verbatim with the CLI (`stores/uri.py`): `s3://`, `gs://`,
   `az://<account>/<container>/<prefix>`; `@`/`?` rejected so credentials never live in config.
   Neither leaks into logs either: the `@`/`?` rejections do not echo the URI, and the
@@ -578,6 +597,88 @@ stays a silent alias read only when the new name is unset or blank (no deprecati
 one day of exposure does not justify log noise), and `_store_env()` returns the variable name
 with the value so every `SinkConfigurationError` names the one the user set.
 
+## Conversation folders (D-conv-layout)
+
+### D-conv-layout (2026-10-20) — a conversation's turns share one folder, sampling keeps it whole, and GC keeps the newest ones whole
+**Problem.** Until 0.7.0 every capture was a flat `captures/<suite>/cap_<hex>.json`, one file per
+turn, so nothing in the layout said which files form a conversation; only the envelope's
+`conversation_id` did, and reading it means parsing every file. Each piece that works per file
+cut conversations apart. Sampling drew per run, so a 10% rate kept about one turn in ten of every
+conversation. GC evicted the oldest files of a suite, so a long conversation lost its first turns
+while its later ones stayed. And the CLI's per-suite budgets cut a bucket or a suite at a file
+boundary, so they could take a conversation's later turns without its first. Each gap leaves the
+CLI rebuilding a turn whose earlier turns are missing.
+
+**Decision.** An `agent_session` / `agent_session_async` turn with a `conversation_id` lives at
+`captures/<safe_suite>/conv_<16 hex>/<capture_id>.json`, where the folder is `conv_` plus the first
+16 hex digits of `sha256(str(conversation_id))` (`sinks/_layout.py`): lowercase, fixed length,
+safe on every filesystem, and never the raw id, which can carry `:` or a user id. Every other
+capture stays at `captures/<safe_suite>/<capture_id>.json`. Disk and store share the relative
+layout, so a bucket stays a mirror of `.evalshift/`; toolset sidecars stay at `toolsets/<hex>.json`.
+Captures from the `@capture.agent` decorator (sync and async) are always written flat, even with
+a `conversation_id` (`api._finalize(flat=True)` wraps the write in `writing_flat()`): the id is
+fixed when the function is decorated, so it names no conversation, and its runs would make one
+never-ending folder that is the whole suite. Their envelope still carries the id.
+`EVALSHIFT_CAPTURE_LAYOUT=flat` (any case, stripped) or `layout="flat"` on `FileSink` /
+`ObjectStoreSink` writes every capture flat; `EVALSHIFT_CAPTURE_LAYOUT=conversation` or
+`layout="conversation"` writes the folder layout; the argument wins over the env var. With
+neither (unset, empty or a typo, so a typo can never stop captures), the SDK writes the folder
+layout unless an evalshift CLI older than 1.6 is installed in the same environment (see
+Compatibility). With a sample rate,
+`agent_session` / `agent_session_async` given a `conversation_id` draw
+`conversation_draw(conversation_id)` (`hygiene/sample.py`: 64 bits of
+`sha256("evalshift-sample:" + id)`), so a conversation is kept or dropped whole in every process;
+the decorator, runs without an id and the LangChain handler draw `random.random()`.
+`max_captures` / `capture_ttl` work per suite dir, by unit: a flat capture, or a whole `conv_*`
+folder timed by its newest file's mtime (`hygiene/gc.py`). The TTL goes first: it evicts every
+file older than it, so an idle conversation goes whole and a live one loses only its old turns.
+The count then runs on the files the TTL leaves: units ordered newest first by their newest
+surviving file, the files within a unit newest first, and the first `max_captures` files in that
+order are kept. So the newest units are kept whole while they fit, the unit at the boundary keeps
+its newest turns that fit, and every older unit is evicted whole. A folder or file GC cannot read
+(another uid, or gone mid-scan) is skipped, which only lowers the count. A folder left empty is
+removed, and a suite with only flat files behaves exactly as before (the newest `max_captures`).
+Dedup is unchanged: `input_hash` already folds in `conversation_id` and `turn_index`
+(`trace/serialize.py`), so it never drops a distinct turn.
+*Why not one mutable file per conversation:* object stores cannot append, so every turn would
+rewrite the whole object; turns of one conversation land on different processes (behind a load
+balancer, after a restart), whose read-modify-write cycles on the one file would race and lose
+turns; and rewriting the full history on every turn makes a conversation's cost grow with the
+square of its length. One immutable file per turn keeps every write a single create, and the
+folder gives readers the grouping.
+
+**Trade-offs.** The newest unit keeps only its newest `max_captures` files, which is what keeps a
+never-ending conversation (an `agent_session` given a user id as its `conversation_id`) bounded.
+So a conversation longer than `max_captures`, or at the boundary of the count, or older than
+`capture_ttl`, loses its oldest turns locally, and older conversations go whole; the CLI then
+reports later rebuilt turns as missing earlier turns. That is the price of a bound; apps should
+pass the full messages list. Keeping the boundary conversation's newest turns, rather than
+evicting it whole, is what keeps one newer write from evicting more than one older capture (a
+conversation at the cap plus one newer flat capture keeps its newest `max_captures - 1` turns).
+The decorator's static id keeps drawing at random, since
+hashing it would capture every run or none, forever, and keeps writing flat, since its folder
+would hold the whole suite. And GC, or the CLI's
+`capture clean`, can remove a folder it has just emptied in the instant another process writes a
+new turn into it: that one write fails with an `OSError`, `FileSink` drops the capture and
+returns `None`, fail-open, and nothing is raised into the host.
+
+**Compatibility.** The CLI ships first: evalshift CLI 1.6.0 reads both layouts (`<suite>/*.json`,
+then `<suite>/conv_<16 hex>/*.json`, one level, never recursive), and SDK 0.7.0 requires
+evalshift CLI 1.6.0 or later. An older CLI does not see the folders, and its `capture clean` can
+sweep toolset sidecars that only folder captures use, so the CLI must be upgraded first. CLI 1.5.x
+declares `evalshift-sdk>=0.4.0` with no upper bound, so installing it can pull SDK 0.7.0 in
+without the user choosing it. So if an evalshift CLI older than 1.6 is installed in the same
+environment (`importlib.metadata.version("evalshift")`, checked once per process, its leading
+`major.minor`), the SDK writes flat and logs a warning (set `EVALSHIFT_CAPTURE_LAYOUT=conversation`
+to override); an older CLI in a *different* environment (e.g. CI pinned to 1.5) can't be
+detected, so upgrade it first. A missing CLI or an unreadable version means no guard. Code that
+globs `captures/<suite>/*.json` must also read `captures/<suite>/conv_*/*.json`.
+`EVALSHIFT_CAPTURE_LAYOUT=flat` is the escape hatch: a production rollback lever and a bridge
+for tooling that globs the flat layout, with no code change. The envelope and `schema_version`
+are unchanged. Amends D-6, D-6a, D-toolset's GC note and D-stores in place. Verified by
+`tests/test_conversation_layout.py`, `tests/test_hygiene_gc.py`, `tests/test_hygiene_sample.py`,
+`tests/test_capture_sampling.py` and `tests/test_hygiene_sink.py`.
+
 ## Open follow-ups (not blocking v1)
 - ~~**D1-followup:** unify packaging so CLI + SDK co-install cleanly~~ — resolved 2026-09-09 in
   the CLI-depends-on-SDK form; see D-pkg.