diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md new file mode 100644 index 00000000..831a926b --- /dev/null +++ b/DEVELOPMENT.md @@ -0,0 +1,64 @@ +# Development + +Requires Rust 1.85+ (edition 2024); the exact toolchain is pinned via +`rust-toolchain.toml`. Day-to-day tasks are driven by +[`just`](https://github.com/casey/just); run `just help` to list the +recipes. + +## The dev loop + +```bash +just check # auto-format, then run the full CI verification +just ci # the same quality gates CI runs, without auto-format +``` + +Both take gate names to narrow or exclude: the gates are `format`, +`shellcheck`, `clippy`, `test`, `doc`, `examples`, `plugin`, and +`site`, and a `-` prefix excludes one. + +```bash +just ci test # just the workspace tests +just check -site # everything except the site build +``` + +Individual pieces are also available directly: + +```bash +just fmt # format all code (Rust + site) +just clippy # clippy across the workspace, warnings denied +cargo test --workspace +``` + +The `examples` gate validates every document under `examples/` with +`path p validate`. + +## Other recipes + +```bash +just site # site dev server (eleventy + wasm watcher) +just refresh-openapi # re-fetch the Pathbase OpenAPI spec into pathbase-client +just test-pathbase-live # live smoke test against a Pathbase deployment +just release-check # dry-run publish of all workspace crates (safe) +just release-publish # publish to crates.io for real +``` + +## Agent session format notes + +[docs/agents/formats/](docs/agents/formats/README.md) records our +understanding of each agent's on-disk session format, written down +while building the derive crates. That includes a twelve-part reference +for Claude Code's JSONL (envelope, entry types, session chains, +compaction, a writing-compatible guide), the writer contract the +Copilot CLI loader appears to enforce, and single-file references for +Codex, Gemini, opencode, Cursor, and Pi. + +None of these formats are documented by their vendors. The notes come +from observed behavior of specific versions: they are works in +progress, with gaps, and the agents can change their formats at any +time. Corrections are welcome. If you're building your own session +tooling, they're a useful starting point even if you never run our +code. + +The notes are meant to stay in sync with the derive crates. If you +change what a crate reads or writes, update the matching document under +`docs/agents/formats/`. diff --git a/README.md b/README.md index d3a82eba..3252d80c 100644 --- a/README.md +++ b/README.md @@ -1,32 +1,60 @@ # Toolpath -A tool-agnostic format for tracking artifact transformation provenance. - -## What is this? - -Toolpath records the complete history of how code (and other artifacts) evolved: - -- **Who** made changes (humans, AI agents, formatters, linters, CI) -- **What** they changed (unified diffs + structural AST operations) -- **Why** they changed it (intent, linked issues, reasoning) -- **What else they tried** (dead ends preserved for reflection) -- **Verification** (cryptographic signatures, identity resolution) - -## Three core objects - -| Object | What it represents | Example | -|-----------|-------------------------------------------|------------------------| -| **Step** | A single change to artifact(s) | One commit, one edit | -| **Path** | A sequence of steps with a base context | A PR, a coding session | -| **Graph** | A collection of related paths | A release | - -Steps form a DAG via parent references. Dead ends are implicit: steps not in the ancestry of `path.head`. - -``` - +-- step-3a -- step-4a (dead end) -step-1 -- step-2 --+ - +-- step-3b -- step-4b -- step-5b (head) -``` +> *The story of how we got here.* + +Toolpath is an open format for coding-agent sessions: one schema that +records what an agent did, why, what it tried that didn't work, and +what it cost, independent of which agent did it. + +Every agent writes its own undocumented session log. Claude Code keeps +rotating JSONL chains, Codex writes rollout files, Gemini has chat +directories, Copilot an event stream, opencode a SQLite database, +Cursor a composer store, Pi session trees. Toolpath reads them all into +one format and writes them back. With everything in one format you can: + +- **Portability.** Share a session and resume it on another machine or + in another harness. A session started in Claude Code can be picked up + in Codex. +- **An archive you control.** `path p cache sync` incrementally ingests + every session on the machine into plain JSON under `~/.toolpath/`. +- **Unified tooling.** A tool written against one schema works for + every harness at once, instead of needing a parser per agent. + `path query` is the built-in example: it searches and aggregates + across every session on the machine, whoever wrote them, using jq + filters. +- **Token accounting.** Real session totals, per-step attribution where + the source reports it, and reasoning breakdowns, even though every + provider reports usage differently. +- **Provenance.** Who changed what, why, and what was tried and + abandoned. The record that git collapses at merge time. + +## Supported harnesses + +| Harness | Read | Write | Resume | Notes | +|---|:-:|:-:|:-:|---| +| Claude Code | ✓ | ✓ | ✓ `claude -r` | Rotated session files merged into one chain; projected sessions load in Claude Code | +| Gemini CLI | ✓ | ✓ | ✓ `gemini --resume` | Sub-agent sessions folded in; reasoning tokens round-trip losslessly | +| Codex CLI | ✓ | ✓ | ✓ `codex resume` | Real unified diffs on every file change; per-step token attribution | +| Copilot CLI | ✓ | ✓ | ✓ `copilot --resume` | Preview; projected sessions verified to resume in copilot 1.0.67 | +| opencode | ✓ | ✓ | ✓ `opencode --session` | File diffs recovered from opencode's git snapshots | +| Cursor (IDE) | ✓ | ✓ | ✓ opens workspace | Projected composers render in Cursor.app's chat sidebar | +| Pi | ✓ | ✓ | ✓ `pi --session` | Branching session trees preserved as DAGs | + +Read means `path p import`, `path p list`, and `path show`. Write means +`path p export`. Resume means `path resume`, which projects the session +into the harness's on-disk layout and execs its resume command. Any +readable session can be projected into any writable harness; a +cross-harness conformance matrix in CI exercises the pairs. + +## Beyond sessions + +Agent sessions are one kind of path. Toolpath is a general format for +how artifacts change across the development lifecycle, so the session +that wrote a change, the pull request that reviewed it, and the release +that shipped it can share one graph. `path p import git` derives paths +from commit history and `path p import github` from GitHub pull +requests, review threads, CI runs and comments included; `path p merge` +combines documents into a single `Graph`. ## Install @@ -34,13 +62,13 @@ step-1 -- step-2 --+ # Prebuilt binary (macOS arm64, Linux x86_64/arm64) curl --proto '=https' --tlsv1.2 -fsS https://toolpath.net/install.sh | bash -# from source +# From source cargo install path-cli ``` Both install a binary called `path`. -Claude Code users can skip the manual install entirely — the Toolpath plugin +Claude Code users can skip the manual install entirely: the Toolpath plugin bundles the CLI (downloaded and installed globally on first use) and adds `/path:share` and `/path:query` slash commands: @@ -51,104 +79,179 @@ bundles the CLI (downloaded and installed globally on first use) and adds See [plugins/claude-code](plugins/claude-code/) for details. -> The older `toolpath-cli` crate name still works — `cargo install toolpath-cli` is now a thin shim that pulls in `path-cli` and installs the same binary. New users should reach for `path-cli` directly; the shim will eventually be retired. +## Quick start -## Workspace +```bash +# Archive every agent session on this machine (all harnesses, incremental) +path p cache sync -``` -crates/ - toolpath/ Core types, builders, query API - toolpath-convo/ Provider-agnostic conversation types, traits, and Toolpath-Path derivation - toolpath-git/ Derive from git repository history - toolpath-github/ Derive from GitHub pull requests - toolpath-claude/ Derive from Claude conversation logs - toolpath-gemini/ Derive from Gemini CLI conversation logs - toolpath-codex/ Derive from Codex CLI rollout files - toolpath-copilot/ Derive from GitHub Copilot CLI session logs (preview) - toolpath-opencode/ Derive from opencode SQLite databases - toolpath-cursor/ Derive from Cursor (IDE) state.vscdb bubble store - toolpath-pi/ Derive from Pi (pi.dev) agent sessions - toolpath-dot/ Graphviz DOT visualization - toolpath-md/ Markdown rendering for LLM consumption - pathbase-client/ Progenitor-derived typed client for the Pathbase HTTP API - path-cli/ Unified CLI (binary: path) - toolpath-cli/ Deprecated shim that re-exports path-cli -``` +# Query across all of them with a jq filter, regardless of which agent +# produced them: dead ends, expensive turns, and so on +path query 'map(select(.dead_end)) | length' +path query --source claude 'map(select(any(.change[].structural.token_usage; .input_tokens > 50000)))' -See each crate's README for library-level documentation. +# Share a session: interactive picker across every installed harness, +# sessions from the current project ranked first +path share -## Quick start +# Resume a shared session, in the original harness or a different one +path resume https://pathbase.dev/alex/pathstash/path-pr-42 +path resume claude- --harness codex -C /path/to/project +``` + +Each of those is built from lower-level plumbing commands you can +compose yourself: ```bash -# Build everything -cargo build --workspace +# Import a specific session into the local cache (~/.toolpath/documents/), +# or straight to stdout for shell composition +path p import claude --project /path/to/project --session +path p import codex --no-cache | path p render md --detail full -# Import a Toolpath document from this repo's git history (cached under ~/.toolpath/documents/) +# Derive from git history or a GitHub PR path p import git --repo . --branch main +path p import github https://github.com/owner/repo/pull/42 -# Visualize it +# Visualize the step DAG path p import git --repo . --branch main --no-cache | path p render dot | dot -Tpng -o graph.png -# Render as Markdown for an LLM -path p import git --repo . --branch main --no-cache | path p render md +# Project a cached document into a harness's on-disk format +path p export codex --input claude- --project /path/to/sandbox -# Import from a GitHub pull request -path p import github https://github.com/owner/repo/pull/42 +# Merge, validate, walk ancestry +path p merge doc1.json doc2.json --title "Release v2" +path p validate --input examples/step-01-minimal.json +path p query ancestors --input doc.json --step-id step-003 +``` -# Import from Claude conversation logs -path p import claude --project /path/to/project +## The format -# Import from Gemini CLI conversation logs -path p import gemini --project /path/to/project +Three objects, from a single change up to a release: -# Import from Codex CLI rollout files (most recent session by default) -path p import codex +| Object | What it represents | Example | +|-----------|-------------------------------------------|------------------------| +| **Step** | A single change to artifact(s) | One commit, one edit | +| **Path** | A sequence of steps with a base context | A PR, a coding session | +| **Graph** | A collection of related paths | A release | -# Import from opencode session database (most recent session by default) -path p import opencode +Steps record an actor (`human:alex`, `agent:claude-code`, `tool:rustfmt`), +changes (unified diffs and/or structural operations), intent, and +verification. They form a DAG via parent references, so abandoned +approaches stay in the record: a dead end is any step outside the +ancestry of `path.head`. -# List what's in the cache -path p cache ls +``` + +-- step-3a -- step-4a (dead end) +step-1 -- step-2 --+ + +-- step-3b -- step-4b -- step-5b (head) +``` -# Ingest new/changed agent sessions into the cache (all harnesses, or named ones) -path p cache sync -path p cache sync claude codex +A path can declare a **kind**, a versioned, immutable URI naming what +its steps mean. Agent sessions use +`https://toolpath.net/kinds/agent-coding-session/v1.1.0`; `path kind +agent-coding-session` prints the field reference. [RFC.md](RFC.md) is +the full specification and +[schema/toolpath.schema.json](schema/toolpath.schema.json) is the JSON +Schema. + +## Token accounting + +Providers report usage in incompatible ways: Claude streams cumulative +per-block snapshots, Codex re-emits stale cumulative counters, Gemini +reports reasoning as an additive sibling that's easy to drop, Copilot +reports only output per message. The `agent-coding-session` kind +normalizes all of it into three fields with strict semantics: + +- `token_usage` is the total for a message. It appears once per message + group, so summing it over a path gives the true session total. +- `attributed_token_usage` is a step's own spend. It is populated only + when the source genuinely reports per-step numbers (Codex does), and + never guessed from streaming snapshots. +- `breakdowns` holds informational decompositions, such as reasoning + tokens within output (Gemini, Codex, opencode). These are never added + into any total. + +Each provider's quirks (cumulative counters, repeated message totals, +zero-filled placeholders) are handled at derivation time, so +`path query` can aggregate spend across harnesses without knowing about +any of them. -# Export a cached document back into a Claude Code session -path p export claude --input claude- --project /path/to/resume +## Using the libraries -# Push a cached document to Pathbase -path auth login -path p export pathbase --input claude- +Everything the CLI does is a library call. Each source has its own crate +(see [Workspace](#workspace)); `toolpath` holds the core types and +`toolpath-convo` the provider-agnostic conversation model. -# Pull a path from Pathbase back into the local cache -# (full URL or bare `//` triple) -path p import pathbase https://pathbase.dev/alex/pathstash/path-pr-42 +### Deriving a document from a session -# Resume a Toolpath document into your coding agent of choice (interactive -# harness picker; project the session and exec the harness's resume command) -path resume https://pathbase.dev/alex/pathstash/path-pr-42 -path resume claude- --harness claude -C /path/to/project +```rust +use toolpath_claude::{ClaudeConvo, derive::{derive_path, DeriveConfig}}; + +let convo = ClaudeConvo::new(); +let conversation = convo.read_conversation("/path/to/project", "session-id")?; +let path = derive_path(&conversation, &DeriveConfig { + include_thinking: true, + ..Default::default() +}); +let json = toolpath::Graph::from_path(path).to_json_pretty()?; +``` -# Query the whole local cache with a jaq (jq) filter over wrapped steps -# (e.g. dead ends, or turns by an agent actor) -path query 'map(select(.dead_end))' -path query --input doc.json 'map(select(.step.actor | startswith("agent:")))' +Every provider crate follows the same shape: a reader for the harness's +on-disk format, a `ConversationProvider` impl producing a +provider-agnostic `ConversationView`, and the shared +`toolpath_convo::derive_path` mapping views to `Path` documents. +Supporting a new harness means implementing the view, not the mapping. +`toolpath-claude` and `toolpath-gemini` also ship filesystem watchers +(default-on `watcher` feature) for tailing live sessions. -# List bundled document kinds, or print a kind's schema (the field reference) -path kind -path kind agent-coding-session +### Core types -# Walk the ancestry of a step (plumbing) -path p query ancestors --input doc.json --step-id step-003 +```rust +use toolpath::{Step, Path, Base, Graph}; + +let step = Step::new("step-001", "human:alex", "2026-01-29T10:00:00Z") + .with_parent("step-000") + .with_raw_change("src/main.rs", "@@ -1,1 +1,1 @@\n-hello\n+world") + .with_intent("Fix greeting"); -# Merge multiple documents into a graph -path p merge doc1.json doc2.json --title "Release v2" --pretty +let path = Path::new( + "path-pr-42", + Some(Base::vcs("github:org/repo", "abc123")), + "step-001", +); -# Validate a document -path p validate --input examples/step-01-minimal.json +// Graph is the single root type of every Toolpath document. Wrap a single +// Path as a one-path Graph for serialization: +let graph = Graph::from_path(path); +let json = graph.to_json_pretty()?; ``` +### Query operations + +```rust +use toolpath::query; + +let ancestors = query::ancestors(&path.steps, &path.path.head); +let dead_ends = query::dead_ends(&path.steps, &path.path.head); +let by_actor = query::filter_by_actor(&path.steps, "agent:"); +let artifacts = query::all_artifacts(&path.steps); +``` + +### Rendering + +```rust +let dot = toolpath_dot::render(&doc, &toolpath_dot::RenderOptions::default()); +let md = toolpath_md::render(&doc, &toolpath_md::RenderOptions::default()); +``` + +## Using the format without the crates + +The format does not require Rust. [RFC.md](RFC.md) is the +specification, +[schema/toolpath.schema.json](schema/toolpath.schema.json) validates +documents in any language, and [examples/](examples/) holds 12 documents +covering steps, paths, and graphs. + ## CLI reference ``` @@ -158,7 +261,9 @@ path claude --project PATH --session ID gemini --project PATH --session UUID codex --session ID + copilot --session ID opencode --session ID + cursor --session UUID pi --project PATH --session ID [--base DIR] share # one-shot interactive picker + Pathbase upload resume # project a doc into a coding agent and exec --resume @@ -177,7 +282,9 @@ path claude [--project PATH] [--format ...] gemini [--project PATH] [--format ...] codex [--format ...] + copilot [--format ...] opencode [--project ID] [--format ...] + cursor [--project PATH] [--format ...] pi [--project PATH] [--base DIR] [--format ...] import # writes to ~/.toolpath/documents/ by default git --repo PATH --branch NAME[:START] [--base COMMIT] [--remote NAME] [--title TEXT] @@ -185,15 +292,23 @@ path claude [--project PATH] [--session ID] [--all] gemini [--project PATH] [--session UUID] [--all] codex [--session UUID|STEM] [--all] + copilot [--session ID] [--all] opencode [--session ID] [--all] [--project ID] [--no-snapshot-diffs] + cursor [--session UUID] [--all] [--project PATH] pi [--project PATH] [--session ID] [--all] [--base DIR] pathbase TRACE-ID-OR-URL [--url URL] # global: [--force] [--no-cache] export claude --input REF [--project DIR | --output FILE] + gemini --input REF [--project DIR | --output FILE] + codex --input REF [--project DIR | --output FILE] + opencode --input REF [--project DIR | --output FILE] + copilot --input REF [--project DIR | --output FILE] + cursor --input REF [--project DIR | --output FILE] + pi --input REF [--project DIR | --output FILE] pathbase --input REF [--url URL] cache - ls | rm CACHE-ID + ls | rm CACHE-ID | sync [TYPE...] render dot [--input FILE] [--output FILE] [--show-files] [--show-timestamps] md [--input FILE] [--output FILE] [--detail summary|full] [--front-matter] @@ -214,28 +329,23 @@ path Global: `--pretty` for formatted JSON output. -**Breaking** (pre-1.0). The previous top-level commands `path import`, -`path export`, `path cache`, `path list`, `path render`, `path merge`, -`path validate`, `path derive`, `path project`, `path incept`, and -`path track` were **removed** in `path-cli` 0.10.0 — they all now live -exclusively under `path p`. - ## Interactive selection When `path p import ` is run with no `--session` and stdin/stderr are TTYs, the CLI launches a fuzzy picker so you can pick a session by -topic. TAB selects multiple — the result is a `Graph`. `path share` and -`path resume` use the same picker. +topic. TAB selects multiple sessions, which produces a `Graph`. +`path share` and `path resume` use the same picker. Two backends, selected at runtime: - **External `fzf`** is preferred when it's on `$PATH` (so your fzf config and keybindings keep working). - **Embedded `skim`** (Rust fzf-clone) is shipped in the default build - and used when `fzf` isn't installed. Same `{1}`/`{2}` preview - placeholders, same column-selection grammar — visually similar UX. - Build with `--no-default-features` to drop it for a ~2 MB smaller - binary; without either backend the CLI prints a manual recipe. + and used when `fzf` isn't installed. It honors the same `{1}`/`{2}` + preview placeholders and column-selection grammar, so the UX is + close. Build with `--no-default-features` to drop it for a ~2 MB + smaller binary; without either backend the CLI prints a manual + recipe. Use the global `--picker auto|fzf|skim` flag to force a backend (default `auto`): @@ -247,13 +357,13 @@ path --picker fzf p import claude # error out if fzf isn't installed The picker leans on two machine-readable surfaces you can also use yourself: -- `path p list --format tsv` — one session per line, tab-delimited. +- `path p list --format tsv`: one session per line, tab-delimited. For project-keyed providers (claude, gemini, pi) the columns are `\t\t\t\t`. For single-keyed providers (codex, opencode): `\t\t\t\t`. `--format` defaults to `pretty` on a TTY and `tsv` when piped. -- `path show --…` — markdown summary for one session (the +- `path show --…`: markdown summary for one session (the picker's `--preview` command). Manual recipe (project-keyed; substitute `claude` for `gemini` or `pi`): @@ -276,66 +386,28 @@ path p list codex --format tsv \ | xargs -I{} path p import codex --session {} ``` -## Using the libraries - -### Core types - -```rust -use toolpath::{Step, Path, Base, Graph}; - -let step = Step::new("step-001", "human:alex", "2026-01-29T10:00:00Z") - .with_parent("step-000") - .with_raw_change("src/main.rs", "@@ -1,1 +1,1 @@\n-hello\n+world") - .with_intent("Fix greeting"); - -let path = Path::new( - "path-pr-42", - Some(Base::vcs("github:org/repo", "abc123")), - "step-001", -); - -// Graph is the single root type of every Toolpath document. Wrap a single -// Path as a one-path Graph for serialization: -let graph = Graph::from_path(path); -let json = graph.to_json_pretty()?; -``` - -### Query operations - -```rust -use toolpath::query; - -let ancestors = query::ancestors(&path.steps, &path.path.head); -let dead_ends = query::dead_ends(&path.steps, &path.path.head); -let by_actor = query::filter_by_actor(&path.steps, "agent:"); -let artifacts = query::all_artifacts(&path.steps); -``` - -### Git derivation - -```rust -use toolpath_git::{derive, DeriveConfig}; +## Workspace -let repo = git2::Repository::open(".")?; -let config = DeriveConfig { remote: "origin".into(), title: None, base: None }; -let doc = derive(&repo, &["main".into()], &config)?; ``` - -### DOT rendering - -```rust -use toolpath_dot::{render, RenderOptions}; - -let dot_string = render(&doc, &RenderOptions::default()); +crates/ + toolpath/ Core types, builders, query API + toolpath-convo/ Provider-agnostic conversation types, traits, and Toolpath-Path derivation + toolpath-git/ Derive from git repository history + toolpath-github/ Derive from GitHub pull requests + toolpath-claude/ Derive from Claude conversation logs + toolpath-gemini/ Derive from Gemini CLI conversation logs + toolpath-codex/ Derive from Codex CLI rollout files + toolpath-copilot/ Derive from GitHub Copilot CLI session logs (preview) + toolpath-opencode/ Derive from opencode SQLite databases + toolpath-cursor/ Derive from Cursor (IDE) state.vscdb bubble store + toolpath-pi/ Derive from Pi (pi.dev) agent sessions + toolpath-dot/ Graphviz DOT visualization + toolpath-md/ Markdown rendering for LLM consumption + pathbase-client/ Progenitor-derived typed client for the Pathbase HTTP API + path-cli/ Unified CLI (binary: path) ``` -### Markdown rendering - -```rust -use toolpath_md::{render, RenderOptions}; - -let md_string = render(&doc, &RenderOptions::default()); -``` +See each crate's README for library-level documentation. ## Documentation @@ -343,9 +415,9 @@ let md_string = render(&doc, &RenderOptions::default()); - [FAQ.md](FAQ.md) -- Design rationale and FAQ - [CHANGELOG.md](CHANGELOG.md) -- Release history - [schema/toolpath.schema.json](schema/toolpath.schema.json) -- JSON Schema -- [examples/](examples/) -- 11 example documents covering steps, paths, and graphs -- [docs/agents/formats/](docs/agents/formats/README.md) -- Reference for the on-disk - formats emitted by agents we derive from (Claude Code today; more as they land) +- [examples/](examples/) -- 12 example documents covering steps, paths, and graphs +- [DEVELOPMENT.md](DEVELOPMENT.md) -- Building and testing, plus our working notes + on the on-disk session formats of the agents we derive from ## Requirements