Skip to content

Share and resume to S3, an S3-compatible endpoint, or a folder - #168

Open
akesling wants to merge 2 commits into
mainfrom
alex/share-targets
Open

Share and resume to S3, an S3-compatible endpoint, or a folder#168
akesling wants to merge 2 commits into
mainfrom
alex/share-targets

Conversation

@akesling

@akesling akesling commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Adds S3 (and any S3-compatible endpoint, and plain folders) as a share
destination alongside Pathbase, configured once instead of per call.

path auth s3 login                      # credentials, if you're using a bucket
path target ~/Dropbox/toolpath-traces   # a folder — no credentials needed
path target s3://my-bucket/traces       # or a bucket; checked before it's stored
path share                              # → wherever you pointed it
path target                             # what's in effect, and why

path share --to /tmp/scratch            # override for one call
path share --to pathbase

path resume ~/Dropbox/toolpath-traces   # lists what you've shared, pick one
path resume s3://my-bucket/traces/2026-08-07-fix-the-parser-claude-abc.json

Where shares go

One setting in one place: default_target in ~/.toolpath/config.json.
Not inside either credential file — if each backend's own config could
claim to be the default, "where does my next share go?" becomes a
precedence puzzle the day a third backend appears.

path target is a top-level verb rather than a subcommand of auth,
because it's a persistent setting and it isn't authentication.
path auth s3 keeps the credentials, which genuinely are.

Resolution: --to$TOOLPATH_SHARE_TARGET → the stored target →
Pathbase. Nothing is inferred from which credentials happen to exist: a
share that silently changes destination is a data-egress bug, not a
convenience. The one guard sits at the bottom — with S3 credentials
stored, no Pathbase session, and no target set, share refuses rather
than falling through to the anonymous public Pathbase endpoint.

The Pathbase-only flags (--anon, --repo, --public, --url,
--name) select Pathbase on their own, overriding an object target;
pairing one with an explicit object --to is an error rather than a
silent resolution.

Targets are verified when set

Setting a target does what a share does — writes a small
.toolpath-access-check object, then removes it — and refuses to store
one that fails. A target is set once and used many times, so a wrong one
that survives costs a session pick and a derivation later, at the moment
you wanted a result rather than a setup step. A listing would only prove
s3:ListBucket, which isn't the permission a share needs.

--no-verify stores unchecked, for a bucket you haven't created yet or
a laptop that's offline; it prints (not verified) so the state is
visible. Folders take the same path, which also creates them.

Destinations you can read back out of

Documents land at <date>-<topic>-<cache-id>.json — e.g.
2026-08-07-add-s3-support-to-share-claude-6f2a1c9e.json. Every
component is a pure function of the document, so re-sharing a session
that grew overwrites its own object instead of leaving near-duplicates;
the date is the session's first step, so it doesn't move as the
conversation continues.

path resume <bucket|prefix|folder> lists the destination and offers a
picker — the counterpart to share's picker across every harness. Rows
are built from object names alone, so browsing a hundred shared sessions
costs one list request and zero downloads. A destination holding one
document skips the picker.

Two traps closed

  • A bare relative target (my-bucket/traces) is rejected rather than
    quietly creating ./my-bucket/traces and reporting success. It's
    overwhelmingly a bucket name typed from memory; ./my-bucket/traces
    opts in explicitly.
  • memory:// is rejected: a fresh per-process store, so anything
    "shared" there is gone before the command exits.

Implementation

Transport is the object_store
crate — one code path for AWS S3, any S3-compatible endpoint (R2, MinIO,
Ceph, B2), and file://. It adds 18 crates but no new crypto backend
(aws-lc-rs was already in the lock via rustls).

  • store.rs splits where a document goes (Destination, ObjectUri,
    ObjectName — pure URL parsing and naming) from how to reach it
    (S3Settings at ~/.toolpath/s3.json, connection + credentials
    only, so one stored credential serves any number of buckets).
  • target.rs sits above and owns the Pathbase-vs-object choice.
  • Preflights cap timeouts and retries — object_store's default ten
    retries would otherwise multiply a 5s connect timeout into a ~50s
    stall — and its retry epilogue is stripped from user-facing errors.
  • Config files under the config dir now share
    config::write_private_json / read_private_json, which own the
    0600-file / 0700-parent story.

Plumbing counterparts: p export object / p import object, both
aliased s3.

Testing

scripts/quality_gates.sh 8/8. 413 unit + 154 integration tests for
path-cli (was 358 + 121).

The folder backend isn't only a testing affordance, but it's what makes
the tests real: share and resume round-trip end to end through the
actual object_store code path with no network and no mock HTTP server.
Verification failures are tested deterministically offline by pointing
at http://127.0.0.1:1.

wasm32-unknown-emscripten still builds — the new modules are cfg-gated
like cmd_pathbase.

Note

path-cli and the toolpath-cli shim bump to 0.17.0. Nothing here was
released, so --s3--to and the s3.json shape changed freely
during review; there's no migration to write.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

`path share` uploaded to Pathbase and nowhere else. It can now be
pointed at an S3 bucket or a plain folder once, and stay there.

Designating a target:

    path auth default ~/Dropbox/toolpath-traces   # no credentials needed
    path auth default s3://my-bucket/traces
    path auth default pathbase
    path auth default                             # what's in effect, and why

The value lives in one place, `default_target` in
`~/.toolpath/config.json`, so "where does my next share go?" has one
answer rather than a precedence puzzle across each backend's own
config. A scheme-less value is a local path; a bucket is spelled
`s3://`. Folder targets persist as `file://` URLs so a stored default
can't drift with the working directory, but display as plain paths so
the printed location pastes straight into `path resume`.

Resolution: `--to`, then `$TOOLPATH_SHARE_TARGET`, then the stored
default, then Pathbase. Nothing is inferred from which credentials
happen to exist — a share that silently changes destination is a
data-egress bug, not a convenience. The one guard sits at the bottom of
the order: with S3 credentials stored, no Pathbase session, and no
default set, share refuses rather than falling through to the anonymous
public endpoint.

`path share --to <target>` overrides for one call. The Pathbase-only
flags (`--anon`, `--repo`, `--public`, `--url`, `--name`) select
Pathbase on their own, overriding an object default; pairing one with an
explicit object `--to` is an error rather than a silent resolution.
Target selection splits into a pure `plan_target` that runs before the
harness scan and the picker, and an `open_target` that probes
credentials before the picker fires — so neither a typo nor a bad
credential costs a session pick.

`path resume s3://bucket/key.json` is the inverse, caching downloads
under an `s3-<bucket>-<key>` id so `--force` and `--no-cache` behave as
they do for Pathbase. A document shared to a folder resumes from its
plain path, which already worked.

`path auth s3 login` stores connection settings and credentials at
`~/.toolpath/s3.json` (0600) — deliberately not a destination, so one
stored credential serves any number of buckets and a folder target needs
no `auth s3` at all. It merges rather than replaces, and prompts without
echoing the secret. Stored values beat the AWS environment; with
neither, the AWS credential chain still applies.

Plumbing: `path p export object` / `path p import object`, both aliased
`s3`. With no `--to`, export writes wherever `path share` would.

Transport is the `object_store` crate, so one code path serves AWS S3,
any S3-compatible endpoint (R2, MinIO, Ceph, B2), and `file://` for a
folder. The folder backend is what the tests round-trip against, so
share and resume are covered end-to-end with no network and no mock
HTTP server.

`crates/path-cli/src/store.rs` splits where-to-write (`Destination`,
`ObjectUri` — pure URL parsing and key layout) from how-to-authenticate
(`S3Settings`); `crates/path-cli/src/target.rs` sits above it and owns
the Pathbase-vs-object choice. Config files under the config dir now
share `config::write_private_json` / `read_private_json`, which own the
0600-file / 0700-parent story.
Sharing to a destination was a one-way trip, and pointing at one was an
act of faith. Three changes, in the order they bite.

## Destinations you can read back out of

`share` had a picker across every harness; resuming from where you
shared to had nothing, and the object was named `<cache-id>.json` — an
opaque uuid. Twenty shares later a folder was twenty blobs you could
only read by `cat`-ing JSON.

A shared document now lands at `<date>-<topic>-<cache-id>.json`, e.g.
`2026-08-07-add-s3-support-to-share-claude-6f2a1c9e.json`. Every
component is a pure function of the document, so a re-share overwrites
its own object rather than leaving near-duplicates; the date is the
session's *first* step, so it doesn't move as the conversation grows.

`path resume <bucket|prefix|folder>` lists what is there and offers a
picker — the counterpart to `share`'s. Rows come from object names
alone, so browsing a hundred sessions costs one list request and zero
downloads; that is what legible names buy. A destination holding one
document skips the picker, and anything ending in `.json` is still
fetched directly.

## Targets checked when they're set

Setting a target stored it and hoped. A typo'd bucket or a missing
permission surfaced in the middle of a share — after a session pick and
a derivation, at the moment the user wanted a result rather than a setup
step. A target is set once and used many times, so configuration time is
where it should fail.

`path target <s3-or-folder>` now verifies before storing, by doing what
a share does: write a small `.toolpath-access-check` object, then remove
it. A listing would only prove `s3:ListBucket`, which isn't the
permission a share needs; only a write answers the question actually
being asked. Verification failure means nothing is stored. `--no-verify`
is the escape hatch for a bucket that doesn't exist yet or a laptop
that's offline, and it says `(not verified)` when it stores one.

Folders take the same path, which also creates them; a credential that
can write but not delete leaves the probe behind and gets a note rather
than a failure.

Both preflights bound their own runtime. Per-request timeouts alone
don't: object_store's default ten retries multiply them, turning a 5s
connect timeout into a ~50s stall on a hanging endpoint. `open_for_check`
builds the S3 store directly to cap retries at 2, since `parse_url_opts`
has no key for retry policy. Its retry epilogue ("after 10 retries,
max_retries: 10, retry_timeout: 180s") is stripped from user-facing
errors, and an unreachable endpoint now says so instead of quoting a
transport trace. Share keeps a weaker list-based probe: the upload is
about to happen and will report its own failure, so all that check needs
to buy is not wasting a derivation.

## Naming and guards

`path target` replaces `path auth default`. "Auth" means credentials;
where shares go is a persistent setting, not authentication, and nobody
would look for it under `auth`. `path auth s3` keeps the credentials.

A bare relative value (`my-bucket/traces`) is rejected instead of
silently creating `./my-bucket/traces` and reporting success. It is
overwhelmingly a bucket name typed from memory; `./my-bucket/traces`
opts in explicitly.

`memory://` is no longer accepted: a fresh per-process store, so
anything "shared" there is gone before the command exits.

Plumbing renamed to match: `p export object` / `p import object`, both
aliased `s3`.
@github-actions

Copy link
Copy Markdown

🔍 Preview deployed: https://ace50488.toolpath.pages.dev

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants