Share and resume to S3, an S3-compatible endpoint, or a folder - #168
Open
akesling wants to merge 2 commits into
Open
Share and resume to S3, an S3-compatible endpoint, or a folder#168akesling wants to merge 2 commits into
akesling wants to merge 2 commits into
Conversation
`path share` uploaded to Pathbase and nowhere else. It can now be
pointed at an S3 bucket or a plain folder once, and stay there.
Designating a target:
path auth default ~/Dropbox/toolpath-traces # no credentials needed
path auth default s3://my-bucket/traces
path auth default pathbase
path auth default # what's in effect, and why
The value lives in one place, `default_target` in
`~/.toolpath/config.json`, so "where does my next share go?" has one
answer rather than a precedence puzzle across each backend's own
config. A scheme-less value is a local path; a bucket is spelled
`s3://`. Folder targets persist as `file://` URLs so a stored default
can't drift with the working directory, but display as plain paths so
the printed location pastes straight into `path resume`.
Resolution: `--to`, then `$TOOLPATH_SHARE_TARGET`, then the stored
default, then Pathbase. Nothing is inferred from which credentials
happen to exist — a share that silently changes destination is a
data-egress bug, not a convenience. The one guard sits at the bottom of
the order: with S3 credentials stored, no Pathbase session, and no
default set, share refuses rather than falling through to the anonymous
public endpoint.
`path share --to <target>` overrides for one call. The Pathbase-only
flags (`--anon`, `--repo`, `--public`, `--url`, `--name`) select
Pathbase on their own, overriding an object default; pairing one with an
explicit object `--to` is an error rather than a silent resolution.
Target selection splits into a pure `plan_target` that runs before the
harness scan and the picker, and an `open_target` that probes
credentials before the picker fires — so neither a typo nor a bad
credential costs a session pick.
`path resume s3://bucket/key.json` is the inverse, caching downloads
under an `s3-<bucket>-<key>` id so `--force` and `--no-cache` behave as
they do for Pathbase. A document shared to a folder resumes from its
plain path, which already worked.
`path auth s3 login` stores connection settings and credentials at
`~/.toolpath/s3.json` (0600) — deliberately not a destination, so one
stored credential serves any number of buckets and a folder target needs
no `auth s3` at all. It merges rather than replaces, and prompts without
echoing the secret. Stored values beat the AWS environment; with
neither, the AWS credential chain still applies.
Plumbing: `path p export object` / `path p import object`, both aliased
`s3`. With no `--to`, export writes wherever `path share` would.
Transport is the `object_store` crate, so one code path serves AWS S3,
any S3-compatible endpoint (R2, MinIO, Ceph, B2), and `file://` for a
folder. The folder backend is what the tests round-trip against, so
share and resume are covered end-to-end with no network and no mock
HTTP server.
`crates/path-cli/src/store.rs` splits where-to-write (`Destination`,
`ObjectUri` — pure URL parsing and key layout) from how-to-authenticate
(`S3Settings`); `crates/path-cli/src/target.rs` sits above it and owns
the Pathbase-vs-object choice. Config files under the config dir now
share `config::write_private_json` / `read_private_json`, which own the
0600-file / 0700-parent story.
Sharing to a destination was a one-way trip, and pointing at one was an
act of faith. Three changes, in the order they bite.
## Destinations you can read back out of
`share` had a picker across every harness; resuming from where you
shared to had nothing, and the object was named `<cache-id>.json` — an
opaque uuid. Twenty shares later a folder was twenty blobs you could
only read by `cat`-ing JSON.
A shared document now lands at `<date>-<topic>-<cache-id>.json`, e.g.
`2026-08-07-add-s3-support-to-share-claude-6f2a1c9e.json`. Every
component is a pure function of the document, so a re-share overwrites
its own object rather than leaving near-duplicates; the date is the
session's *first* step, so it doesn't move as the conversation grows.
`path resume <bucket|prefix|folder>` lists what is there and offers a
picker — the counterpart to `share`'s. Rows come from object names
alone, so browsing a hundred sessions costs one list request and zero
downloads; that is what legible names buy. A destination holding one
document skips the picker, and anything ending in `.json` is still
fetched directly.
## Targets checked when they're set
Setting a target stored it and hoped. A typo'd bucket or a missing
permission surfaced in the middle of a share — after a session pick and
a derivation, at the moment the user wanted a result rather than a setup
step. A target is set once and used many times, so configuration time is
where it should fail.
`path target <s3-or-folder>` now verifies before storing, by doing what
a share does: write a small `.toolpath-access-check` object, then remove
it. A listing would only prove `s3:ListBucket`, which isn't the
permission a share needs; only a write answers the question actually
being asked. Verification failure means nothing is stored. `--no-verify`
is the escape hatch for a bucket that doesn't exist yet or a laptop
that's offline, and it says `(not verified)` when it stores one.
Folders take the same path, which also creates them; a credential that
can write but not delete leaves the probe behind and gets a note rather
than a failure.
Both preflights bound their own runtime. Per-request timeouts alone
don't: object_store's default ten retries multiply them, turning a 5s
connect timeout into a ~50s stall on a hanging endpoint. `open_for_check`
builds the S3 store directly to cap retries at 2, since `parse_url_opts`
has no key for retry policy. Its retry epilogue ("after 10 retries,
max_retries: 10, retry_timeout: 180s") is stripped from user-facing
errors, and an unreachable endpoint now says so instead of quoting a
transport trace. Share keeps a weaker list-based probe: the upload is
about to happen and will report its own failure, so all that check needs
to buy is not wasting a derivation.
## Naming and guards
`path target` replaces `path auth default`. "Auth" means credentials;
where shares go is a persistent setting, not authentication, and nobody
would look for it under `auth`. `path auth s3` keeps the credentials.
A bare relative value (`my-bucket/traces`) is rejected instead of
silently creating `./my-bucket/traces` and reporting success. It is
overwhelmingly a bucket name typed from memory; `./my-bucket/traces`
opts in explicitly.
`memory://` is no longer accepted: a fresh per-process store, so
anything "shared" there is gone before the command exits.
Plumbing renamed to match: `p export object` / `p import object`, both
aliased `s3`.
|
🔍 Preview deployed: https://ace50488.toolpath.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds S3 (and any S3-compatible endpoint, and plain folders) as a share
destination alongside Pathbase, configured once instead of per call.
Where shares go
One setting in one place:
default_targetin~/.toolpath/config.json.Not inside either credential file — if each backend's own config could
claim to be the default, "where does my next share go?" becomes a
precedence puzzle the day a third backend appears.
path targetis a top-level verb rather than a subcommand ofauth,because it's a persistent setting and it isn't authentication.
path auth s3keeps the credentials, which genuinely are.Resolution:
--to→$TOOLPATH_SHARE_TARGET→ the stored target →Pathbase. Nothing is inferred from which credentials happen to exist: a
share that silently changes destination is a data-egress bug, not a
convenience. The one guard sits at the bottom — with S3 credentials
stored, no Pathbase session, and no target set,
sharerefuses ratherthan falling through to the anonymous public Pathbase endpoint.
The Pathbase-only flags (
--anon,--repo,--public,--url,--name) select Pathbase on their own, overriding an object target;pairing one with an explicit object
--tois an error rather than asilent resolution.
Targets are verified when set
Setting a target does what a share does — writes a small
.toolpath-access-checkobject, then removes it — and refuses to storeone that fails. A target is set once and used many times, so a wrong one
that survives costs a session pick and a derivation later, at the moment
you wanted a result rather than a setup step. A listing would only prove
s3:ListBucket, which isn't the permission a share needs.--no-verifystores unchecked, for a bucket you haven't created yet ora laptop that's offline; it prints
(not verified)so the state isvisible. Folders take the same path, which also creates them.
Destinations you can read back out of
Documents land at
<date>-<topic>-<cache-id>.json— e.g.2026-08-07-add-s3-support-to-share-claude-6f2a1c9e.json. Everycomponent is a pure function of the document, so re-sharing a session
that grew overwrites its own object instead of leaving near-duplicates;
the date is the session's first step, so it doesn't move as the
conversation continues.
path resume <bucket|prefix|folder>lists the destination and offers apicker — the counterpart to
share's picker across every harness. Rowsare built from object names alone, so browsing a hundred shared sessions
costs one list request and zero downloads. A destination holding one
document skips the picker.
Two traps closed
my-bucket/traces) is rejected rather thanquietly creating
./my-bucket/tracesand reporting success. It'soverwhelmingly a bucket name typed from memory;
./my-bucket/tracesopts in explicitly.
memory://is rejected: a fresh per-process store, so anything"shared" there is gone before the command exits.
Implementation
Transport is the
object_storecrate — one code path for AWS S3, any S3-compatible endpoint (R2, MinIO,
Ceph, B2), and
file://. It adds 18 crates but no new crypto backend(
aws-lc-rswas already in the lock via rustls).store.rssplits where a document goes (Destination,ObjectUri,ObjectName— pure URL parsing and naming) from how to reach it(
S3Settingsat~/.toolpath/s3.json, connection + credentialsonly, so one stored credential serves any number of buckets).
target.rssits above and owns the Pathbase-vs-object choice.object_store's default tenretries would otherwise multiply a 5s connect timeout into a ~50s
stall — and its retry epilogue is stripped from user-facing errors.
config::write_private_json/read_private_json, which own the0600-file / 0700-parent story.
Plumbing counterparts:
p export object/p import object, bothaliased
s3.Testing
scripts/quality_gates.sh8/8. 413 unit + 154 integration tests forpath-cli(was 358 + 121).The folder backend isn't only a testing affordance, but it's what makes
the tests real: share and resume round-trip end to end through the
actual
object_storecode path with no network and no mock HTTP server.Verification failures are tested deterministically offline by pointing
at
http://127.0.0.1:1.wasm32-unknown-emscriptenstill builds — the new modules are cfg-gatedlike
cmd_pathbase.Note
path-cliand thetoolpath-clishim bump to 0.17.0. Nothing here wasreleased, so
--s3→--toand thes3.jsonshape changed freelyduring review; there's no migration to write.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.