Skip to content

feat: S3 conditional writes on PutObject (If-Match / If-None-Match) — celld-compatible fencing - #20

Open
light-merlin-dark wants to merge 1 commit into
coollabsio:mainfrom
light-merlin-dark:feat/put-object-conditional-writes
Open

feat: S3 conditional writes on PutObject (If-Match / If-None-Match) — celld-compatible fencing#20
light-merlin-dark wants to merge 1 commit into
coollabsio:mainfrom
light-merlin-dark:feat/put-object-conditional-writes

Conversation

@light-merlin-dark

Copy link
Copy Markdown

What

Adds S3 conditional writes to PutObject: If-None-Match: * (create-if-absent) and If-Match: <etag> (compare-and-swap), failing with 412 PreconditionFailed. MaxIO already evaluated RFC 7232 conditional headers, but only in get_object/head_objectput_object ignored them silently.

Why

AWS S3 shipped conditional writes in Nov 2024, and coordination clients now build on them. The motivating one: celld (Deno Land) — self-hosted, distributed Durable Objects. Each cell is a single-threaded V8 isolate with a private SQLite database, replicated as LTX segments into an S3-compatible bucket you own; the bucket also coordinates the fleet: cell ownership is claimed with one atomic conditional write, and takeover/fencing depends on those conditions being enforced. The benefits of that model (hibernating per-entity state, ~zero-cost idle cells, RPO=0 failover between nodes) rest entirely on this one bucket feature.

celld's fencing doc certifies S3/R2/GCS/Azure/Tigris and disqualifies MinIO CE, B2, Hetzner, and Spaces for lacking conditional writes — with an explicit warning:

A store can also accept the conditional headers and not apply the condition, and that store fails late and silently, so test a store before you trust a fleet to it.

That is exactly stock MaxIO's behavior today, measured live:

Probe (aws CLI, --endpoint-url local MaxIO) S3-correct Stock MaxIO
PUT If-None-Match: * on existing key 412 200 — object overwritten
PUT If-Match: "stale-etag" 412 200

With this patch, a self-hosted MaxIO bucket becomes a valid celld coordination store — which also makes MaxIO the only self-hosted S3-compatible option on celld's qualified list.

How

  • FilesystemStorage::put_object_conditional: evaluates the preconditions against the current object while holding a per-key async lock, so check-and-write is atomic in-process and two racing conditional writers cannot both pass.
  • 412 (never 304) on PUT failures, per S3 semantics: If-None-Match: * fails when the object exists; If-Match fails when the object is absent or the ETag differs. *, quoted/unquoted ETags, and comma lists are handled.
  • API layer routes conditional PUTs through the new path; the unconditional path is byte-for-byte unchanged.

Evidence

  • 3 new integration tests: create-then-conflict, If-Match CAS (missing object + stale etag + success), and a concurrent create race asserting exactly one winner.
  • Full suite green: 41 unit + 163 integration.
  • End-to-end: a two-node celld fleet on loopback backed by a patched MaxIO bucket — SIGKILL of the owning node, peer acquired the lease in ~12s, restored from the LTX replica, and resumed with zero lost acknowledged writes (RPO=0).

Deliberately out of scope (follow-ups)

  • CopyObject and CompleteMultipartUpload conditionals (celld's fencing path uses plain PutObject).
  • Bounding growth of the per-key lock map.

Attribution

Built by Merlin (@light-merlin-dark) together with his AI coding agent (pi harness, k3-256k model), during an evaluation of running celld on fully self-hosted infrastructure. Happy to adjust semantics, naming, or test placement to match your conventions.

AWS S3 shipped conditional writes in Nov 2024. MaxIO already evaluated
RFC 7232 conditional headers, but only for GET/HEAD — PutObject ignored
If-Match / If-None-Match silently. Silent ignore is the unsafe case for
coordination clients that fence ownership on conditional writes: two
writers can both believe they won a compare-and-swap.

One such client is celld (https://celld.dev, denoland): self-hosted,
distributed Durable Objects — V8 isolates, one private SQLite per cell,
LTX replication into an S3-compatible bucket that also coordinates cell
ownership through conditional creates and compare-and-swaps. The celld
fencing doc disqualifies MinIO CE, B2, Hetzner, and Spaces for lacking
exactly this feature, and warns that a store which accepts the headers
without applying them fails late and silently.

This change:

- Adds FilesystemStorage::put_object_conditional: evaluates If-Match /
  If-None-Match against the current object while holding a per-key async
  lock, so check-and-write is atomic in-process and two racing
  create-if-absent writers cannot both succeed.
- Maps failure to 412 PreconditionFailed (never 304 on PUT), matching
  S3 conditional-write semantics: If-None-Match: * fails when the object
  exists; If-Match fails when the object is absent or its ETag differs.
- Routes conditional PUTs in the API layer through the new path; the
  unconditional path is unchanged.

Verified live: all probes pass against a local server, and a two-node
celld fleet backed by a patched MaxIO bucket survived a SIGKILL
ownership takeover with zero lost acknowledged writes (RPO=0).

Tests: three integration tests (create-then-conflict, If-Match etag
compare-and-swap incl. missing-object and stale-etag failure, and a
concurrent create race asserting exactly one winner). Full suite green:
41 unit + 163 integration.

Follow-ups deliberately out of scope: conditionals on CopyObject and
CompleteMultipartUpload, and bounding the put-locks map growth.

Built by Merlin (light-merlin-dark) with his AI coding agent
(pi harness, k3-256k model) during an evaluation of celld on
self-hosted infrastructure.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant