feat: S3 conditional writes on PutObject (If-Match / If-None-Match) — celld-compatible fencing - #20
Open
light-merlin-dark wants to merge 1 commit into
Conversation
AWS S3 shipped conditional writes in Nov 2024. MaxIO already evaluated RFC 7232 conditional headers, but only for GET/HEAD — PutObject ignored If-Match / If-None-Match silently. Silent ignore is the unsafe case for coordination clients that fence ownership on conditional writes: two writers can both believe they won a compare-and-swap. One such client is celld (https://celld.dev, denoland): self-hosted, distributed Durable Objects — V8 isolates, one private SQLite per cell, LTX replication into an S3-compatible bucket that also coordinates cell ownership through conditional creates and compare-and-swaps. The celld fencing doc disqualifies MinIO CE, B2, Hetzner, and Spaces for lacking exactly this feature, and warns that a store which accepts the headers without applying them fails late and silently. This change: - Adds FilesystemStorage::put_object_conditional: evaluates If-Match / If-None-Match against the current object while holding a per-key async lock, so check-and-write is atomic in-process and two racing create-if-absent writers cannot both succeed. - Maps failure to 412 PreconditionFailed (never 304 on PUT), matching S3 conditional-write semantics: If-None-Match: * fails when the object exists; If-Match fails when the object is absent or its ETag differs. - Routes conditional PUTs in the API layer through the new path; the unconditional path is unchanged. Verified live: all probes pass against a local server, and a two-node celld fleet backed by a patched MaxIO bucket survived a SIGKILL ownership takeover with zero lost acknowledged writes (RPO=0). Tests: three integration tests (create-then-conflict, If-Match etag compare-and-swap incl. missing-object and stale-etag failure, and a concurrent create race asserting exactly one winner). Full suite green: 41 unit + 163 integration. Follow-ups deliberately out of scope: conditionals on CopyObject and CompleteMultipartUpload, and bounding the put-locks map growth. Built by Merlin (light-merlin-dark) with his AI coding agent (pi harness, k3-256k model) during an evaluation of celld on self-hosted infrastructure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds S3 conditional writes to
PutObject:If-None-Match: *(create-if-absent) andIf-Match: <etag>(compare-and-swap), failing with412 PreconditionFailed. MaxIO already evaluated RFC 7232 conditional headers, but only inget_object/head_object—put_objectignored them silently.Why
AWS S3 shipped conditional writes in Nov 2024, and coordination clients now build on them. The motivating one: celld (Deno Land) — self-hosted, distributed Durable Objects. Each cell is a single-threaded V8 isolate with a private SQLite database, replicated as LTX segments into an S3-compatible bucket you own; the bucket also coordinates the fleet: cell ownership is claimed with one atomic conditional write, and takeover/fencing depends on those conditions being enforced. The benefits of that model (hibernating per-entity state, ~zero-cost idle cells, RPO=0 failover between nodes) rest entirely on this one bucket feature.
celld's fencing doc certifies S3/R2/GCS/Azure/Tigris and disqualifies MinIO CE, B2, Hetzner, and Spaces for lacking conditional writes — with an explicit warning:
That is exactly stock MaxIO's behavior today, measured live:
--endpoint-urllocal MaxIO)If-None-Match: *on existing keyIf-Match: "stale-etag"With this patch, a self-hosted MaxIO bucket becomes a valid celld coordination store — which also makes MaxIO the only self-hosted S3-compatible option on celld's qualified list.
How
FilesystemStorage::put_object_conditional: evaluates the preconditions against the current object while holding a per-key async lock, so check-and-write is atomic in-process and two racing conditional writers cannot both pass.If-None-Match: *fails when the object exists;If-Matchfails when the object is absent or the ETag differs.*, quoted/unquoted ETags, and comma lists are handled.Evidence
SIGKILLof the owning node, peer acquired the lease in ~12s, restored from the LTX replica, and resumed with zero lost acknowledged writes (RPO=0).Deliberately out of scope (follow-ups)
CopyObjectandCompleteMultipartUploadconditionals (celld's fencing path uses plainPutObject).Attribution
Built by Merlin (@light-merlin-dark) together with his AI coding agent (pi harness,
k3-256kmodel), during an evaluation of running celld on fully self-hosted infrastructure. Happy to adjust semantics, naming, or test placement to match your conventions.