fix: harden sync error handling, unwrap the gateway envelope, and report RLS denials - #64
Open
marcobambini wants to merge 11 commits into
Open
fix: harden sync error handling, unwrap the gateway envelope, and report RLS denials#64marcobambini wants to merge 11 commits into
marcobambini wants to merge 11 commits into
Conversation
cloudsync_payload_apply now keeps the first error instead of letting a
later successful row overwrite it, so a lock-blocked apply reports the
failure rather than returning quietly. Test 39 encoded the old lenient
behaviour and aborted the script under ON_ERROR_STOP; it now tolerates
the error the way tests 41, 46 and 53 already do, and still asserts the
row kept its old value.
Also restores the changelog workflow's v-prefixed tag filter. Widening
it made the workflow fire but it then failed: the called workflow
derives the version with ${GITHUB_REF#refs/tags/v} and rejects our
unprefixed tags. Fixing that needs a change in changelog-action, so the
filter goes back and the note records why, keeping the manual run.
Adds the 1.1.4 changelog entry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
Key lookups are now scoped to one object, but six reads on the send and
status path take their key from a raw response body, where the gateway
wraps every success payload in {"data": ...}: the upload URL, the three
sync-state fields, and both failure stages. Root-scoped, they stopped
resolving, so every send failed with "missing 'url' in upload response"
while the local suites stayed green.
Those readers now resolve the payload first, the way the /check path
already does, keeping lookups scoped to a single object. Chunk objects
sliced out of chunks[] and legacy unwrapped bodies fall through
unchanged.
The new test covers the documented shapes in both directions: an
enveloped status payload with gaps and failures, a legacy unwrapped
body, an enveloped url staying invisible to a root-scoped read, and a
sliced chunk object resolving directly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
The opportunistic failures.check read takes its key from the raw /check
response body, which the gateway wraps in {"data": ...}. Root-scoped it
returned NULL, so a server-reported check failure was never surfaced —
silently, since the field is optional.
This was the one site missed by the previous commit; every remaining
lookup now reads either an unwrapped payload, a chunk object sliced out
of chunks[], or an already-extracted sub-object.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
andinux
force-pushed
the
pg-fixes11092026
branch
from
September 11, 2026 18:13
90565fc to
a1f3ea3
Compare
…ursor
A denial suppressed the receive checkpoint, which gave neither progress
nor a signal. The rows are permanently not this site's to hold, so the
next check re-delivered them, they were denied again, and nothing ever
surfaced to break the cycle. One denied row also stalled every later
change behind it.
Worse in a chunked batch: policy_denied was a local of one apply call,
but each chunk is a separate call. A denial in a non-final chunk
suppressed a checkpoint that was already a no-op (non-final chunks pass
CHECKPOINT_NONE), the flag died with the call, and the final chunk
advanced the cursor past the denied rows — dropping them silently, the
shape this was meant to prevent.
Denied entries are now counted, skipped, and the cursor advances. The
count accumulates across the drain on the context and is reported as
receive.denied, so discarding stays visible: a non-zero denied with zero
rows is the shape of an apply connection with no session identity.
Deliberately not an error, even when every row is denied: a single-row
payload belonging to another user is denied in full and is a correct
outcome, which tests 27 and 29 already assert.
Test 27 now checks the cursor moves past a denied apply. Verified it
fails ("left the checkpoint at 4, expected > 4") with the old
suppression restored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
Reading the row back is part of writing a block column: without its text
there is nothing to split, and a row its own session cannot select could
not sync anyway. So the failure stays fatal — but it has to be legible.
Five paths across local_block_insert and block_migrate_existing_rows
returned a bare code with no message. databasevm_step clears the error
text on entry, so PG's cloudsync_insert raised the user's INSERT with an
empty errmsg — the "not an error" confusion the comment in
cloudsync_payload_apply already warns about. Two of them returned the
raw negative from pk_decode_prikey, which is not a DBRES value at all
(-1 is neither OK nor any known error), so callers testing for a known
code fell through.
Each now names the table and the column, and the unreadable-row case
points at the SELECT policy. Aborting the migration stays recoverable:
its Phase 1 scan skips already-migrated rows, so a re-run resumes.
Test 57 covers the unreadable case end to end. Verified it fails
("reported a blank error") with the message removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
CURLOPT_TIMEOUT was applied to both pooled handles, but they carry very different traffic. An S3 presigned URL is not one of the API endpoints, so artifact GETs and PUTs used the artifact handle and inherited the 300-second cap: 256 MiB (the new decompressed limit) inside 300s demands a sustained ~875 KB/s, so a healthy transfer on a slow link was killed mid-flight and reported as a timeout, indistinguishable from a dead server, on every retry. API calls keep the elapsed-time cap, which is the right shape for small JSON. Artifact transfers now abort after 60 seconds below 1 KB/s, which also catches a real stall five times sooner than the 300s cap did, and keeps a 1-hour backstop because there is no progress callback to cancel a transfer that trickles just fast enough to stay alive. The stalled-server test covered the artifact handle only (every endpoint in its stub context is NULL). It now runs both policies. Verified the artifact case fails when the low-speed options are removed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
error_message was snapshotted only on the way into cleanup, so arriving with rc OK left it empty. A database_commit_savepoint that then failed set rc but its message — a deadlock or serialization failure, say — was replaced by the generic "Unable to flush pending changes". Snapshot after the commit attempt as well, before the rollback, which touches the error state itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
…rows written
Three consequences of the denial work, which only covered the row path.
The v3 fragment path had no denial branch, so a denied oversize value
still returned POLICY_DENIED up through network_apply_payload_buffer,
became a receive error, and aborted the whole drain — skipping the rest
of the payload, stalling the cursor, and leaving the staged fragments
undeleted to churn until stale cleanup. It gets the same treatment as
the row path: count, skip, checkpoint. A denied value's fragments are as
finished as an applied one's, so they are dropped too; any other failure
still keeps them for the retry.
receive.rows counted denied entries, because it came from the payload's
entry count. An all-denied receive reported {"rows":N,"denied":N} while
tables was correctly empty, and the diagnostic the CHANGELOG describes —
a non-zero denied with a zero rows — could never occur. API.md has
always documented the field as rows "received and applied", so the
number now matches its own contract.
Fixed in the drain rather than in the apply return value: that return
counts payload entries including denied ones, which tests 27 and 29
asserts as part of the SQL surface. Both paths accumulate an accurate
applied count on the context instead, next to the denied one.
API.md documents denied in both receive shapes and all six samples.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
…sable 067f3fb made the v3 fragment path skip a denied value and checkpoint past it, matching the row path. That is wrong on PostgreSQL: a denial leaves the transaction unusable, so the next statement — the checkpoint write — fails with "buffer pin is not owned by resource owner TopTransaction". The symptom is worse than the behaviour it replaced, which at least reported the denial cleanly. Neither a savepoint around the per-value apply nor dropping the staged-fragment delete recovers the state; both were tried and both still fail. The row path is safe only because merge_flush_pending rolls back its own savepoint around the write. So the v3 path goes back to failing on a denial, and the comment records why. The gap the revert leaves open is real and now covered: the cursor does not advance, so a denied oversize value is re-delivered on every drain. 58_v3_denied_checkpoint.sql builds a genuine fragmented payload, applies it under a WITH CHECK policy, and pins that behaviour, with a note to flip the assertion when the apply leaves a recoverable state. Closing it properly needs the fragment apply to roll back to a savepoint the way merge_flush_pending does, which is more than a follow-up to the reporting work. The CHANGELOG now scopes the skip-and-advance claim to the row path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
The denial path no longer reaches it, so "already applied or permanently denied" describes a state that cannot occur. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release 1.1.4. The substance is @marcobambini's audit hardening (
e0ebb77); everything after it is follow-up work from running that hardening against CI and then reviewing it.Sync error handling (
e0ebb77)The headline fix is a silent data-loss path:
cloudsync_payload_applyreassignedrcon every row, so an error on one row was overwritten by any later success. A failed write could report success and advance the receive checkpoint, dropping the changes for good. Errors are now sticky.Alongside it: little-endian IEEE754 byte order made explicit for PK doubles on every architecture, per-statement SPI tuple-table ownership, block-LWW rollback on write failure, 64-bit clocks above
UINT32_MAX, network deadlines, a 256 MiB decompressed-payload cap, and Nodeia32rejection. Details indocs/internal/audit-regressions.md.Gateway data envelope (
c6200a8,a1f3ea3)Scoping key lookups to a single object was correct, but seven reads take their key from a raw response body, and per
API.mdthe gateway wraps every success payload in{"data": <payload>}. Root-scoped, they stopped resolving — every send failed withmissing 'url' in upload responsewhile all local suites stayed green, because no fixture used the real shape.Affected: the upload URL, the three sync-state fields, and both failure stages on the send path, plus
failures.checkon the receive path. Each now resolves the payload first viajson_response_payload, mirroring the unwrap/checkalready did.jsmn_find_keyis unchanged frome0ebb77— the hardening is intact, and envelope handling is explicit per call site rather than hidden in a shared helper.RLS denials: skip, advance, report (
f68c523,067f3fb,4a1cfd2)A denial suppressed the receive checkpoint, which gave neither progress nor a signal: the rows are permanently not this site's to hold, so the next check re-delivered them, they were denied again, and one denied row stalled every later change behind it. In a chunked batch it was worse —
policy_deniedwas a local of one apply call, so a denial in a non-final chunk suppressed a checkpoint that was already a no-op and the final chunk advanced past the denied rows anyway.Denied entries are now counted, skipped, and the cursor advances. The count accumulates across the drain and is reported as
receive.denied, so discarding stays visible.receive.rowsreports rows actually written, matching what API.md has always documented.Deliberately not an error even when every row is denied: a single-row payload belonging to another user is denied in full and is a correct outcome, which tests 27 and 29 already assert.
Known gap: this does not extend to a value large enough to be fragmented. Making the v3 path skip a denial leaves PostgreSQL's transaction unusable — the next statement fails with
buffer pin is not owned by resource owner— and neither a savepoint around the per-value apply nor dropping the staged-fragment delete recovers it.067f3fbtried;4a1cfd2reverts it.58_v3_denied_checkpoint.sqlbuilds a real fragmented payload, applies it under aWITH CHECKpolicy, and pins the current behaviour with a note to flip the assertion once the apply leaves a recoverable state.Block-column failures (
6bf127b)Reading the row back is part of writing a block column, and a row its own session cannot select could not sync anyway — so the failure stays fatal, but it has to be legible. Five paths returned a bare code with no message, two of them the raw negative from
pk_decode_prikey, which is not aDBRESvalue at all.databasevm_stepclears the error text on entry, so PostgreSQL aborted the user'sINSERTwith an emptyerrmsg. Each now names the table and column.Artifact transfer deadlines (
7077df7)CURLOPT_TIMEOUTapplied to both pooled handles, but a presigned S3 URL is not an API endpoint, so artifact transfers inherited the 300-second cap: 256 MiB inside 300s demands a sustained ~875 KB/s. API calls keep the elapsed-time cap; artifact transfers now abort after 60 seconds below 1 KB/s — which also catches a real stall five times sooner — with a 1-hour backstop, since nothing can cancel a transfer in flight.Also
merge_flush_pendingreplaced a failing commit's real error with a generic string (b4a5b3f).ON_ERROR_STOPonce errors became sticky (9d89fbb).changelog.yml's tag filter is restored: widening it made the workflow fire, but the called workflow derives the version with${GITHUB_REF#refs/tags/v}and rejects our unprefixed tags. Fixing that needs a change inchangelog-action, so the filter goes back and a note records why.deniedin both receive shapes and all six samples.Verification
5171b77: 37 pass, 1 skipped (release, correct on a branch)Memory Leaks Check: OKmake postgres-check-migration:1.1.3 -> 1.1.4, SQL surface identical, no migration script requiredOpen
endian-unittestcannot detect anything:pk.cno longer references any endian-conditional symbol, so the forced-big-endian object is byte-identical to the normal one (verified by hash). The target is also unreachable fromtest:and from CI. Delete it or make it real — @marcobambini's call, since it is his audit scaffolding.{"errors":[...]}) is never parsed by the client — error bodies are passed through verbatim — so root-scoped lookups cannot affect it. That also means a server error surfaces as a raw JSON blob rather than itsdetailstring. Pre-existing, worth a follow-up.🤖 Generated with Claude Code
https://claude.ai/code/session_01CCE6B54Qtf3UsaCVAJtVQF