Skip to content

Bind message signatures to the overlay and bound DHT storage - #724

Open
RyanKung wants to merge 21 commits into
masterfrom
fix/message-domain-separation-and-storage-bounds
Open

Bind message signatures to the overlay and bound DHT storage#724
RyanKung wants to merge 21 commits into
masterfrom
fix/message-domain-separation-and-storage-bounds

Conversation

@RyanKung

@RyanKung RyanKung commented Sep 3, 2026

Copy link
Copy Markdown
Member

Closes #704, closes #701, closes #725, closes #726.

Signature domain separation (#704)

Every MessageVerification now signs a domain-separated transcript

len(tag) || tag || network_id || ts_ms || ttl_ms || data

where the domain (tag, network_id) names the message family and the overlay. Every signature is
issued by a MessageSigner (a session key acting inside one overlay) and verified against the
receiver's domain, never a value carried in the message, so:

  • a signature issued inside overlay A does not verify inside overlay B even when the session
    key is shared;
  • the Transaction and MessagePayload signatures over the same transaction hash are no longer
    interchangeable, and node descriptors / onion backward payloads sign under their own tags;
  • node descriptors are signed by the node's MessageSigner and verified under the receiver's
    network_id; signing a body that states another overlay is refused, so a verified descriptor's
    stated overlay is the overlay it was signed for (the former matches_network post-filters are
    gone).

API shape:

  • DomainTag (built with domain_tag!, which checks the length-prefix law at compile time),
    SigningDomain, and MessageSigner<S: Borrow<SessionSk>> are exported from
    rings_core::message. MessageSigner<&SessionSk> is the Copy form signing functions take;
    MessageSigner<SessionSk> is what long-lived runtimes (chunked transfers, onion exits) store, so
    the key/overlay pairing is never split across two fields.
  • Transaction::new*, MessagePayload::new*, PayloadSender, and the descriptor constructors take
    a MessageSigner; MessageVerificationExt::verify and the descriptor verify_signature /
    is_live_at / latest_valid_by_* take the receiver's network_id.

This is a wire-incompatible change (signatures from earlier builds no longer verify), hence the
bump to 0.21.0.

Storage bounds (#701)

  1. Native storage enforces its byte budget. SledStorage keeps an in-memory index of file
    sizes in write order (rebuilt from the directory on open, stale .tmp files removed). A put
    beyond the budget retires the least recently written other keys until the value fits; a value
    larger than the whole budget is rejected with Error::StorageValueExceedsCapacity; the index is
    updated only after the file-system operation succeeded, so a failed write leaves index and
    directory consistent; the budget is restored on open. A record the current schema cannot decode
    (written by an earlier build) is retired on the read that discovers it. MemStorage::bounded
    gives the local fetched-entry cache least-recently-written eviction at LOCAL_CACHE_CAPACITY
    entries. Both backends share one WriteOrderedMap.

  2. Entries carry a retention bound. Entry::expires_at_ms is stamped at the operation boundary
    (EntryOperation::stamped(now, actor)) with DEFAULT_TTL_MS and joins by max, so the product
    of the payload lattice and the bound lattice stays a join-semilattice. Every storage read goes
    through PeerRing::live_storage_entry / live_storage_entries, which retire an expired value
    and report it absent, so hand-off, republish, lookup, and ack-delete never see an expired value.
    A stored value without a bound is treated as not live. The retention model lives in
    dht/entry/retention.rs.

  3. Admission bounds peer-supplied deltas. Entry::validate_admissible_at(now) requires a live
    bound at most now + MAX_TTL_MS + TS_OFFSET_TOLERANCE_MS, every version (dots, tombstones,
    register) at most now + TS_OFFSET_TOLERANCE_MS, and every payload element at most
    ENTRY_PAYLOAD_MAX_BYTES (32 KiB). It is enforced on the peer-supplied delta at the two
    write funnels (PeerRing::operate_storage_entry for operations, join_storage_entry for
    replicated values), pre-checked in the sync batch so a failing entry rejects the batch before
    any earlier entry is written (one admission clock per batch), and applied in local_cache_put.
    A u128::MAX register can no longer pin a key; a locally derived compaction floor is never
    mistaken for a peer clock running ahead.

    The per-payload size bound is element-intrinsic, so filtering by it is a lattice morphism and,
    with the existing ENTRY_DATA_MAX_LEN count cap, bounds a carrier at their product, which a
    compile-time law keeps inside one transport message. A byte
    budget over the whole carrier was tried and rejected: "the newest payloads that fit" depends on
    the sizes of payloads a replica may already have dropped, so it is not a lattice morphism and
    replicas diverge (three-element counterexample in the review).

  4. One storage transition per ring. Every read-modify-write of a slot (operate_storage_entry,
    join_storage_entry, the ack-gated remove_storage_entry_confirmed_by, and the retirement a
    read performs) runs under PeerRing::storage_transition. The inbound actor and the stabilizer
    write the same slots concurrently (a hold arriving while the recipient drains its inbox), and
    the store orders single puts only, so two interleaved read-modify-writes used to overwrite each
    other; a test with a yielding store double shows the lost hold without the transition.

Relay inbox (#725)

EntryKind::RelayMessage gains its producer, its witness, its authority, and its policy:

  • Producer. A CustomMessage reaching the node responsible for its destination's ring
    position (destination ∈ (predecessor, self]; a node with successors but no known predecessor
    is uninformed, not responsible) while the destination has no connection attempt at all is
    held: wrapped in a HeldMessage under the holder's signature (domain
    rings-core:relay-inbox:held-message:v1, timestamp = hold instant) and written into the inbox
    carrier destination + 1 by an ordinary Extend. Relay carriers are placed by the ring
    geometry in every storage mode, so the placement law holds with virtual nodes enabled (the
    production default); a relay carrier has exactly one placement, its DID, under any configured
    redundancy (kind.replication bounds the affine set a placed operation or synced entry may
    name), and its own storage namespace (StorageKey = (kind, placement); relay slots render as
    relay:<placement>, data slots keep the bare placement), so a data topic any node parks at
    d + 1 through storage_store cannot shadow the inbox kept for d. A lookup reads the data
    slot only.
  • Witness (dht/entry/inbox.rs). An element is admissible iff it decodes as a HeldMessage
    whose payload is a CustomMessage whose transaction and relay are both addressed to
    inbox_destination(entry.did) (so the recipient has nothing to forward), held no later than the
    receiver's clock, whose holder signature verifies inside the receiver's overlay, and whose
    payload verifies as of the hold instant (MessageVerificationExt::verify_at, which judges
    proof liveness and session validity at a given instant). Admissibility is therefore monotone
    in time. A reset floor is rejected, and a delta larger than the inbox is refused before the
    first signature is verified.
  • Write law (checked by the shell, which knows the writer). A hold (Extend) only from the
    node the owner itself routes the destination to (PeerRing::inbox_hold_authority, the local
    answer of find_successor), every element held by that node, and only while the held message's
    sender proof is still live by the owner's own clock: the holder signs the hold instant, so this
    freshness bound is what makes it honest and bounds replay by the sender's proof lifetime.
    Authority is judged before any signature is verified. A removal (Tombstone, per add dot) only
    from the recipient; a relocation only as an OwnershipHandoff whose authenticated sender
    (the transaction signer, never the relay path) is the receiver's predecessor, otherwise the
    carrier is skipped without an ack; no other operation. A relay carrier is never fetched,
    cached, replicated, or returned to a lookup.
  • Policy. The newest RELAY_INBOX_MAX_LEN (64) messages per inbox and as many tombstones;
    retention 24 h after the last hold, at most 7 d; data topics keep 10 min / 100 min.
  • Delivery. The storage maintenance phase emits one intent toward the application layer,
    dht::InboxDelivery::deliver_inbox, and Swarm::stabilizer injects the swarm's interpreter
    (swarm::inbox::SwarmInboxDelivery), so the DHT never reaches into the delivery pipeline. The
    interpreter reads the local inbox, retires the elements that fail the witness, and then,
    element by element, delivers through the logical stage of the inbound pipeline
    (swarm::callback::LocalDelivery over LogicalInbound, the connection-independent stage the
    per-connection InboundProcessor also runs: application validation, dispatch, on_inbound,
    each under the inbound deadline) and tombstones the delivered element by its dot, so a pass cut
    short by its step deadline resumes where it stopped. A removal never extends a carrier's
    retention, so a drained inbox expires when its last hold would have. The callback (SwarmCallbackSlot) is resolved at delivery
    time, so Swarm::stabilizer is infallible and set_callback after listen is honoured.
    Delivery is at least once.
  • Tests: witness and authority laws (dht/entry/test_inbox.rs), dot-based retirement against a
    stale copy, caps, and two end-to-end cases (tests/default/test_inbox.rs): the peer returns
    through its successor or straight through its predecessor; in both the owner's pass hands the
    carrier over, the recipient drains it, and the owner's copy is removed by the ack.

Ownership hand-off follows the successor head (#726)

The hand-off used to run only inside HandleMsg<NotifyPredecessorReport>, one of three inputs
that can move the successor head (the others: correct_stabilize's topology query and a directly
connected peer admitted as head). Placement is a function of the ring state, so:

  • Head law (pure). topology::step emits TopologyAction::SuccessorHeadChanged(h) iff the
    head moved to h, once per transition, after the event's own actions; topology::successor_head
    is the single definition of the head, shared with find_successor.
  • Request, not send. The action lowers to CoreEffect::RequestStorageRepair, which only
    records the repair intent (request_storage_repair). A send at admission time was tried and
    rejected: it can precede the peer's own admission of this node and is dropped as coming from an
    unadmitted connection.
  • Storage maintenance phase. Two steps: deliver_inbox, then repair_storage, which
    restores placement in one bounded pass (offer the live entries beyond (self, head] to the
    head as an OwnershipHandoff with ack-gated cleanup, republish to missing affine owners
    additively) through the existing repair window (tracked sends,
    STORAGE_REPAIR_FRESH_CONNECTION_GRACE_MS, one delivery per step). The maintenance loop runs the
    phase when requested and periodically, so the inbox drain, previously reachable only through the
    one-shot stabilize(), now runs in production as well. A head change is the local
    PeerRingAction::StorageRepairDue.
  • The report handler only connects; the SendStorageSync effect, whose sole producer it was, is
    gone.

Tests: head law in topology/tests.rs; test_stabilization/test_storage_handoff.rs (direct
connection moves the head, admission sets the request, the pass hands off after the grace, the
ack removes the local copy); a second inbox end-to-end test where the returning peer connects
straight to its predecessor.

Tests

  • verify.rs: transcript layout, overlay binding, tag binding, ext-verify uses the type's tag,
    owned/borrowed authorities sign identically, over-long tag label rejected.
  • test_payload.rs: transaction/payload signatures not interchangeable; foreign overlay rejected.
  • test_entry.rs: stamping law, max join of bounds across every operation, liveness, admission
    bounds for lifetime, versions, and payload size, normalization/affine preserve the bound, legacy
    value without bound is not live.
  • dht/storage/tests: live reads retire expired values, write funnel rejects pinned register /
    expired / over-long bound, join keeps the later bound, cache shares admission and retention.
  • memory.rs / test_sled.rs / write_ordered.rs: eviction order, rewrite does not evict,
    oversize rejected without change, reopen restores the budget in modification order.

🤖 Generated with Claude Code

r2d2-rs and others added 3 commits September 3, 2026 19:09
Closes #704, closes #701.

Every MessageVerification now signs a domain-separated transcript
`len(tag) || tag || network_id || ts || ttl || data`, verified against the
receiver's own overlay and message family. Transaction and payload
signatures over the same hash are no longer interchangeable, node
descriptors and onion backward payloads sign under their own tags, and a
signature issued inside one overlay does not verify inside another.
Message constructors and PayloadSender take a MessageSigner (a session
key acting inside one overlay); MessageVerificationExt::verify takes the
receiver's network_id.

DHT entries carry a retention bound stamped at the operation boundary
with DEFAULT_TTL_MS and joined by max; every storage read retires an
expired value before serving, replicating, or acknowledging it. The
single write funnel rejects entries whose bound is expired or beyond
MAX_TTL_MS, and whose CRDT versions run ahead of the receiver's clock by
more than the message skew tolerance, so a forged register floor cannot
pin a key. The native file store enforces its byte budget by retiring
the least recently written keys and rejecting oversize values, and the
fetched-entry cache is bounded; both share one write-ordered map.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signatures from earlier builds no longer verify, so the wire change gets a
minor bump; the changelog records the breaking changes and the storage
bounds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Retention policy is now a property of the entry kind. Data entries keep
DEFAULT_TTL_MS / MAX_TTL_MS (10 min / 100 min); relayed messages held for
an offline destination are stamped with DEFAULT_RELAY_ENTRY_TTL_MS (24 h)
and admitted up to MAX_RELAY_ENTRY_TTL_MS (7 d), so a peer that returns
within a day of the last relayed message still receives its inbox.

Materialization also bounds each carrier in encoded bytes next to the
existing payload count (ENTRY_DATA_MAX_BYTES = 1 MiB for data,
RELAY_INBOX_MAX_BYTES = 16 MiB for a relay inbox), keeping the newest
payloads that fit, so an inbox that keeps receiving messages retains only
its most recent ones and a payload that alone exceeds the budget is
dropped rather than retained over it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Pushed a1856c8: retention is now a property of the entry kind, and each carrier has a byte budget.

  • EntryKind::default_lifetime_ms / max_lifetime_ms: data entries keep 10 min / 100 min; relayed messages held for an offline destination (the inbox) are stamped with 24 h and admitted up to 7 d (DEFAULT_RELAY_ENTRY_TTL_MS / MAX_RELAY_ENTRY_TTL_MS). The inbox bound joins by max, so it is "24 h after the last relayed message".
  • EntryKind::max_data_bytes: materialization keeps the newest payloads that fit 1 MiB (data) / 16 MiB (relay inbox) next to the existing 1024-payload count cap, so a busy inbox retains only its newest messages; a payload that alone exceeds the budget is dropped rather than retained over it.
  • Tests: test_relay_entries_use_their_own_retention_policy, test_byte_budget_keeps_newest_payloads. CHANGELOG, SECURITY.md, and the PR description are updated.

The 1 MiB / 16 MiB figures are proposals; both are one constant each.

…e receiver overlay

Review round for #724 (three independent reviews).

The whole-carrier byte budget is withdrawn: "the newest payloads that
fit" depends on payloads a replica may already have dropped, so it is
not a lattice morphism and replicas diverge. Admission instead bounds
every payload element at ENTRY_PAYLOAD_MAX_BYTES, an element-intrinsic
predicate that commutes with union; with ENTRY_DATA_MAX_LEN it bounds a
carrier at their product. The relay-inbox policy is withdrawn as well:
EntryKind::RelayMessage has no production producer and the kind is
peer-declared, so a wider policy only widened an attacker's budget
(#725 tracks the real inbox). Retention now has one policy and lives in
dht/entry/retention.rs with its laws.

Admission is applied to the peer-supplied delta, never to the join
result: PeerRing::operate_storage_entry is the single funnel for local
and remote operations, join_storage_entry admits replicated values, and
each boundary reads the clock once (the sync batch shares one admission
time between validation and persistence). The wall-clock wrappers on
Entry and EntryOperation are gone; the operation-boundary time is
explicit.

Descriptors are signed by the node's MessageSigner and verified under
the receiver's network_id; signing a body that states another overlay is
refused, so the matches_network post-filters are redundant and removed.
MessageSigner is generic over how it holds the key (borrowed, Copy form
for signing functions; owned form for chunked transfers and onion exit
runtimes) and is the only signing authority. Domain tags are built with
domain_tag!, which checks the length-prefix law at compile time.

SledStorage writes the temporary file before touching the index,
restores the previous record when the rename fails, clears per file,
removes stale .tmp files on open, and retires a record the current
schema cannot decode (postcard is not self-describing, so a legacy value
cannot be lifted by a serde default). Lock poisoning is reported as
Error::StorageLockPoisoned.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Pushed db7d770, addressing the three independent review rounds (consensus items first).

Design

  • Byte budget over a carrier is gone: "newest payloads that fit" is not a lattice morphism (three-element counterexample in the reviews). Replaced by a per-element bound ENTRY_PAYLOAD_MAX_BYTES (64 KiB) checked at admission; filtering by an element-intrinsic predicate commutes with union, and with ENTRY_DATA_MAX_LEN it bounds a carrier at their product. materialize_elements is back to the rank-based count cap.
  • Relay policy withdrawn: EntryKind::RelayMessage has no production producer and kind is peer-declared, so 24 h / 7 d / 16 MiB only widened the attacker's budget. One policy for both kinds; the real inbox (producer + receiver-verifiable witness + its own bounds) is DHT: wire the offline relay inbox (RelayMessage) with a receiver-verifiable witness and its own retention policy #725.
  • Descriptors are signed by the node's MessageSigner and verified under the receiver's overlay (verify_signature(network_id), is_live_at(now, network_id), latest_valid_by_*(…, network_id, …)); signing a body that states another overlay is refused. The matches_network post-filters became redundant and are removed. SignedDescriptor now names its Body so the tag has one source.
  • Legacy on-disk values: postcard is not self-describing, so #[serde(default)] cannot lift an old record. SledStorage now retires a record the current schema cannot decode on the read that discovers it (documented decode law); test_legacy_value_without_bound_is_not_live keeps the in-memory/JSON case.
  • Admission is applied to the peer-supplied delta, not the join result: PeerRing::operate_storage_entry(now, placement, op) (one funnel for local and remote operations, replacing the duplicated read/operate/join in chord/storage.rs and the handler) validates op, and join_storage_entry(now, key, incoming) validates replicated values. One clock per boundary: the sync batch samples now_ms once for validate and persist; repair/republish thread the same now_ms.
  • SledStorage::put writes the temp file first, then adjusts the index, renames, and records; a failed rename restores the previous record; clear forgets per file; stale .tmp files are removed on open.

Abstraction

  • Retention/admission moved to dht/entry/retention.rs with the laws in the module doc; entry.rs is back under 1000 lines.
  • The six wall-clock wrappers are gone: stamped/operate/overwrite/extend/touch/compact_data take now_ms; only the storage boundary reads the clock. stamped is one try_map_entry over an EntryWitness instead of five copies.
  • MessageSigner<S: Borrow<SessionSk>>: MessageSigner<&SessionSk> is the Copy form functions take, MessageSigner<SessionSk> is what FrameSource::Chunked and the onion exit runtimes store (no more (SessionSk, u32) pairs re-assembled per chunk). MessageVerification::new is no longer a second signing authority; MessageSigner::sign is the only one.
  • DomainTag::new is total (Option); tags are built with domain_tag!, which checks the length-prefix law at compile time. LOCAL_CACHE_CAPACITY no longer needs unreachable!.
  • Error::StorageLockPoisoned replaces the borrowed DHTSyncLockError; dashmap is a dev-dependency; TEST_NETWORK_ID lives in crate::tests for both crates; duplicated #[cfg], the test-only inherent method, and the fixture doc are fixed.

Not changed, on purpose

Closes #725.

A CustomMessage that reaches the node responsible for its destination's
ring position while the destination is not connected is held in the
carrier `destination + 1` (EntryKind::RelayMessage) by an ordinary
Extend write, routed to the carrier's storage owner like any other.

Storage admits a relay element only under a witness the owner verifies
itself: it decodes as a signed MessagePayload addressed to the inbox's
peer, carries an application message, and both signatures verify inside
the local overlay regardless of proof liveness
(MessageVerificationExt::verify_signature) because retention, not the
proof, governs how long the inbox holds a message. The kind is therefore
no longer self-declared, which makes the per-kind policy safe: inboxes
are retained 24 h after the last held message, up to 7 d.

Once the peer is online its inbox key lies in its own storage interval
and ownership hand-off moves the carrier to it. Every stabilization
round the peer drains its local inbox to the application (on_validate,
then on_inbound) and compacts the delivered messages out of the carrier;
compaction now applies to relay carriers, so delivered elements and
their tombstones fall below the floor at every replica. The stabilizer
carries the swarm callback for this, so Swarm::stabilizer returns a
Result. Const-generic redundancy stays at the public storage API only;
the internal operate/fetch helpers take the configured redundancy.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Pushed 1a614af: the relay inbox (#725) is now implemented in this PR instead of being deferred, so the PR also closes #725.

  • Producer. A CustomMessage that reaches the node responsible for its destination's ring position (destination ∈ (predecessor, self]) while the destination is not connected is held in the carrier destination + 1 (EntryKind::RelayMessage) by an ordinary Extend write, routed to the carrier's storage owner like any other write. Decided in the CustomMessage handler (custom_message_effects(local, ctx, destination_offline)), executed by the new CoreEffect::HoldForOfflineDestination.
  • Witness (dht/entry/inbox.rs). Admission of a relay element requires that it decodes as a signed MessagePayload addressed to inbox_destination(entry.did), carries an application message, and that both signatures verify inside the receiver's overlay (MessageVerificationExt::verify_signature, liveness excluded: retention governs the message's life in the inbox). Entry::validate_admissible_at(now, network_id) therefore takes the overlay; PeerRing::network_id() supplies it at both funnels.
  • Policy. With the witness in place the per-kind bounds are safe again: EntryKind::default_lifetime_ms / max_lifetime_ms give the inbox 24 h / 7 d (DEFAULT_RELAY_INBOX_TTL_MS / MAX_RELAY_INBOX_TTL_MS); data keeps 10 min / 100 min.
  • Drain. Once the peer is online, destination + 1 lies in its own storage interval, so ownership hand-off moves the carrier to it. Each stabilization round (drain_inbox step) the peer reads its local inbox, delivers every witnessed element through on_validate then on_inbound, and issues CompactData for the delivered messages. Compaction now applies to relay carriers, so delivered elements and their tombstones fall below the floor at every replica — which is also the bound for DHT storage: capacity not enforced (Sled), no entry TTL, unbounded attacker-controlled CRDT version #701's relay tombstone growth. Stabilizer::new(transport, callback), hence Swarm::stabilizer() -> Result.
  • Internals: const-generic redundancy stays at the public ChordStorageInterface<R> boundary only; entry_operate_with_redundancy, entry_lookup_for_fetch(key, redundancy), operate_entry, and fetch_entry take the transport's configured redundancy.
  • Tests: witness laws and per-kind retention (dht/entry/test_inbox.rs), compaction against stale copies, and an end-to-end dummy-network test (tests/default/test_inbox.rs): a message to an offline peer is held by the owner, handed over when the peer rejoins through its successor, delivered to the peer's callback, and compacted.

The ownership hand-off (sync_entries_with_successor) ran only inside the
NotifyPredecessorReport handler, one of three inputs that can move the
successor head; a head moved by correct_stabilize's topology query or by
a directly connected peer left entries at a node that no longer owned
them until they expired. Placement is a function of the ring state, so
the stabilizer now offers the live local entries beyond (self, head] to
the current head every round; deliveries are joins and cleanup is
ack-gated, so repetition is idempotent. A transition-triggered one-shot
send was rejected: it can precede the peer's admission of this node and
is dropped, and repair does nothing at redundancy 1.

The report handler only connects now, and the SendStorageSync effect,
whose sole producer it was, is removed. topology::successor_head names
the head for find_successor and the stabilizer alike.

Closes #726.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 5 — 73487b6: ownership hand-off as a stabilization invariant (#726).

Finding. sync_entries_with_successor ran only inside HandleMsg<NotifyPredecessorReport>. That is one of three inputs that move the successor head; correct_stabilize's topology query (TopologyEvent::Stabilize) and a directly connected peer admitted as head (TopologyEvent::Admit) never handed off, so entries stayed at a node whose interval no longer contained them until retention retired them. Under Chord the interval, not the reporting message, defines responsibility; CorrectChord proves the ring converges, and reachability is a function of that ring. The inbox from round 4 made it visible: a peer returning through its predecessor never received its held messages.

Change. The stabilizer runs hand_off_storage every round: it offers the live local entries beyond (self, head] to the current head (topology::successor_head, now shared with find_successor). Deliveries are joins and cleanup is ack-gated, so repetition is idempotent and steady state is a no-op scan. The report handler only connects; the SendStorageSync effect, whose sole producer it was, is removed together with its orphaned test.

Why not a transition effect. I first emitted a SuccessorHeadChanged action from the pure step and lowered it to a one-shot hand-off at admission. The inbox end-to-end test failed deterministically: the delivery was sent the instant the owner admitted the returning peer, arrived before the peer had admitted the owner, and was dropped as coming from an unadmitted connection (pending_connection_allows_message); storage repair does not resend at redundancy 1. The periodic form is the Chord-shaped one and needs no extra vocabulary in the topology core.

Tests. tests/default/test_stabilization/test_storage_handoff.rs (direct connection moves the head, the owner's round hands off, the ack removes the local copy); test_inbox.rs gains the predecessor-return case and both cases now assert the owner's copy is gone after the ack. The old report-handler hand-off test is replaced by these. Gates: core dummy 714, node 323, clippy native / wasm core / browser / ffi, rustdoc, taplo, typos, nightly fmt.

The pure topology step now states the head law: a transition whose
successor head moves emits SuccessorHeadChanged(head), once, after its
own actions. The shell lowers it to CoreEffect::RequestStorageRepair,
which only records the repair intent; sending at admission time was
rejected because the delivery can precede the peer's own admission of
this node and is dropped.

The storage repair pass restores the whole placement invariant: it
drains the local inbox, offers the live entries beyond (self, head] to
the head as an ack-gated ownership hand-off, and republishes to missing
affine owners, all through the existing repair window with its
fresh-connection grace. The maintenance loop runs the pass when
requested and periodically, so the inbox drain, previously reachable
only through the one-shot stabilize(), now runs in production as well.

Closes #726.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 6 — 01b82e1: the head change requests a repair round instead of sending (#726).

Head law (pure). topology::step emits TopologyAction::SuccessorHeadChanged(h) iff the successor head moved to h, once per transition, after the event's own actions (topology/tests.rs::assert_head_law over admit / stabilize / remove / join, and the no-change cases). PeerRing maps it to RemoteAction::HandOffStorage, lowered to CoreEffect::RequestStorageRepair, whose interpretation is transport.request_storage_repair() and nothing else. Admission therefore records an intent; the delivery happens under the repair schedule, whose fresh-connection grace (STORAGE_REPAIR_FRESH_CONNECTION_GRACE_MS, 30 s) outlives the peer's own admission of this node, which is exactly the race a send at admission time lost in round 5.

Placement repair pass. repair_storage now restores the whole placement invariant in one bounded pass: drain the local inbox, offer the live entries beyond (self, head] to the head (OwnershipHandoff, ack-gated cleanup), republish to missing affine owners (additive). All three go through the existing repair window (tracked sends, one delivery per step, deferral reasons), so the hand-off inherits the sync-storm bounds from #691. The report handler only connects.

A production gap this closed. Stabilizer::wait_with, the loop the node runs, executes only the topology phase and the repair phase. The one-shot stabilize() is where round 4 put drain_inbox and round 5 put the hand-off step, so neither ran in production. Both now live in the repair pass, which the loop runs when requested and periodically.

Tests. test_storage_handoff.rs: after a direct connection moves the head, storage_repair_requested() is set, and run_requested_storage_repair() completes the hand-off once the connection is aged past the grace (force_peer_connected_at), with the ack removing the owner's copy. Both inbox end-to-end cases (return through the successor, return through the predecessor) go through the same pass. Dummy-only tests are gated so cargo clippy --all --tests without the feature stays green. Gates: core dummy 719, node 323, clippy native / wasm core / browser / ffi, rustdoc, taplo, typos, nightly fmt.

Trade-off to be aware of. Hand-off latency is now bounded by the repair schedule: the quiet gap after stabilization, the 30 s fresh-connection grace, and one delivery batch per repair step. For the inbox that means a returning peer receives its held messages roughly half a minute after connecting. Lowering the grace for OwnershipHandoff deliveries specifically would be a one-line policy change if you want it faster.

…ing-placed

Review of the relay inbox (#725) found it unsound as shipped: it never
drained under the production storage mode, its compaction floor dropped
undelivered messages and could be pushed by anyone, its witness bound
only the content shape so any peer could fill, replay, read, or wipe a
victim's inbox, and admissibility depended on the sender's session
staying live.

An inbox element is now a HeldMessage: the payload wrapped and signed by
the node that held it, under its own domain tag, with the hold instant as
the signature timestamp. The witness verifies the holder inside the
receiver's overlay and the payload as of the hold instant
(MessageVerificationExt::verify_at; the liveness-free verify_signature is
gone), requires a CustomMessage addressed to the recipient, and rejects a
reset floor. Authority is checked at the write: a hold only from the node
the owner routes the destination to (PeerRing::inbox_hold_authority), a
removal only from the recipient, a relocation only as an ownership
hand-off from the predecessor; a relay carrier is never fetched, cached,
replicated, or returned to a lookup by anyone but its recipient. Removal
is per element by add dot; the inbox keeps the newest RELAY_INBOX_MAX_LEN
elements and as many tombstones. Relay carriers are placed by the ring
geometry in every storage mode, so the placement law holds with virtual
nodes enabled.

Delivery goes through the inbound pipeline (validation, dispatch,
on_inbound under the inbound deadline) via swarm::callback::
deliver_local_payload, and the stabilizer resolves the swarm callback at
delivery time, so Swarm::stabilizer is infallible again and set_callback
after listen is honoured. The storage maintenance phase is two steps,
deliver_inbox then repair_storage; a head change is a local
PeerRingAction::StorageRepairDue.

Also: a node with successors but no predecessor no longer claims the
whole ring and its interval is (predecessor, self] exactly; a
destination counts as offline only with no connection attempt at all;
ENTRY_PAYLOAD_MAX_BYTES is 32 KiB with a compile-time law that a full
carrier fits one transport message; descriptors verify only when the body
states the receiver's overlay and onion backward payloads are verified
under the receiver's overlay; MessageSigner::to_owned is owned; storage
count overflow has its own error.

Closes #726.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 7 — 3083402: the relay inbox is redesigned after a three-reviewer read of the full diff (findings summarized below), plus the non-inbox findings from the same review.

Inbox findings that were structural (all fixed here):

  1. Never drained under the production storage mode (dht_virtual_nodes defaults to 160; the d + 1 placement law held only with virtual nodes off). Relay carriers are now placed by the ring geometry in every mode (find_storage_owner_for(key, kind), storage_sync_target, hand-off, should_persist_synced_entry), and are never replicated.
  2. Compaction floor was reset semantics: it dropped undelivered elements and an empty CompactData from anyone could push it. Relay carriers reject a reset floor; removal is per add dot (Tombstone), issued by the recipient alone; tombstones are capped at the inbox length.
  3. Replay oracle / writer unbound / non-monotone admissibility. An element is now a HeldMessage: the payload wrapped and signed by the holder (own domain tag, timestamp = hold instant). The witness verifies the holder inside the receiver's overlay and the payload as of the hold instant (verify_at), so a sender's session expiring later does not unverify a held message and a holder can only hold inside the sender's proof lifetime. Authority at the write: a hold only from the node the owner routes the destination to (inbox_hold_authority = the local find_successor answer), a relocation only as an OwnershipHandoff from the predecessor, a removal only from the recipient; never fetched, cached, or served to anyone but the recipient. Only Message::CustomMessage (not the Application class, which includes Chunk).
  4. is_responsible_for: (predecessor, self] exactly; no predecessor ⇒ responsible only when alone. "Offline" = no connection attempt at all, admitted or pending.
  5. Delivery goes through the inbound pipeline (deliver_local_payload: on_validate, dispatch, on_inbound, each under INBOUND_CALLBACK_TIMEOUT); the stabilizer resolves the callback slot at delivery time, so Swarm::stabilizer is infallible again and set_callback after listen is honoured. The storage maintenance phase is deliver_inbox then repair_storage; a delivery failure no longer aborts placement.
  6. Carrier bound exceeded the transport ceiling (64 MiB > 60 MB): ENTRY_PAYLOAD_MAX_BYTES is 32 KiB with a const assertion that a full carrier fits one transport message.

Other findings fixed: the head change is a local PeerRingAction::StorageRepairDue (no RemoteAction carrying an ignored DID); descriptors verify only when the body states the receiver's overlay, and onion backward payloads are verified under the receiver's overlay, not expected_exit.network_id; MessageSigner::to_ownedowned; MessageSizeOverflow is no longer used for storage counts (StorageCountOverflow); fetch_entry inlined; operate_storage_entry free function renamed; MemStorage predicate named by role; the inbox module is pub(crate); doc drift (stamped, SuccessorHeadChanged, stateright comments); the vacuous PeerRingAction::Some(_) assertion asserts the witness; the two storage poll loops share wait_for_storage_state.

Left as is, deliberately: StorageLockPoisoned beside DHTSyncLockError (different subsystems; a rename of the latter is a separate cleanup); MessageSigner::session_sk() (the onion runtimes still need the key for decryption); SledStorage's name; the retention fixtures across test modules; fully qualified crate::tests::live_entry call sites.

Trade-off stated in SECURITY.md: a malicious holder is one identity in one ring position; it can hold junk for the peers it is responsible for (newest 64 per inbox) or redeliver inside the sender's proof lifetime, and every element names it by signature. Held messages are relocated in the clear between owners like every DHT value.

Gates: core dummy 726, node 323, clippy native / wasm core / browser / ffi, rustdoc, taplo, typos, nightly fmt.

…ailing the batch

Self-review of the inbox redesign: a relay carrier offered by anyone but
the receiver's predecessor, or under any purpose but an ownership
hand-off, was rejected with an error, which failed the whole sync batch
including the data topics sharing it, the same one-element-rejects-all
non-monotonicity the review found in admission. The carrier is not
invalid, only not this receiver's to take yet, so it is now skipped
without an acknowledgement (its owner keeps it and offers it again) and
the rest of the batch is accepted; test added.

Also: the hold authority is derived only for a hold, not for a removal;
the stacked doc comment on PeerRing::operate_storage_entry is merged;
a redundant error alias and an inline path in a test are removed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 8 — 61c6cb4: self-review of round 7 (full read of 01b82e15..30834021).

Found and fixed:

  1. Relocation law failed the whole batch. A relay carrier offered by anyone but the receiver's predecessor, or under any purpose but OwnershipHandoff, returned Err from StorageSyncBatch::validate_one, rejecting the data topics sharing its batch, the same one-element-rejects-all non-monotonicity round 7 removed from admission. The carrier is not invalid, only not this receiver's to take yet, so it is now skipped without an acknowledgement (relay_relocation_permits, alongside should_persist_synced_entry); its owner keeps it and offers it again on a later pass. Test: test_persist_synced_entries_skips_a_relay_carrier_from_a_stranger.
  2. PeerRing::inbox_hold_authority was derived for every relay operation; only a hold needs it.
  3. PeerRing::operate_storage_entry carried two stacked doc comments; merged.
  4. A redundant Error as CoreError alias and an inline super::EntryVersion path in a test.

Checked and sound: the holder signs the canonical postcard encoding of the payload (the same bytes Encoder for HeldMessage embeds); the relayed OperateEntry keeps the origin's transaction signer as writer; a drained payload dispatched through the inbound pipeline is not re-held (should_forward_from(local) is false at its destination); the owner's find_successor(d) names succ(d) exactly when the owner is pred(d), and the node itself when alone; an empty Extend from the holder only joins retention.

Gates: core dummy 727, node 323, clippy native / wasm core / browser / ffi, rustdoc, taplo, typos, nightly fmt.

…ner key, shared fixtures

The items left open after the round-7 review:

- DHTSyncLockError, CallbackSyncLockError, and StorageLockPoisoned named
  one failure; they are Error::LockPoisoned.
- storage::sled::SledStorage has been a file-per-key store under a byte
  budget since the budget landed; it is storage::file::FileStorage.
- MessageSigner no longer hands out its session key; the onion runtimes
  only needed session_public_key.
- StorageSyncBatch::new takes the admission clock instead of reading it.
- The retention fixtures (bounded, live_at, expired) are one family in
  crate::tests (live, live_entry, with_retention, expired), and every
  fully qualified crate::tests::live_entry call site imports it; the
  test_entry NOW_MS constant is declared before its uses.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 9 — f2a54c9: the items round 7 left open are closed.

  • One poisoned-lock error: DHTSyncLockError, CallbackSyncLockError, and StorageLockPoisoned are Error::LockPoisoned.
  • storage::sled::SledStoragestorage::file::FileStorage (module and test file renamed with it); it has been a file-per-key store under a byte budget since the budget landed.
  • MessageSigner no longer exposes session_sk(); the onion runtimes only needed session_public_key(), which it now provides.
  • StorageSyncBatch::new(msg, origin, now_ms) takes the admission clock instead of reading it in a constructor.
  • Retention fixtures are one family in crate::tests (live, live_entry, with_retention, expired); the local bounded / live_at / expired helpers are gone, every crate::tests::live_entry(...) call site imports the name, and test_entry.rs declares NOW_MS before its uses. The stateright model keeps its own MODEL_RETENTION_BOUND_MS = Some(u128::MAX): it is a model parameter (never expires under the checker), not a fixture.

CHANGELOG records the three renames under breaking changes. Gates: core dummy 727, node 323, clippy native / wasm core / browser / ffi, rustdoc, taplo, typos, nightly fmt.

…, retire per element

Fresh review of the relay inbox:

- The relocation law read its sender from the relay path, a peer-declared
  field; any connected node could forge a hand-off from the receiver's
  predecessor and inject held messages. The sender is now the transaction
  signer, the one origin the transport authenticated.
- The hold instant is signed by the holder and was bounded only above, so
  a holder could judge any old message at a time of its choosing and the
  documented replay bound did not exist. The write law now requires the
  held message's sender proof to be live by the owner's own clock; the
  witness stays timeless for relocation and delivery, which keeps them
  monotone.
- Authority is judged before any signature is verified and a relay delta
  larger than the inbox is refused before the first verification, so a
  stranger costs the owner nothing.
- The drain tombstoned only after delivering every element; cut short by
  the step deadline it redelivered the same prefix forever. It now retires
  each element as it is delivered, and builds the delivery pipeline once.
- The witness binds the payload's relay destination too, so a drained
  message is never forwarded; a relay carrier is absent to every lookup.

Also: Entry::partition_inbox / handlers::inbox::deliver_inbox naming,
EntryKind::{is_relay_inbox, replication}, topology::is_responsible_for as
a pure law with tests, SwarmCallbackSlot newtype, LocalDelivery,
MessageSigner::sign_at, MessageVerification::{verify_live, verify_live_at}
replacing verify_unexpired, DomainTag::new asserts its law once,
EntryWitness::Reference, SwarmTransport::session, placed_beyond,
StorageSyncPurpose::is_ownership_handoff, one retire-on-read step,
FileStorage docs and test shape, and the tests the review found vacuous.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 10 — d78f7fb: fresh three-reviewer read of the full diff, plus my own; everything below is fixed in this commit.

Correctness (relay inbox)

  1. Relocation sender was unauthenticated. relay_relocation_permits judged ctx.relay.try_origin_sender() (= relay.path[0], a peer-declared field that forwards unchanged) against the receiver's predecessor. Any connected node could set path = [pred_X, me] and inject a witnessed carrier into X's inbox, delivered on X's next pass with peer: None. The sender is now ctx.transaction.signer(), the one origin the transport authenticated (as the report handler already required). Test: test_sync_entries_handler_ignores_a_forged_relay_origin_for_a_relay_carrier, plus the relocation test now installs a predecessor and covers both the stranger and the predecessor.
  2. The hold instant was bounded only above. The holder signs it, so it could judge any historic message "as of" a chosen past instant and the documented replay bound did not exist. The write law (EntryOperation::validate_inbox_write) now also requires the held message's sender proof to be live by the owner's own clock (RelayMessageHoldStale); the witness stays timeless for relocation and delivery, which keeps them monotone. Replay is bounded by the sender's proof lifetime, as the docs claimed.
  3. Authority before signatures; bounded delta. Admission verified every element's signatures before checking who wrote it. The relay path now checks responsible == writer first, then witnesses each element once (witnessed_inbox_elements, no second decode), and refuses a delta larger than the inbox before the first verification (RelayInboxDeltaExceedsCapacity). Data topics keep their cap-at-materialization law.
  4. Drain progress is durable. The tombstone was written after delivering every element; a pass cut short by the step deadline redelivered the same prefix forever. deliver_inbox now retires rejected elements first, then delivers and retires element by element (Entry::partition_inbox pairs each element with its dot), building the pipeline (LocalDelivery) once.
  5. The witness binds relay.destination too, so a drained payload is never re-forwarded by the recipient's handler; a relay carrier is absent to every lookup (SearchEntry returns no data for it, whoever asks), since only local drain reads it.

Abstraction / naming / docs

  • topology::is_responsible_for(state, id) is a pure law with tests; PeerRing delegates.
  • EntryKind::{is_relay_inbox, replication} replace raw kind comparisons and the two shapes of "never replicated"; StorageSyncPurpose::is_ownership_handoff; PeerRing::placed_beyond; one retire-on-read step (retire_unless_live).
  • SwarmCallbackSlot is a newtype with current / replace; MessageSigner::sign_at makes the hold instant explicit; MessageVerification::{verify_live, verify_live_at} replace verify_unexpired and are the one composition of liveness + session + signature that Ext::verify_at uses; DomainTag::new asserts its law once (the macro is a const evaluation); EntryWitness::Reference; SwarmTransport::session (no key exposure); gen_default_entry(&self) drops a clone; Reachability replaces a bare bool.
  • Docs: ChordStorageSync mode law (relay geometric, data virtual), FileStorage placeholder docs, WriteOrderedMap clock exclusion, log message, CHANGELOG (32 KiB, no intra-PR deltas, freshness clause), inbox module doc names the failure-detection window where a hold is refused.
  • Tests: tombstone cap now witnessed with two full drains (== RELAY_INBOX_MAX_LEN, oldest dropped); hold law covers a responsible writer relaying another node's hold and a stale hold; both sides of the hold-instant boundary; test_file.rs first test rewritten to the file's standard.

Gates: core dummy 730, node 323, clippy native / wasm core / browser / ffi, rustdoc, taplo, typos, nightly fmt.

The stabilizer owned the application callback slot and called through
message::handlers::inbox into swarm::callback to deliver the inbox, so
the DHT layer depended on the swarm's delivery pipeline. It now names
the one intent the storage maintenance phase has toward the application
layer, dht::InboxDelivery::deliver_inbox, and Swarm::stabilizer injects
the swarm's interpreter, swarm::inbox::SwarmInboxDelivery, which owns the
callback slot and the LocalDelivery pipeline. message::handlers::inbox
keeps only the producer (holding for an offline destination).

The tombstone-cap test read a synthetic clock while its holder
signatures read the real one; two rounds of signing outran the skew
tolerance and the witness refused the holds. It reads the real clock.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 11 — b28dede: inbox delivery is an intent the DHT emits and the swarm interprets.

  • dht::stabilization no longer holds the application callback or calls into message::handlers::inbox / swarm::callback. It defines dht::InboxDelivery (one method, deliver_inbox), the single intent the storage maintenance phase has toward the application layer, and Stabilizer::new(transport, SharedInboxDelivery) takes its interpreter.
  • Swarm::stabilizer injects swarm::inbox::SwarmInboxDelivery, which owns the SwarmCallbackSlot and the LocalDelivery pipeline; the drain body moved there unchanged (retire rejected elements, then deliver-and-retire per element). message::handlers::inbox keeps only the producer.
  • Dependency direction is now dht → (trait) ← swarm, the same shape as StorageRepairDueRequestStorageRepair.
  • One test fix: test_inbox_keeps_the_newest_elements_and_bounds_its_tombstones judged holds against a synthetic clock while their signatures read the real one; two rounds of signing (~4 s) outran the 3 s skew tolerance and the witness refused the second round. It reads the real clock at every step.

Gates: core dummy 730, node 323, clippy native / wasm core / browser / ffi, rustdoc, taplo, typos, nightly fmt.

…ne placement

Two ways a relay inbox could be denied or misplaced by a peer:

- Storage slots were keyed by placement alone, so a data topic any node
  parks at `d + 1` through `storage_store` made every hold for `d` fail
  with `EntryKindNotEqual` for as long as the topic was refreshed.
  `StorageKey = (kind, placement)` partitions storage by kind; relay
  slots render as `relay:<placement>`, data slots keep the bare
  placement so earlier values stay addressable, and a lookup reads the
  data slot only (the relay-absent branch of `SearchEntry` is now a
  property of the key, not a match).
- The affine placement set was derived from the configured redundancy
  for every kind, so a placed operation or synced entry could name a
  rotated replica key for a relay carrier and be admitted there under
  the authority for that position's own inbox. The set is now
  `rotate_affine(kind.replication(redundancy))`: one placement, the
  DID, for a relay carrier.

Tests: key law (partition, rendering, round trip), the parked topic and
the hold coexisting at one position through the write funnel, and the
single-placement law for placed operations and synced entries. The
inbox fixture moves to `crate::tests::held_inbox_for(destination,
holder)`.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 12 — bbf7636: storage partitioned by kind, one placement per relay carrier

A fresh read of the whole diff found two ways a peer could deny or misplace an inbox; both are fixed in this commit.

1. Any node could block a peer's inbox by parking a data topic at its position. Storage slots were keyed by placement alone, and storage_store lets a node overwrite a Data carrier at any DID. Parking one at d + 1 made every hold for d fail at the owner with EntryKindNotEqual (Entry::join refuses to merge kinds) for as long as the topic was refreshed. Fix: StorageKey = (kind, placement) is the identity of a stored carrier; relay slots render as relay:<placement>, data slots keep the bare placement so values written by earlier builds stay addressable, and the law from_str(key.to_string()) == key is tested. Every read and write of DHT storage goes through it (live_storage_entry, live_storage_entries, join_storage_entry, operate_storage_entry, ack-gated removal, hand-off, virtual-owner copy). A lookup reads the data slot only, so the relay-absent branch in the SearchEntry handler is gone: it is a property of the key now, not a match on the value found.

2. A relay carrier could be admitted at a rotated replica key. PlacedEntryOperation::validate_placement and PlacedEntry::validate_placement derived the affine set from the configured redundancy for every kind, so a hold naming rotate_affine(d + 1)[i] was in-set, and the owner of that position judged the hold under the authority for that position's inbox (inbox_hold_authority(placement - 1)) while the elements were witnessed against entry.did. The set is now rotate_affine(kind.replication(redundancy)): one placement, the DID, for a relay carrier.

Tests: key partition/rendering/round trip; a parked topic and a hold coexisting at one position through the write funnel (test_data_topic_at_the_inbox_position_does_not_block_the_hold); the single-placement law for placed operations and synced entries. The inbox fixture is now crate::tests::held_inbox_for(destination, holder); the storage wait helpers take a StorageKey. CHANGELOG, SECURITY.md, the inbox module doc, and the PR body are updated.

Checked and not changed: HeldMessage tombstones carry only dots, so the tombstone witness is trivially satisfied and cannot be used to verify signatures for free; acknowledge_synced_entries takes the kind from the ack, which take_pending_storage_sync_ack matched exactly against the recorded outbound batch; MeasureStorage is a separate store, so the snapshot key never reaches live_storage_entries.

Gates: core (dummy) 733, node 323, clippy native/wasm core/browser/ffi, rustdoc -D warnings, taplo, typos, nightly fmt.

…mped value is retired

The browser test `test_storage_repair_load_does_not_starve_three_node_stabilization`
seeded node1's storage with `Entry::new` values, which carry no retention bound. Under
the retention law such a value is not live, so the first read retired it and the repair
pass never republished it; the test has failed in CI on every commit of this branch
since retention landed. The fixture now uses `crate::tests::live_entry`, as the native
repair tests already do.

A storage-level test pins the law that bit the fixture: a value written without a
bound is reported absent, removed, and never republished.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 13 — 1d51e1d: the CI failure on every commit of this branch

QACI's "Run core browser tests" step has failed on every push since retention landed, always in test_storage_repair_load_does_not_starve_three_node_stabilization ("real browser repair did not persist both remote placements"). Cause: the browser test seeded node1's storage with Entry::new values, which carry no retention bound; under the retention law an unstamped value is not live, so the first read retired it and the repair pass never republished it. The native repair tests were already on crate::tests::live_entry; the browser fixture was not, and I had not run the browser suite locally.

Fix: the fixture is stamped live. A storage-level test pins the law that bit it, test_live_storage_read_retires_an_unstamped_value: a value written without a bound is reported absent, removed, and republish_local_entries emits nothing.

Verification: the rest of the core browser suite (28 tests) passes locally under headless Chrome; the three-node test itself cannot complete on my machine on master either (the second WebRTC offer from one node never gathers, a local ICE environment limit, 60-second budget hit before any storage step), so its pass has to come from CI. The dummy-test failure on d78f7fb (test_inbox_keeps_the_newest_elements_and_bounds_its_tombstones, synthetic clock vs. the 3 s skew tolerance on a slow runner) was fixed in b28dede and has passed since.

`FileStorage` retired a record from its index before removing the file
(`retire_until_fits`, `clear`), so a removal the file system refused
left an unindexed file consuming the budget until the next open, against
the module's own index law; a failed retirement during `put` also left
the temporary file behind. Retirement now removes first and forgets
after (`WriteOrderedMap::oldest` peeks; `retire_indexed` is the one
place a record leaves the index), `put` commits through `commit_record`,
which restores the previous record on any failure and always cleans its
temporary file.

Tests: a record whose path is occupied by a directory stays indexed and
counted, the write that needed its bytes fails without a temporary file,
and succeeds once the file is back; `oldest` peeks without removing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 14 — 03ff70c: fresh full read of origin/master..HEAD

This round I read the parts the earlier rounds only skimmed: the storage backends (FileStorage, WriteOrderedMap, MemStorage), the CRDT and stabilization-handler changes, the transfer and onion signing changes, and every test file, then re-read the inbox core with the new StorageKey in mind.

Found and fixed: the file store forgot a record before its file was gone. retire_until_fits and clear popped the oldest record from the index and only then removed its file, so a removal the file system refused left an unindexed file holding budget bytes until the next open, against the module's own index law ("the index is updated only after the file system operation succeeded"). A retirement that failed inside put also left the .tmp file behind. Now WriteOrderedMap::oldest peeks, retire_indexed is the one place a record leaves the index (remove first, forget after), and put commits through commit_record, which restores the previous record on any failure and always cleans its temporary file. Tests: test_refused_retirement_keeps_the_record_indexed occupies a record's path with a directory, so its removal is refused on every platform; the write that needed its bytes fails, the record stays indexed and counted, no .tmp remains, and the same write succeeds once the file is back. test_oldest_peeks_without_removing pins the map law.

Checked and not changed (with the reason, so you can disagree):

  • FileStorage::get and get_all decode outside the read guard and retire an undecodable record under the write guard; no lock is held across the file read and the retirement, so a concurrent put of the same key serializes before or after, never inside.
  • A recipient's inbox tombstone joins its 24 h bound into the carrier, so a fully drained carrier lives up to 24 h as tombstones only. Bounded by MAX_RELAY_INBOX_TTL_MS at admission and identical to how a data-topic tombstone extends its carrier.
  • LocalDelivery rebuilds an InboundProcessor (handler, reassembler) once per maintenance pass so the callback is resolved at delivery time; the reassembler is never fed. A per-pass allocation on a 500 ms cadence, not a hot path.
  • is_responsible_for with predecessor == Some(self) would hold for nothing; that state is unreachable, since a node never notifies itself and the notify handler rejects an origin mismatch.
  • rectify_predecessor keeps a stale predecessor only while no closer candidate has notified; a returning peer that connects to its successor alone learns its predecessor from the predecessor's next stabilize round, which the end-to-end test waits on.
  • The relocation law does not re-check the sender-proof freshness at the receiver: the hold instant was judged fresh by the owner that admitted it, and judging as of that instant is what keeps relocation and delivery monotone.

Gates: core (dummy) 735, node 323, clippy native/wasm core/browser/ffi, rustdoc -D warnings, taplo, typos, nightly fmt; CI on the previous commit was fully green.

r2d2-rs and others added 2 commits September 6, 2026 21:40
…dopt self as predecessor

`FileStorage::get` and `get_all` read a record under the read guard,
release it, and retired the record if it did not decode. A `put` of the
same key between the read and the retirement was then deleted with the
garbage it replaced. `retire_observed` re-reads under the write guard
and removes the file only while it still holds the observed bytes.

`rectify_predecessor` now leaves the current value when the candidate
is the local node, so `(pred, local]` can never be emptied by a
self-reference; the shell already made this unreachable, the pure
transition is now total on its own.

Tests: an undecodable record survives a retirement for bytes it no
longer holds and is retired by the read that observes it; a self
candidate leaves the predecessor and the responsibility interval alone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 15 — dd5b738, 128e8e1: the round-14 "checked" items, fixed where they were real, and a re-read of rounds 12–15 together

Two of the six items I had listed as checked were defects once looked at again, and both are fixed with tests:

  • A read could delete a concurrent writer's record. FileStorage::get and get_all read a record under the read guard, released it, and retired the record if it failed to decode. A put of the same key that landed between the read and the retirement was deleted together with the garbage it had replaced. retire_observed re-reads the file under the write guard and removes it only while it still holds the bytes the read observed; the decode law now says so. Test: test_undecodable_record_is_retired_only_while_unchanged (a retirement for bytes the file no longer holds leaves it; the read that observes the garbage retires it).
  • rectify_predecessor could adopt the local node as its own predecessor. The shell made that unreachable (the notify handler binds the candidate to the authenticated origin), but the pure transition was not total: a self candidate would have emptied (pred, local] and switched every hold off. It now leaves the current value; law and step test added (test_rectify_never_adopts_the_local_node_as_predecessor).

The other four stay, with the reason: the drained inbox living 24 h as tombstones is the lattice law (a join's bound is the max, for tombstones as for adds; special-casing removal would make the bound depend on the operation, not the value); LocalDelivery rebuilding an InboundProcessor per pass allocates empty maps and clones Arcs, on a 500 ms cadence; a stale predecessor is kept until a closer candidate notifies, which is CorrectChord's rectify; relocation judges as of the hold instant by design, since the owner that admitted the hold is the one that checked freshness.

Re-reading rounds 12–15 as one diff found nothing further; the sync-report tests now name the inbox slot as a StorageKey like the end-to-end tests do (128e8e1).

Gates: core (dummy) 738, node 323, clippy native/wasm core/browser/ffi, rustdoc -D warnings, taplo, typos, nightly fmt. The CI runs for 03ff70c and dd5b738 were cancelled by the pushes that followed them; the run on 128e8e1 is the one to read.

…ry runs the logical stage alone

Retention is refreshed by what is held, never by a removal: `Entry::tombstone`
keeps the carrier's own bound instead of joining the removal's, so an inbox
drained to tombstones expires when its last hold would have. The law is
stated in the retention module and pinned for both carrier kinds.

The inbound pipeline's connection-independent stage (application validation,
handler dispatch, `on_inbound`) is now `LogicalInbound`; `InboundProcessor`
composes it behind reassembly and admission, and `LocalDelivery` runs it
alone, so a relay-inbox drain no longer builds a reassembler it never feeds.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 16 — a64d833: the two remaining round-15 items, fixed

  • A removal no longer extends a carrier's retention. Entry::tombstone keeps the carrier's own bound instead of joining the removal's, for a data topic and a relay inbox alike. Retention is refreshed by what is held (adds, overwrites, compaction floors), never by a removal, so an inbox drained to tombstones expires when its last hold would have instead of outliving it by the removal's 24 h. The law is stated in the retention module (Removal), the join test no longer claims the opposite, and test_removal_leaves_the_retention_bound_unchanged pins it for both kinds. Compaction still joins its floor's bound: it is a publisher's reset, not a removal.
  • Local delivery runs the logical stage of the inbound pipeline alone. The connection-independent stage (application validation under the deadline, handler dispatch, on_inbound) is now LogicalInbound; InboundProcessor composes it behind reassembly and the admission gates, and LocalDelivery is built over LogicalInbound directly. The relay-inbox drain no longer constructs a reassembler and a pending-attempt slot it never uses, and the pipeline functions validate_payload / process_logical_message / deliver_local take the stage they need rather than the whole processor.

The two protocol items stay as they are, because they are the protocol: a stale predecessor is kept until a closer candidate notifies or failure detection removes it (CorrectChord rectify, with Remove already clearing a departed predecessor), and relocation judges a held message as of its hold instant because the owner that admitted the hold is the one that checked freshness; re-judging at the receiver would make admissibility non-monotone again.

Gates: core (dummy) 739, node 323, clippy native/wasm core/browser/ffi, rustdoc -D warnings, taplo, typos, nightly fmt. CI on 128e8e1 was fully green; the run on a64d833 is the one to read next.

…te delivery seams

With a removal no longer joining its own bound, a tombstone applied to an
absent carrier produced a result without retention that the write funnel
persisted and the next read retired. `operate_storage_entry` now writes only
a live result and leaves the slot empty otherwise, so a stored value is
always live when written (test for a data topic by value and a relay inbox
by dot).

Exposure: `InboxDelivery`, `Stabilizer::new`, and `SwarmCallbackSlot` are
crate-private; nothing outside the crate can reach a `SwarmTransport` to
build a stabilizer with, and the callback slot is `Swarm::set_callback`'s
implementation. `StorageKey::inbox_of(destination)` names the inbox slot
once instead of composing kind and key at every site.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 17 — b0f33c8: fresh read with the placement and exposure passes

This pass asked two questions of every type the branch adds: is it where it belongs, and who can reach it; plus a doc-vs-code sweep after round 16's retention change.

Found and fixed

  • A removal against nothing held persisted a dead value. Once a tombstone stopped joining its own bound (round 16), applying one to an absent carrier produced a result with no retention, which operate_storage_entry stored and the next read retired. The write funnel now writes only a live result and leaves the slot empty otherwise, so the invariant "a stored value is live when written" holds for every operation. Test test_removal_against_nothing_held_stores_nothing: a data topic by value, and a relay inbox by a dot the recipient once saw, both leave storage empty.
  • Three seams were pub with no reachable use. InboxDelivery (and its shared alias), Stabilizer::new, and SwarmCallbackSlot were exported from the crate although no external code can obtain a SwarmTransport to build a stabilizer with, and the slot is only Swarm::set_callback's implementation. All three are crate-private now; the unused re-export of the alias is gone.
  • The inbox slot was composed at every site. StorageKey::inbox_of(destination) names it once; the production site and the end-to-end test helper use it, and the storage tests read the slot by the same name.

Checked, unchanged: the dht → message dependency the witness introduces (HeldMessage holds a MessagePayload) is the one place the carrier must know its element type, and dht::entry already depended on message::Encoded; EntryKind::{max_data_len, replication, max_tombstones} stay pub as plain policy accessors; retention wording in CHANGELOG, SECURITY.md, and the PR body already reads "after the last hold", which is exactly the round-16 law.

Gates: core (dummy) 740, node 323, clippy native/wasm core/browser/ffi, rustdoc -D warnings, taplo, typos, nightly fmt. CI on a64d833 is superseded by this push; b0f33c8 is the run to read.

…per ring

Every storage operation read a slot, computed, and wrote it back with no
ordering between callers, and the store orders single puts only. The
inbound actor and the stabilizer now write the same slots concurrently:
a hold arriving while the recipient drains its inbox, a hand-off joining
while a repair pass reads. Two interleaved read-modify-writes overwrote
each other, and a held message could be lost.

`PeerRing::storage_transition` (an async mutex) now covers
`operate_storage_entry`, `join_storage_entry`, the ack-gated removal
(`remove_storage_entry_confirmed_by`, so the comparison and the removal
are one transition), and the retirement a read performs. It is held
across the store's own awaits only.

Test: a store double that yields at every access interleaves two holds,
and a hold with a removal, on one inbox; both holds survive. Without the
transition the first pair keeps one element.

Also inlines the one-use `Stabilizer::deliver_inbox` wrapper.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 18 — df7f64b: concurrency pass

This pass asked, for every storage slot, who writes it and whether their writes are ordered.

Found and fixed: two interleaved read-modify-writes on one slot overwrote each other. Every storage operation reads the slot, computes, and writes back, and the store orders single puts only; nothing ordered the callers. Before this branch that was latent (one writer per slot in practice). The relay inbox makes it real: the inbound actor writes a hold into d + 1 while the stabilizer's drain tombstones the same carrier, and a hand-off joins while a repair pass reads and retires. Whichever write landed second won, and a held message could be lost. PeerRing::storage_transition (an async mutex, held across the store's own awaits and nothing else, never nested, never waiting on the network) now covers operate_storage_entry, join_storage_entry, the ack-gated removal (remove_storage_entry_confirmed_by, so the value comparison and the removal are one transition and an ack for an older value cannot remove a newer write), and the retirement a read performs. Test test_interleaved_operations_on_one_slot_do_not_lose_a_write uses a store double that yields to the executor at every access, so two holds, and a hold racing a removal, interleave at each storage await; both holds survive. I verified the test fails without the transition (one element left of two) and passes with it.

Also inlined the one-use Stabilizer::deliver_inbox wrapper at its call.

Checked, unchanged: the fetched-entry cache is put-only (no read-modify-write), so it stays outside the transition; SwarmCallbackSlot and FileStorage's index are guarded by their own locks, held only for the clone or the file operation.

Gates: core (dummy) 741, node 323, clippy native/wasm core/browser/ffi, rustdoc -D warnings, taplo, typos, nightly fmt. CI on b0f33c8 is superseded by this push; df7f64b is the run to read.

`validate_inbox_relocation` returned a `Result` whose error was only ever
turned into a boolean by the sync batch, which skips a carrier it may not
take instead of failing. It is now `relocates_from_predecessor`, a
predicate, and the phantom `Error::RelayInboxNotRelocatable` is gone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rings-auto-reviewer

Copy link
Copy Markdown
Contributor

Round 19 — fa65c24: vocabulary and documentation pass

This pass swept the repository for identifiers the branch renamed or removed (SledStorage, storage::sled, the three poisoned-lock errors, matches_network, verify_unexpired, MessageVerification::new, the round-4/5 step names) in code, docs, and scripts: nothing stale remains outside the CHANGELOG, which records them on purpose. It then checked every error variant the branch adds for a place that surfaces it.

Found and fixed: a law that returned an error nobody could receive. validate_inbox_relocation produced Error::RelayInboxNotRelocatable, but its only caller, the sync batch, turned the result into a boolean, because a relay carrier offered by someone other than the predecessor is not invalid, only not this receiver's to take yet, so the batch skips it rather than failing. The function is now the predicate it always was, relocates_from_predecessor, and the phantom variant is gone; the unit test asserts the predicate. Every other variant the branch adds is surfaced by at least one path.

Also run this round: the local browser suite (28 tests, the three-node WebRTC case excluded as before) passes on the storage transition, so futures::lock::Mutex behaves on the single-threaded target as well.

One observation outside this PR's scope, not changed: test_tracked_cleanup_grace_terminalizes_a_nonresponsive_generation (tests/default/test_chunk_e2e.rs, untouched by this branch except the signer API) bounds the transport's cleanup grace with a 1-second wall-clock timeout; it missed that bound once while four clippy builds and the node suite ran beside it, and passes alone and in the suite run without them. It tests a timeout, so a duration is inherent, but the margin between the grace and the bound is what decides flakiness under load; worth its own issue if it recurs in CI.

Gates: core (dummy) 741, node 323, clippy native/wasm core/browser/ffi, rustdoc -D warnings, taplo, typos, nightly fmt. CI on df7f64b is superseded by this push; fa65c24 is the run to read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment