test(antithesis): model-driver query coverage — filters, indexes, typed metadata (EN-1625)

[Azorlogh]

Scope note: this PR is test-only (tests/antithesis, tests/oracle). The server-side fixes the model checking surfaced have all merged into release/v3.0: #1826 (degenerate range bounds), #1829 (EN-1724 version-bound types), #1830 (index-registry preload stale absence), #1769 (index registry restore), #1809 (forwarded-stream cancellation). The branch is rebased onto that base.

Summary

Extends the model-based antithesis driver (singleton_driver_model) with validated coverage of the filtered, paginated read surface: every list page is checked as an exact ordered window against the oracle, across the full index lifecycle.

Phase 1 — index lifecycle + has-asset: the generator churns CreateIndex/DropIndex; the oracle tracks each index as absent/ambiguous/active with a per-replica readiness poller (GetIndexStatus, promote on CurrentVersion>0 everywhere, demote on any not-ready). everAsset models the account-by-asset projection — asset touches recorded at END of bulk mirroring the exclusion projection exactly.

Phase 2 — tx builtins: reference (exact) and the three date fields (timestamp / inserted_at / reverted_at ranges). Server-stamped dates are LEARNED from commit-response logs (LearnTxStamps) — deterministic FSM outputs, upgrading read validation from skip-if-unknown to equality. matchTxFilter is three-valued Kleene: an unlearned stamp makes the row optional in the predicted window (txWindowMatches handles optional rows with exact pageSize-truncation semantics).

Phase 4 — address-on-transactions: per-tx account→tx index membership (indexedAddrs, src/dst bits) stamped at end of bulk under the same cellExcluded verdict as everAsset; evaluation mirrors the server's two-step resolution (V+M attributes universe, then the mapping index) — a purged, metadata-less account stops reaching its transactions.

Phase 3 — metadata Field indexes: string/int(+datetime)/uint/bool/exists(include_null) on both targets, evaluated with the server's own coercion (commonpb.TypeMatches/ConvertMetadataValue). The compiler's per-leaf order (schema → ready → kind coercion) is one shared per-candidate verdict; kind mismatches must reject as FILTER_COMPILATION_ERROR. CreateIndex on undeclared fields and RemoveMetadataFieldType's DroppedIndex are modeled and probed. Retypes of INDEXED keys are generated too (EN-1724's per-version bindings, #1829, made the serving window well-defined): the driver models each retype's serving window with a two-phase closure proof over GetIndexStatus (last_indexed_sequence + pending_version), accepts either binding while the window is open, and requires the new binding once closed.

Harness / infrastructure scope (beyond query coverage)

These ship in the same PR because the query soaks depended on them; each is isolated and individually documented:

• Antithesis launch interface: bearer ANTITHESIS_API_KEY/ANTITHESIS_TENANT auth, k8s-launch-minutes, check-run, and a 72-hour check-k8s-image-freshness gate in tests/antithesis/Justfile — soak dispatches against week-old images kept invalidating results.
• env -u SOURCE_DATE_EPOCH on four Justfile build recipes: reproducible-build stamping broke image-freshness detection.
• Local model runs (run_model_test.sh) build servers with -tags invariants and plumb MODEL_DUMP_BATCHES for deterministic offline replay of findings.
• WRITES_BLOCKED_DISK_FULL reclassified as transient in workload/internal/client.go: fault-injected disk pressure is an environment condition, not a server verdict; treating it as fatal aborted soaks on the injector's schedule. This applies to every workload, not just the model driver.

Bugs this work surfaced (all fixed)

• fix(state): purge stranded pre-transient volume rows at zero balance #1645 — stranded pre-transient volume rows (ghost accounts in ListAccounts)
• fix(indexbuilder): clear LedgerLog volume-annotation lists on cursor reuse #1655 — index builder cursor-reuse leak silently dropping posting-index writes
• EN-1597 iterator family (fix(EN-1597): honor absolute SeekGE/SeekLE reposition-after-exhaustion in read-store iterators #1635) — the AddressTxIterator leg re-exposed by phase-4 filters
• Plus EN-1624 filed: the checker cannot detect stranded rows (compareVolumes skips excluded keys)

Notes

• Carries cherry-picked EN-1597 iterator fixes (a6c148f equivalent) — collapses to a no-op when fix(EN-1597): honor absolute SeekGE/SeekLE reposition-after-exhaustion in read-store iterators #1635 merges.
• Merged with test(antithesis): model-driver idempotency replay + conflict coverage #1650 (idempotency replay coverage); the union validated by 600s single-node run. Phases individually validated by repeated 600s soaks + 3-node cluster runs with rolling restarts (zero findings on final builds).

Jira: EN-1625

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[NumaryBot]

✅ Approve — automated review

No actionable correctness defects remain in the current diff. The active prior disk-full matcher finding is fixed by requiring both ResourceExhausted and WRITES_BLOCKED_DISK_FULL.

No findings.

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1659 (comment)

[NumaryBot]

NumaryBot posted 2 new inline findings.

Summary: #1659 (comment)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.32%. Comparing base (e3ef770) to head (670e87c).

Additional details and impacted files

@@               Coverage Diff                @@
##           release/v3.0    #1659      +/-   ##
================================================
+ Coverage         77.31%   77.32%   +0.01%     
================================================
  Files               458      458              
  Lines             48243    48243              
================================================
+ Hits              37300    37305       +5     
+ Misses             7807     7804       -3     
+ Partials           3136     3134       -2     

Flag	Coverage Δ
e2e	77.32% <ø> (+0.01%)	⬆️
scenario	77.32% <ø> (+0.01%)	⬆️
unit	77.32% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[flemzord]

Reviewed live head b064d0dc. The production iterator changes preserve absolute re-seek semantics and are covered by focused regressions; the functional build, unit, E2E, scenario, model, and Schemathesis checks are green. The current NumaryBot findings concern Antithesis/model-oracle fidelity, while Dirty and Codecov are CI/tooling. They remain actionable but do not demonstrate a correctness/runtime regression under this cron policy. I found no correctness/runtime blocker on the current head. GitHub approval is intentionally disabled by the cron configuration.

[gfyrag]

Independent review at head 7bb4aeb. Four blocking findings remain; this draft is also conflicting and has no functional CI on the current head.

[P1][blocking] Rebuild index registry state during incremental restore
Location: internal/domain/replay/replay.go (ReplayLedgerLog); demonstrated by tests/e2e/cluster/restore_index_registry_test.go
Evidence: CreateIndex and DropIndex are discarded during RebuildDelta. The added red test proves the restored registry is empty and later metadata removal cannot report/drop the index.
Impact: disaster recovery loses a business-visible primary projection and diverges from the read-side lifecycle. PR #1769 addresses this separately but does not fix this head.

[P2][blocking] Compare learned inserted_at in transaction row validation
Location: singleton_driver_model/queries.go (txRecordMatches)
Evidence: the oracle learns InsertedAt and uses it for filters, but the matcher never compares it with the server row.
Impact: wrong inserted_at responses pass the advertised equality oracle.

[P2][blocking] Bind index readiness to the current incarnation/frontier
Location: singleton_driver_model/indexes.go (reconcileIndexes)
Evidence: a stale status sampled before DropIndex/CreateIndex reuse can promote the new model entry without proving LastIndexedSequence reaches its frontier or identifying the incarnation.
Impact: legal drop/recreate executions can produce false Antithesis findings.

[P2][blocking] Synchronize the HTTP/API contract
Location: internal/adapter/http/error_handler.go, API docs, api-comparison, OpenAPI 503 response
Evidence: all KindUnavailable responses now receive Retry-After, while docs still describe no-leader-only behavior and omit READ_INDEX_NOT_CAUGHT_UP and CHAPTER_ARCHIVE_OUT_OF_ORDER.
Resolution: document the broad retry contract and both public reasons/status/metadata.

Targeted root/nested model tests and tagged E2E compilation passed; the new restore scenario is knowingly red.

[NumaryBot]

NumaryBot posted 2 new inline findings.

Summary: #1659 (comment)

[shipfox-ai]

Arbitrated review — PR #1659 (EN-1625): model-driver query coverage

Summary. This PR is mostly antithesis/oracle test code but also ships production "server fixes" for the EN-1724 retype window. After both reviews, both rebuttals, and my own reading of the code at HEAD 9d60030, the two reviewers converged on six findings and I verified every one against the tree. The query-bounds, checker exclusion-scan, WAL snapshotter, index-version encoding and preload changes are sound and adequately tested. However, there is one genuine production correctness defect — the creation version is bound to the latest schema rather than the schema in force at the CreateIndex sequence — that ships untested for the exact chronology it can break, plus a major test-harness attribution defect in the readiness poller that can both raise false failures and mask real cross-replica divergence. Both are in the critical path this PR exists to protect. Recommendation: request changes.

Agreed findings

1. [major] Creation version is bound to the latest schema, not the schema at the CreateIndex sequence. internal/application/indexbuilder/index_config.go:411-432 (boundTypeAtCreation). The doc comment promises "the schema entry in force when the CreateIndex log folds," but the implementation resolves the type via b.batchSchema.For(ledgerName), a per-batch resolver whose contract is "a schema view at least as fresh as every log the batch is about to process" — it can be fresher than the CreatedIndexLog being folded (which carries only Id/Initial). A batch, or a lagging/restored replica, spanning CreateIndex@N and a later retype @M>N binds the creation version (v1 / v_current) to the post-retype type. Writes then coerce under that version-bound type (builder.go:286-310) and query compilation substitutes it for the pinned schema type (query/compile.go), so two replicas can encode/validate the same field condition under different declared types at the same pin — cross-replica read divergence, the very failure EN-1724 exists to prevent. Especially concrete for an initial index (immediately promoted to current) and for frozen checkpoint/point-in-time pins.

2. [major] Per-node readiness polls can be served by the leader. tests/antithesis/workload/bin/cmds/model/singleton_driver_model/indexes.go:605-606. GetIndexStatus is called with the plain context, which defaults to linearizable consistency (internal/adapter/grpc/consistency.go:32-38). RoutedController.GetIndexStatus routes through readCtrl (internal/bootstrap/controller_routed.go:438-444), which transparently forwards a still-syncing non-leader's read to the leader (controller_routed.go:104-131). A syncing follower can therefore be credited with the leader's CurrentVersion/PendingVersion/LastIndexedSequence, promoting an index or closing a retype window while that replica is still building or serving the old encoding — defeating the per-node attribution contract stated at tests/antithesis/workload/internal/pernode.go:20-25. The repo already provides internal.WithStaleConsistency(ctx) (pernode.go:181), used correctly in eventually_cross_node_identity/main.go:312; apply it to each status request.

3. [minor] Stale poll can promote a newly recreated index (drop+recreate incarnation race). indexes.go:633-641. The poller snapshots only canonical names, runs the RPCs unlocked, and on reacquiring c.mu guards only on existence-by-canonical. The canonical key carries no per-creation incarnation (internal/domain/indexes/id.go:129-147) and the oracle stores only Map[string, bool] (tests/oracle/accessors.go:85-96), so a readyAll computed from a pre-drop incarnation can mark the post-recreate building incarnation active. Track a lifecycle generation/create sequence and require it unchanged before applying the result.

4. [minor] Stale/misleading documentation for dualWriteMetadataIndex. internal/application/indexbuilder/builder.go:251-262 still states the value "is identical across versions — the live path always coerces to the current declared type," which is precisely the pre-EN-1724 behavior this PR removes: coerceForVersion now encodes each version under its own bound type (builder.go:286-317). The same obsolete claim is repeated in the authoritative subsystem docs at docs/technical/architecture/subsystems/indexer/indexer.md:93 and :225-227; fix both.

5. [minor] Dangling doc comments for functions that do not exist. internal/application/indexbuilder/process_logs.go:1081-1113. Three doc blocks — extractMetadataKeyFromReverseMap, parseReverseMapKey, extractVersionFromReverseMap — are attached to nothing (the first sits above the unrelated isExcluded; the other two terminate the file). A non-comment grep for all three identifiers returns nothing. Remove them or restore the functions.

6. [minor] Oracle blind spot: txRecordMatches never checks inserted_at. tests/antithesis/workload/bin/cmds/model/singleton_driver_model/queries.go:623-644. The txRecordView interface exposes InsertedAt() (:611) and the model learns and filters on it (:766, :1352-1356), but the row-equality check compares id/reference/revert fields/timestamp/postings/metadata and never serverTx.GetInsertedAt(). A server returning a wrong inserted_at on committed transactions passes validation. (Adding the equality check proves subsequent consistency with the learned stamp; it cannot independently prove the original server-generated value.)

Ruled disputes

• Severity of the drop+recreate poller race (finding 3). Codex rated it major; Claude rated it minor and Codex conceded on rebuttal. Ruling: minor. Premature activation only narrows the oracle's tolerance (it starts expecting results where the server legitimately returns INDEX_BUILDING), so it can manufacture a false Antithesis failure but cannot broaden accepted results or mask a real bug. Independent of finding 2 — both fixes are required.

• Is the schema reader an "FSM Pebble snapshot"? (part of finding 1). Claude wrote the resolver is "opened over an FSM Pebble snapshot at the start of processLogs"; Codex disputed this. Ruling: Codex is correct. processLogs uses b.pebbleStore.NewDirectReadHandle() (process_logs.go:36-47), documented explicitly as "backed by the DB directly (no snapshot)" (internal/storage/dal/reader.go:73-90). This does not weaken finding 1 — it slightly strengthens it, since the memoized lookup can observe state committed after the log iterator was opened. The correctness defect stands.

• Was EN-1724 shipped with a "deliberate scope cut" that skips indexed-key retypes? (part of finding 1). Claude asserted the model deliberately does not generate retypes of indexed keys, leaving the path untested; Codex disputed this. Ruling: Codex is correct on the facts. generateSetMetadataFieldType retypes indexed keys "like any other" (tests/antithesis/workload/bin/cmds/model/singleton_driver_model/actions.go:930-947), no "deliberate scope cut" comment exists in the tree, and there are focused per-version and per-replica tests (internal/application/indexbuilder/retype_window_test.go, tests/e2e/cluster/metadata_index_per_replica_consistency_test.go). However, none of those cover the specific chronology in finding 1 — a delayed CreateIndex@N folded together with a later retype @M>N observed through a newer schema view — so the regression itself remains untested. Finding 1 stands; only Claude's stated justification for why it is untested is wrong.

Reviewed by Claude (claude-opus-4-8) and Codex (gpt-5.6-sol) via Shipfox; arbitrated by Claude.

[NumaryBot]

NumaryBot review complete: no remaining inline findings.

Resolved 1 stale NumaryBot review thread (0 fixed, 1 outdated).

Summary: #1659 (comment)

[gfyrag]

Independent review at exact head 35c788e27c0fa9f4c3c49ff4887b4a6d65b15fed.

[P2][blocking] Make per-node readiness reads locally attributable
Location: tests/antithesis/workload/bin/cmds/model/singleton_driver_model/indexes.go:605-607
Evidence: GetIndexStatus uses the normal linearizable context, so a follower request may be forwarded to the leader. internal.WithStaleConsistency already exists for locally served per-node observations.
Impact: a rebuilding follower can be credited with the leader's ready state, masking cross-replica divergence.
Resolution: call each status RPC with internal.WithStaleConsistency(ctx).

[P2][blocking] Bind readiness results to the sampled index incarnation
Location: singleton_driver_model/indexes.go:589-641
Evidence: after polling without the checker lock, the result is applied by checking only that the same canonical ID still exists. Drop+recreate can reuse that canonical for a new building incarnation.
Impact: stale readiness can promote the wrong incarnation and manufacture or mask model findings.
Resolution: snapshot a lifecycle generation/create frontier and require it unchanged before applying the poll.

[P2][blocking] Compare learned inserted_at in transaction equality
Location: singleton_driver_model/queries.go:601-643
Evidence: InsertedAt() is learned and used by filters but omitted from txRecordMatches.
Impact: an incorrect server inserted_at value passes the advertised equality oracle.
Resolution: compare it once learned and add a mismatched-value regression.

The nested model unit package passes, but the current distributed Tests-Model job is red (ListLogs returns an internal error). The PR body also depends on #1817, which is now closed without merge; replace that dependency explicitly before merge.

[Azorlogh]

Addressed the three blocking findings in 1b651b9 (rebased onto release/v3.0 first — all the server fixes this branch validated are now merged, body's scope note updated):

1. GetIndexStatus polls now run with WithStaleConsistency — the per-node question gets a locally-attributable answer instead of a syncing follower forwarding to the leader.
2. Readiness is bound to the sampled incarnation via a create frontier (committed CreateIndex sequence, recorded at fold): reports from indexers that haven't folded past the create don't count, and a verdict whose frontier moved mid-poll is discarded. Regression drives promote / pre-create-cursor / mid-poll-recreate.
3. txRecordMatches compares the learned inserted_at (nil stays server-dated); serverTxFromRec carries it; mismatch regression added.

The Tests-Model rerun on the rebased head is the check for the ListLogs internal error you saw — that class matches the now-merged #1809.

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1659 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1659 (comment)

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1659 (comment)

[NumaryBot]

NumaryBot review complete: no remaining inline findings.

Resolved 1 stale NumaryBot review thread (0 fixed, 1 outdated).

Summary: #1659 (comment)

[gfyrag]

[P2][blocking] Require the documented gRPC code for INDEX_BUILDING
Location: tests/antithesis/workload/bin/cmds/model/singleton_driver_model/queries.go:658-665 (isIndexNotReady)
Evidence: isIndexNotReady matches only the ErrorInfo reason. HasErrorReason is independent of the status code, so an InvalidArgument, FailedPrecondition, or any other gRPC status carrying INDEX_BUILDING is classified as the expected not-ready response. The adjacent isIndexNotFound helper correctly requires both FailedPrecondition and INDEX_NOT_FOUND, while the comment here specifies Unavailable plus INDEX_BUILDING.
Impact: the model oracle accepts an API error-code regression instead of emitting an Antithesis finding, weakening the exact failure signal this PR adds.
Resolution: require status.Code(err) == codes.Unavailable together with the INDEX_BUILDING reason, and add a wrong-code/same-reason regression.

The three blockers from the immediately preceding review are fixed: status polling is locally served, readiness is bound to the sampled creation frontier, and learned inserted_at is compared. The later validator-routing and INDEX_NOT_FOUND fixes are also sound. This is one genuinely new finding.

DECISION: REQUEST CHANGES
Head reviewed: ab7aefa
Blocking findings: 1
Non-blocking findings: 0
Previous findings: 3 fixed / 0 still valid / 0 outdated
Validation considered: nested singleton model unit tests + current CI
Residual risk: MEDIUM

[Azorlogh]

Fixed in e5c98f1: isIndexNotReady now requires codes.Unavailable alongside the INDEX_BUILDING reason, matching isIndexNotFound's shape; TestIndexNotReadyRequiresDocumentedCode pins that the reason on a wrong code stays a finding.

[gfyrag]

Review checkpoint for tomorrow: please rebase this PR onto the current release/v3.0 (d095a8b085553f6045573d0317003b605d420193). The previous INDEX_BUILDING blocker is resolved at the current head: the matcher now requires codes.Unavailable and the wrong-code regression is present. CI is green and there is no known active technical finding, but the stale-base gate prevents a valid final approval until the rebase and review-loop rerun.

[Azorlogh]

Rebased onto release/v3.0 @ d095a8b (no overlap with EN-1945 — clean rebase, agent-check PASS). CI rerunning.

[shipfox-ai]

This PR extends the model-based Antithesis driver with validated coverage of the filtered, paginated read surface (ListAccounts, ListTransactions, ListLogs), index lifecycle/retype-window modeling, and a batch of test-harness infrastructure changes. All changed files are under tests/, the implementation is carefully built (candidate-base validation, incarnation guards, two-phase retype-window closure) and backed by new unit tests, and no correctness defect was found in the driver or the oracle changes. Two documented driver conventions are violated with concrete triage impact, and the authoritative test documentation is left stale; the PR description is also out of sync with the diff. Recommendation: approve with comments — please fix the two code items below before merge; the documentation and description updates can follow immediately.

Standards

1. Duplicate assertion names collapse two coverage signals into one (hard). tests/antithesis/workload/bin/cmds/model/singleton_driver_model/indexes.go:783 and indexes.go:1146 both emit assert.Reachable("singleton_driver_model: kind-mismatched field query rejected", …) — one on the indexed transaction-query path, one on the indexed account-query path. tests/antithesis/README.md requires every Always/Sometimes/Reachable/Unreachable name to be globally unique because Antithesis indexes assertions by name and merges sites that share one. As written, the account and transaction kind-mismatch coverage report a single indistinguishable signal. Give each site a target-specific name (e.g. … field query rejected on transactions / … on accounts), matching the neighboring indexed transaction/account query … names.

2. Recheck diagnostics silently swallow RPC/stream errors (moderate). In queries_logs.go, recheckLogIDs discards the ListLogs error at queries_logs.go:363 and the drainStream error at queries_logs.go:368; recheckLogKinds does the same at queries_logs.go:389 and queries_logs.go:394. The README's error-classification section sets a minimum bar of if err != nil && !IsTransient(err) { assert.Unreachable(...) } for stream errors, and AGENTS.md forbids ignoring errors without an explicit, justified discard. Impact: in the singleton_driver_model: log query outside model finding, an empty recheck/serverKinds detail is indistinguishable from "the model's expected logs never appeared" — a failed diagnostic RPC corrupts the triage of a real finding. Classify the errors (tolerate transient/shutdown, surface anything else) or annotate the discard.

3. Authoritative test documentation left stale (moderate). The diff adds ListAccounts/ListTransactions/ListLogs query validation (queries.go, queries_logs.go) and index-lifecycle/retype-window machinery (indexes.go), but docs/technical/contributing/testing.md is untouched: its model-checker section still says only "account, whole-ledger, transaction-by-id, and declared-schema reads are all validated against the model", its file-role table lists only actions.go/reads.go for reads, and its "New persisted projection or read surface → add the read in reads.go" guidance now points to the wrong place. AGENTS.md makes documentation part of the change for new technical mechanisms. Update the coverage summary, the file-role table, and the read-surface guidance.

Rejected as not material: the panic("model: unmatched LOGS condition") in matchLogFilter (queries_logs.go:179) is a generator/matcher drift guard over driver-internal state, conformant with the documented oracle "fail loud on unmodeled input" pattern, not a finding-signaling path; the scattering of Checker methods across topical files follows the pre-existing package-wide layout (the base already defines Checker methods in validate.go, processor.go, debug.go, restore.go, …); the duplicated cursor/window and min/max helpers are baseline smells without correctness impact.

Spec

1. PR description contradicts the implementation on indexed-key retypes (moderate). The description states a deliberate scope cut: "retypes of INDEXED metadata keys are not generated — the rewrite's serving window needs a server-side design decision first (tracked as EN-1724)." The diff does the opposite: actions.go:929-949 retypes indexed keys like any other, and roughly a thousand lines model, observe, and close the resulting serving windows (indexes.go:663-771, checker.go, oracle RetypeWindow), with comments citing EN-1724 as resolved (the server-side support is present throughout internal/, e.g. internal/storage/readstore/store.go:406). The scope-cut note therefore reads as stale, but the description — the authoritative requirement — currently contradicts the code. Either update the description to include indexed-retype coverage and its acceptance criteria, or remove the generation/modeling.

2. Significant delivered scope is not covered by the stated task (minor). The stated task is extending the driver's validated query coverage, but the diff also reworks the Antithesis launch interface (bearer ANTITHESIS_API_KEY/ANTITHESIS_TENANT auth, k8s-launch-minutes, check-run, a mandatory 72-hour check-k8s-image-freshness gate in tests/antithesis/Justfile), adds env -u SOURCE_DATE_EPOCH to four Justfile build recipes, switches all local model-test server builds to -tags invariants and plumbs MODEL_DUMP_BATCHES (run_model_test.sh), and globally reclassifies WRITES_BLOCKED_DISK_FULL as transient for every workload (workload/internal/client.go:358-365,489-494). Each change is individually reasonable and well-commented, but none follows from the query-coverage requirement, and the disk-full reclassification and freshness gate alter retry/launch behavior fleet-wide. Split them into separately specified changes, or add them explicitly to the PR description with rationale.

Reviewed independently by GLM (glm-5.3-flash) and Codex (gpt-5.6-sol) via Shipfox; verified and synthesized by GLM.

[Azorlogh]

Addressed the Shipfox review at 0563c47:

1. Duplicate assertion names — split per target: kind-mismatched field query rejected on transactions / ... on accounts.
2. Swallowed recheck errors — recheckLogIDs/recheckLogKinds now return their RPC/stream errors and the finding details render recheck failed: <err> instead of a silently empty list, so a failed diagnostic can't corrupt triage of a real finding.
3. Stale testing.md — the model-checker section now covers the filtered/paginated list surface and the index lifecycle, the file-role table lists queries.go/queries_logs.go/indexes.go/metadata_filters.go, and the read-surface guidance points at the right pattern per read shape.
4. [Spec] stale scope-cut paragraph — replaced: indexed-key retypes ARE generated and their serving windows modeled (EN-1724's per-version bindings made the window well-defined); the description now says so.
5. [Spec] undeclared scope — the description now carries a "Harness / infrastructure scope" section declaring the launch-interface auth + image-freshness gate, SOURCE_DATE_EPOCH build change, invariants-tagged local model builds + MODEL_DUMP_BATCHES, and the fleet-wide WRITES_BLOCKED_DISK_FULL transient reclassification, each with rationale.

[NumaryBot]

NumaryBot posted 1 new inline finding.

Summary: #1659 (comment)

[NumaryBot]

NumaryBot review complete: no remaining inline findings.

Resolved 1 stale NumaryBot review thread (1 fixed, 0 outdated).

Summary: #1659 (comment)