fix(verification): stop background verification scanning contiguous_data_ids

[vilenarios]

Problem

Background data verification did two full scans of data.db's contiguous_data_ids, one for each batch. Measured on a production gateway with core up 13.2 h and a 72 GB data.db holding 99M rows:

Statement	Calls	Time per call	Why
updateDataItemVerificationStatus (write)	38	~624 s, with ~0 s queue wait	WHERE id = @id OR root_transaction_id = @id with no index on root_transaction_id → SCAN contiguous_data_ids
selectVerifiableContiguousDataIds (read)	79	~229 s	COALESCE(verification_priority, 0) >= @min blocks the range seek, so the in-order walk covers every unverified row whenever fewer than 1000 qualify

data.db has a single writer, so the update scans occupied it for 6.6 of the 13.2 hours. Other data.db writes queued behind them: confirmChunkPlacements waited ~166 s on average and saveDataContentAttributes ~3 s. Their duration was almost entirely queue wait.

Root cause

• The update. Until 2025-07, the query got item IDs from bundles.bundle_data_items, whose root_transaction_id is indexed. b08111a (PE-8335) moved the lookup onto a new contiguous_data_ids.root_transaction_id column so that data.db could be distributed without bundles.db. The migration added the column but no index, so the lookup quietly became a full scan.
• The select. 27431e2 changed a debugging leftover, COALESCE(…, 80), to COALESCE(…, 0) so that NULL means "lowest priority", not "excluded". That is correct, but wrapping the column prevents a range seek. Once only a few rows qualify (53 on this gateway), LIMIT 1000 never fills and the walk covers the whole partial index.

Fix

• Migration: a partial index contiguous_data_ids_root_transaction_id_idx ON (root_transaction_id) WHERE root_transaction_id IS NOT NULL. SQLite plans the OR as a MULTI-INDEX OR: a primary-key probe plus an index probe. The index keeps data.db independent of bundles.db, which was the point of PE-8335.
• Select: when the minimum is positive, which is the default (60), a NULL priority can never qualify, so getVerifiableDataIds uses a bare verification_priority >= @min, which seeks. When the minimum is 0 or below, it keeps the existing COALESCE statement. That preserves the contract that such a minimum verifies all unverified data, the behaviour from before priorities existed, and there LIMIT fills immediately so the walk is cheap.
• getVerifiableDataIds takes the minimum and the retry cap as parameters that default to config, which resolves the method's own TODO.

Measured on a copy of that production data.db

The copy was made with the SQLite backup API and run with the same SQLite (3.45.3) as the core image:

Step	Before	After	Rows
select, min 60	229.46 s (scan)	0.02 s (seek)	53 → 53
select, min 0 (kept statement)	—	0.01 s	1000
mark root verified	587.27 s (scan)	< 0.01 s (multi-index OR)	49 → 49
migration CREATE INDEX	—	692 s, +3.78 GiB	—

Upgrade cost: migrations run in docker-entrypoint.sh before the gateway starts, so a gateway this size starts about 11.5 minutes later on the first boot after upgrading. The 2025-05-28 migration built an index on this same table at startup too. The build costs about the same as one of the scans verification was running every 20 minutes. Rows with a root transaction are 97% of the table, so making the index partial saves little space, but it keeps L1 rows out.

Tests

• Query-plan assertions, following the existing selectStableTransactionOffsetById test: the select must SEARCH … (verification_priority>?), and the update must probe contiguous_data_ids_root_transaction_id_idx and never SCAN contiguous_data_ids. Both fail on develop.
• Selection behaviour: priority order, the retry cap, NULL excluded at a positive minimum, and NULL counted as 0 at a minimum of 0.
• The skipped saveVerificationStatus test is replaced. It only passed when run on its own ("works when running the test individually") because the main-thread saveDataContentAttributes dedupe (7-minute TTL) skipped an ID an earlier test had already written, after afterEach emptied the table. It now uses fresh IDs and checks the current root_transaction_id semantics.
• The down migration and a re-run of the up migration were checked on a fresh DB.
• test/data-schema.sql was regenerated with ./test/dump-test-schemas. Only the new index is committed; the dump also picked up unrelated drift in the core and bundles schemas, which I left out.
• yarn test: 3407 pass, 1 fail. The failure is parquet-exporter.test.ts, whose duckdb binding can't load on the local glibc; it is not related to this change.
• Lint is clean. yarn typecheck is at the develop baseline (430 errors).

Docs

• A CHANGELOG entry under Fixed, including the upgrade note.
• MIN_DATA_VERIFICATION_PRIORITY and MAX_VERIFICATION_RETRIES were in docker-compose.yaml but missing from docs/envs.md; both are now documented.

This came out of evaluating #448 (partitioned verification). Most of the cost that #448 would have saved was these scans.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AF5ZbmQGHvRjPbYPcxo3iQ

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: b89de65a-7789-4ea0-8a69-8c159e18f780

📥 Commits

Reviewing files that changed from the base of the PR and between 2ead704 and 13c4d46.

📒 Files selected for processing (8)

• CHANGELOG.md
• docs/envs.md
• migrations/2026.09.29T15.00.00.data.add-root-transaction-id-index.sql
• migrations/down/2026.09.29T15.00.00.data.add-root-transaction-id-index.sql
• src/database/sql/data/verification.sql
• src/database/standalone-sqlite.test.ts
• src/database/standalone-sqlite.ts
• test/data-schema.sql

Included review availability: This review used your included allowance. Your plan provides up to 2 included reviews per hour; 1 remain after this review.

---

📝 Walkthrough

Walkthrough

The background verification worker now accepts priority and retry limits, and selects data using a query suited to the configured minimum priority. A partial index supports root-transaction lookups. Tests cover selection behavior, status updates, and index use.

Changes

Background data verification

Layer / File(s)	Summary
Priority-based verification selection src/database/sql/data/verification.sql, src/database/standalone-sqlite.ts, src/database/standalone-sqlite.test.ts, docs/envs.md	The worker accepts configurable priority and retry limits. Positive minimum priorities exclude rows with null priority; minimums of zero or lower treat null priority as zero. Tests cover selection criteria, ordering, and query-plan use.
Root-transaction index and status updates migrations/2026.09.29T15.00.00.data.add-root-transaction-id-index.sql, migrations/down/2026.09.29T15.00.00.data.add-root-transaction-id-index.sql, test/data-schema.sql, src/database/sql/data/verification.sql, src/database/standalone-sqlite.test.ts, CHANGELOG.md	A partial index covers non-null root transaction IDs, with a rollback migration. Tests check status updates and index use. The changelog records the index and query changes.

Priority: ⬆️ High

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 13c4d

This change speeds up background verification queries and adds an index. The migration may take several minutes on very large databases, as the changelog notes, but no merge-blocking risk was identified.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 13c4d

The verification changes preserve the existing access path and status-update scope, with tests covering selection and index use. A failed migration could leave the gateway running without the intended performance fix, but that startup behavior predates this PR. No introduced security finding was established.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The established exposure is the gateway's data.db verification and write path. The supplied evidence does not establish tenant-wide reach or an external privilege change.

Trust Boundaries and Controls

• observed — Production verification selection continues through the argumentless database facade and worker dispatch. The optional arguments are not forwarded across that boundary by these callers.

Resilience and Maintainability Implications

• inferred — Migration failure is not a fail-closed startup condition in the existing entrypoint. For this PR, that can leave the intended write-contention reduction unapplied, although it does not worsen the pre-PR query behavior.

Hardening Proposals

• proposed — Make gateway startup stop on migration failure, and verify the index before relying on the write-performance improvement in a rollout.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: preventing background verification scans of contiguous_data_ids.
Description check	✅ Passed	The description directly explains the scan problem, database and query fixes, tests, performance results, upgrade cost, and documentation changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.58%. Comparing base (2ead704) to head (13c4d46).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #957      +/-   ##
===========================================
+ Coverage    81.57%   81.58%   +0.01%     
===========================================
  Files          227      227              
  Lines        87981    87998      +17     
  Branches      7700     7699       -1     
===========================================
+ Hits         71768    71793      +25     
+ Misses       16088    16080       -8     
  Partials       125      125              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.