test(scale): complete the Fly S18-S26 lifecycle ladder

[DecisionNerd]

Problem

GraphForge still lacks one reproducible provider-native Graph500 ladder that proves the ordinary product lifecycle from lower qualification scales through the final billion-edge result. Developer laptops are disk-constrained control planes, not systems under test. The ladder must expose architectural amplification before an expensive rung and must not compensate by increasing RAM or timeout.

Objective

Execute a sequential disposable Fly qualification and certification ladder:

S18 -> S19 -> S20 -> S22 -> S24 -> S25 -> S26

Every rung runs the identical ordinary public lifecycle under BenchExec authority. Each completed rung produces sanitized evidence; the next rung runs only when the controller admits it from required prior evidence and the explicitly authorized maximum scale permits it.

Requirements

• Freeze the exact merged commit, immutable OCI digest, generator identity, EF16, seed, profiles, region, volume, Machine class, tool versions, and maximum authorized scale.
• Use the completed public runner/controller (refactor(bench): extract a public-API scale certification runner #955/test(bench): define the complete S18-S26 qualification ladder #956), BenchExec authority (test(bench): enforce resource limits and normalize evidence with BenchExec #957), disposable Fly adapter (build(bench): provide the disposable Fly ladder adapter #958), and ordinary staged-construction path (fix(api): route staged Parquet import through construction sessions #991).
• Build remotely and keep datasets, projects, portable exports/imports, OCI layers, and large artifacts off the Mac.
• Generate each scale's deterministic dataset exactly once per ladder attempt and reuse those exact bytes for that rung's ingest, recovery, verification, and any authorized same-commit retry. Do not derive smaller canonical rungs by truncating/remapping a larger-scale dataset.
• After a rung's evidence is independently accepted, delete that rung's datasets/projects/exports/imports from the Fly volume before generating the next rung, while retaining only sanitized evidence and immutable identities.
• Run every rung through generate, ingest, reopen/recount, canonical one-hop/two-hop ordered-LIMIT queries, portable-v2 export, full verify, clean import, imported reopen/recount, and matching post-import queries.
• Stop at the first typed failure. Never skip a rung, rerun an unchanged failing configuration, or emit a pass for an interrupted rung.
• Reconcile raw attempts, rejects, duplicates, self-loops, live rows, source/imported counts, construction chunks, application bytes/calls, artifacts, fsyncs, reader calls, and publication work.
• Record BenchExec process-tree wall/CPU/RSS/physical I/O and GraphForge logical/storage/construction evidence for every phase.
• Require bounded or plateauing phase RSS. Continued material RSS growth with edge count is an architectural failure signal.
• S20 requires conservative adjacent S18/S19 projection within four hours, 4 GiB RSS, and the 500 GB Fly volume with reasonable headroom.
• S26 requires completed adjacent S24/S25 projection. Projected transient peak must be <=425 GB before launch. Select the smallest measured Machine with reasonable RSS/CPU headroom; 128 GiB is only the M5 ceiling, never the default.
• Treat the four-hour limit as a per-rung runaway/certification envelope, not a universal SLA.
• Preserve prior CURRENT on cancellation, corruption, resource failure, or interrupted publication; re-entry must not duplicate rows or work.
• Validate closed sanitized evidence independently before authorizing the next rung.
• Tear down and independently verify removal of every Machine, volume, app, image attachment, secret, and temporary token after the terminal rung.

Acceptance criteria

• S18 and S19 complete on Fly with the full unchanged lifecycle and linear/bounded evidence.
• Evidence proves one generation per rung, exact dataset identity reuse within the rung, and post-acceptance reclamation before the next scale.
• The S18/S19 projection admits S20 with documented runtime, RSS, storage, I/O, reader-call, and publication headroom.
• S20 completes the full source -> export -> verify -> clean import -> reopen lifecycle within four hours and 4 GiB with matching correctness evidence.
• S22, S24, and S25 run only after their preceding gates pass and preserve bounded/plateauing RSS plus reasonable storage/I/O slopes.
• The S24/S25 projection admits S26 below the 425 GB transient storage threshold and with measured machine headroom.
• S26 completes with >=1,000,000,000 live persisted edges and matching source/imported counts, queries, fingerprints, and portable-v2 verification.
• Checked-in evidence contains only versioned sanitized JSON/journals and exact immutable identities; no datasets, graph/package content, credentials, UUID inventories, provider IDs, or absolute host paths.
• Complete Fly teardown is proven by independent inventory and resource ledger.
• Any implementation/evidence/docs PR has exact-head green CI and CI Gate.

Non-goals

Laptop scale runs, benchmark-only engine paths, weakening queries/durability/verification, increasing RAM/timeouts to conceal amplification, treating bounded memory alone as success, skipping directly to S20/S26, or claiming a failed/interrupted rung.

Relationships

• Parent/final certification: test(scale): certify the Fly S26 billion-edge round trip #745
• Harness epic: epic(bench): externalize scale certification and add GDC benchmark suites #952
• Provider adapter: build(bench): provide the disposable Fly ladder adapter #958
• Native ingest: feat(storage): stage topology append-only with linear ingest I/O #901/fix(api): route staged Parquet import through construction sessions #991/test(scale): attribute physical bytes per node and edge before SCALE26 #951
• Native queries: fix(exec): bound reopen and fixed-hop LIMIT query RSS at scale #904

[DecisionNerd]

Root-cause update from the bounded local reproduction on commit 6d93bc7:

• S16, one edge publication: about 668 MB observed RSS.
• S17, two fixed one-million-row publication windows: 963,362,816-byte process maximum RSS before repair.
• S18, four fixed windows: 1,369,128,960-byte process maximum RSS before repair.
• The second S17 publication decoded and recopied all 1,048,576 previously persisted edges. It materialized 3,043,346 cumulative topology output rows to retain 1,863,698 edge rows plus 131,072 nodes.
• S18 materialized 10,358,282 cumulative topology output rows for about 3.80 million retained edges and reached about 1.07 GB disk.

Confirmed owners:

1. GraphWriter open decoded complete node and edge topology to recover surrogate maxima.
2. Fixed-schema append decoded the complete accumulated Parquet file, concatenated it with the new request, and materialized one complete replacement RecordBatch.
3. Bulk normalization, endpoint sets/maps, writer rows, adjacency deltas, mutation subjects, and Arrow receipts stack several request-sized allocations.
4. Each internal bulk window publishes a progressively larger immutable project generation, causing superlinear retained disk.

The first repair now streams existing Parquet through bounded 64K-row batches and reads surrogate maxima from only the final bounded row group. On the same diagnostics:

• S17 peak fell from 963,362,816 to 675,184,640 bytes; the second window completed at about 584 MB rather than setting a new peak.
• S18 peak fell from 1,369,128,960 to 819,511,296 bytes; later windows completed at approximately 693 MB, 705 MB, and 561 MB.
• Complete node-table reads during writer open fell to zero.

This proves accumulated-edge RSS is now plateauing; remaining memory is bounded by the fixed request/node working set. It does not solve the independently confirmed quadratic I/O and generation-retention problem. Native sub-issue #901 now tracks the append-only/shard-staged linear-I/O repair and blocks this canonical gate. #900 must remain open until #901 and the same-class Fly S20/S22 evidence are complete.

[DecisionNerd]

Bounded-memory repair #902 merged through PR #903 as eccb6e06726d05cdef9e5242cad885be80565eee.

Exact-head gates passed, including Rust Quality, Bazel Bootstrap (//:ci_rust_tests authority), Python/Node bindings, macOS/Windows durability, Concurrency Matrix, Native Durability Aggregate, and CI Gate. CodeRabbit's final review found heartbeat lifecycle races; commit c4e34c3a fixed panic-safe shutdown, guaranteed the initial snapshot, and reset per-rung state before the worker starts. All review threads are resolved.

Controlled lower-rung evidence remains:

• before: S17 peak RSS 963,362,816 bytes; S18 1,369,128,960 bytes
• after: S17 peak RSS 675,184,640 bytes; S18 819,511,296 bytes, with later S18 windows about 693, 705, and 561 MB
• writer reopen full node/edge topology reads: zero

This demonstrates that accumulated topology is no longer materialized in RSS. It does not claim linear I/O: #901 remains open for append-only/shard-staged construction. Next gate is the same 4 GiB Fly S20 rerun at the immutable merged SHA; #900 remains open until that evidence passes.

[DecisionNerd]

Fly S20 qualification is running against merged commit eccb6e06726d05cdef9e5242cad885be80565eee on a private 2-performance-vCPU / 4 GiB machine with one encrypted volume and an exact pinned OCI platform digest.

Interim aggregate evidence at the midpoint of ingest:

• process high-water RSS: about 644 MB and plateaued across successive publish windows
• complete node/edge topology reads: zero
• 8,388,608 new rows caused 30,408,704 topology output rows to be rewritten
• retained/live graph accounting: about 4.27 GB, while physical volume use had already reached roughly 13-16 GB during publication
• direct process accounting showed more than 100 GB of userspace read traffic and more than 24 GB of write traffic during this same lower-scale construction

This confirms the merged #902 repair bounded memory on ordinary 4 GiB infrastructure. It also confirms that #901 is the remaining root blocker: cumulative immutable-generation recopy and generation retention amplify storage traffic and physical peak usage. This is not a Fly/RAM admission failure and does not justify a larger-memory machine or relaxed criteria.

The run remains in progress. Final pass/failure classification will require the complete source → export → verify → clean import → reopen sequence or a typed first failure; this interim comment is not certification.