[MrAlders0n]

Pace the one-off backfills so they cannot starve a small server's disk

Why

On the dev box (8 vCPU, 3.8 GB RAM, a rotational QEMU disk) the first packet_sites fill drove I/O pressure to full avg10=78%. The whole VM stalled twice: sshd, journald and the app all froze, and the box had to be rebooted.

Until now the backfills paused a fixed 100 ms between slices, whatever each slice cost. A packet_sites slice of 500,000 observation ids takes about 14 s of solid I/O on a laptop SSD, and far longer on a slow disk, so 100 ms of rest is nothing.

What changes

Every one-off fill now paces itself by duty cycle: the packet_sites backfill, the channel_message_iatas (cmi) backfill and the observation payload-type backfill. After each slice or batch it sleeps

max(min_pause, elapsed × (1 − duty) / duty)

where elapsed is how long that slice took. No single pause is longer than 10 minutes, and the arithmetic cannot overflow. With duty: 0.5 the fill is busy at most half the time. A slow disk makes each slice slower, so the pauses stretch on their own. A stop request still ends the pause at once.

The new settings sit under readmodels.fill:

Key	Default	Range
duty	0.5	0.01 to 1
min_pause	100ms	0s to 1m
packet_sites_slice	500000	1 to 10,000,000
cmi_batch	10000	1 to 1,000,000

• A value out of range stops startup with an error that names the key.
• The values in force are logged in the existing config: loaded line (fillDuty, fillMinPause, fillPacketSitesSlice, fillCmiBatch).
• The settings are documented in config.yaml.example.

Progress logging. Each backfill logs at most once a minute with ids_done, ids_remaining and an eta computed from the rate so far. For packet_sites and cmi this replaces the old line written after every 30 s step. The payload-type line gains the same three fields. The ETA comes from the rate the fill's position moves, applied to the current remaining count, so it holds while ingest adds ids. Progress resets when a fill starts over or completes.

Nothing else changes. Pacing changes timing only. The fills run the same statements over the same ranges, the trust and verification rules are untouched, and progress is still saved after every slice. Every existing read-model test passes without changes.

Reconcile and audit are left as they were.

• The reconcile runs every 5 minutes and covers only the ids since the run before last.
• The audit refills one 500,000-id slice an hour.
• Both are unpaced, but in steady state they are light.
• The one heavy case is the first reconcile after a long outage, which can cover many slices back to back. It is rare, and pacing it would delay trust.

A second cause: generic plans on a new, unanalyzed table

The stall. While measuring, the 6th 500,000-id packet_sites slice ran past a 2-minute statement timeout. The five slices before it took 4 to 7 s each.

The cause. I reproduced it with autovacuum off on packet_sites, which stands in for a fill that outruns autoanalyze.

• With automatic plan caching, the prepared fill statement switches to a generic plan after five custom executions.
• That plan is built while the new table has no statistics, and it assumes a few thousand rows per slice.
• On the same data, with custom plans forced on the session, all six slices finish in 10 to 14 s.

The fix (second commit):

• Every read-model fill and reconcile session (packet_sites and channel_message_iatas) sets plan_cache_mode = force_custom_plan for itself and resets it on close.
• The payload-type backfill statement runs with SET LOCAL plan_cache_mode = force_custom_plan.
• ANALYZE runs only on a table that has never been analyzed (reltuples = -1). It is checked at session start, where it runs only if the table already holds rows, and again after the session's first slice or batch.
• A table that already has statistics is never analyzed again. A restart or rebuild partway through a fill does not cause another ANALYZE burst on a large table.
• ANALYZE runs with no statement timeout and a 10 s lock timeout. A failure is logged and the fill goes on.

How it relates to the dev-box freezes. This is the likely second cause of the stalls, alongside the slow disk:

• A stuck slice is one long, unbroken stretch of I/O. On a rotational disk, that would pin I/O pressure the way the dev box showed.
• No pause setting can break that stretch up.

There is one caveat. The maint pool already sends plan_cache_mode=force_custom_plan as a startup parameter.

• A server that connects straight to Postgres, as the standard compose stack does, would already have planned these fills with custom plans.
• The generic plan applies when a pooler drops that parameter (PgBouncer's ignore_startup_parameters), or when the store runs on another pool.
• I could not check which way the dev box connects (no SSH).

The session-level setting makes custom plans hold either way.

Tests for this fix. TestFillSessionsPlanCustomPostgres uses a pool that forces generic plans, standing in for the old path.

• On that pool's own connection, the fill's plan is generic (it shows $1).
• A fill session on the same pool shows plan_cache_mode = force_custom_plan, and its plan has the slice bounds as constants.
• The new tables have statistics after the session starts and after the first slice or batch.
• The table ends exact, and the pool's own setting is back after Close.
• On the previous commit this test fails: fill session plan_cache_mode force_generic_plan.

TestFillAnalyzeOnlyNeverAnalyzedPostgres covers when ANALYZE runs.

• A never-analyzed table that already holds rows is analyzed once, at session start, and not again after the first slice.
• A table that already has statistics gets no ANALYZE across two rebuild sessions.

The payload-type step checks that current_setting('plan_cache_mode') is force_custom_plan inside the backfill transaction, that every untyped row was typed, and that the pool's own setting is back afterwards.

TestFillAnalyzeFailureIsIgnoredPostgres holds a lock that blocks ANALYZE. The ANALYZE was tried and failed (reltuples still -1), and the backfill still completed with an exact table.

Measurements

Measured on a scratch Postgres on a laptop SSD with 3,000,000 synthetic observations and the packet_sites backfill. "Busy" is the time spent in fill statements. These runs had custom plans forced on the session by hand, which the second commit now does itself.

Slice	duty	Slices	Fill time	Busy	Longest slice
500,000	1	6	77 s	99 %	14.2 s
500,000	0.5	6	153 s	50 %	15.3 s
500,000	0.25	6	316 s	25 %	14.7 s
100,000	1	30	85 s	96 %	3.2 s
100,000	0.5	30	169 s	50 %	3.6 s

Re-measured with the fix in the stall's own conditions: autovacuum off on packet_sites, a fresh reltuples = -1 table, a pool with automatic plan caching, and no manual setting.

Slice	duty	Slices	Fill time	Busy	Longest slice
500,000	1	6 of 6	73 s	99 %	13.7 s
500,000	0.5	6 of 6	148 s	50 %	13.7 s

Before the fix, the 6th slice ran past the 2-minute timeout in the same conditions.

• The duty setting holds exactly. The fill takes 1/duty times as long.
• Slices of 100,000 ids cost about 7 % more fill work but cut the longest burst from about 14 s to about 3 s.
• The slice default stays at 500,000. Lowering it would change what existing tests assert about slice boundaries. 100,000 is the documented choice for slow disks.

Operator note

On a slow disk (rotational, network or shared storage), set duty: 0.25. Consider packet_sites_slice: 100000 too, for shorter bursts. While a fill runs, watch full avg10 in /proc/pressure/io, and lower duty further if it climbs toward 50 %.

readmodels:
  fill:
    duty: 0.25
    packet_sites_slice: 100000

Tests

• Unit:
  • the pause formula: duty 1 gives the floor only, 0.5 gives elapsed, 0.25 gives 3× elapsed;
  • a context cancel during a pause;
  • the progress ETA;
  • config defaults, validation and the boot line;
  • payload-type batches: the same ranges at every duty, waits that scale with duty, and a cancel mid-pause that keeps the saved position.
• Integration:
  • TestPacedBackfillSlicesPostgres checks that the slice, batch and floor come from the store's pacing and that the table ends exact.
  • TestPacedFillsEarnTrustPostgres boots both read models over data with no read-model rows, using duty: 0.25 and tiny slices. Both fills complete, every page matches the observations, and trust is earned.
  • Both tests are added to the ci.yml -run lists.