[MrAlders0n]

Observer comparison: read each observer's receptions on their own

Problem

Since the API pool started making a fresh plan on every call (plan_cache_mode = force_custom_plan, no parallel workers), /stats/observer-comparison got slow on the dev box. 24 h windows took 6.6 to 13 s. 7 d windows always hit the handler's 15 s limit and returned 503. The SQL did not change.

Before the upgrade the server used automatic plan caching. After 5 calls on a connection it switched to a cached generic plan, and that plan read each observer's rows through idx_observations_observer. With the observer ids as literal values, the planner now does one of two things:

• 24 h: it walks every reception in the window through idx_observations_heard and filters out the other observers. On the report's page (busy and mid observer, 22 h) that read 351K buffers, where the old generic plan read 78K.
• 7 d: it hash-joins against every route 0/1 packet, scanning the whole of idx_packets_type_last_heard. On a small box whose visibility map is not current, that turns into a heap visit for every packet.

Dev data makes both choices likely: 189 observers, and the busiest ones hear about 4 % of all receptions.

Change

The statement now reads each observer's receptions in its own scan, keyed by (observer_id, heard_at). The two sets are joined with a full join on packet_hash. Because (packet_hash, observer_id) is unique, each side holds a packet at most once. The route type is then checked once per heard packet, where before it was checked once per reception.

The observer ids are passed as scalar subqueries. The planner then costs every observer the same way, so a busy observer's statistics can no longer push it toward the window walk or the full packet scan. In both plan modes, on every input tried, the plan is:

• two idx_observations_observer range scans;
• a hash full join;
• one packets_pkey probe per packet.

The result is unchanged: same counts and same known flags. The query does not use idx_observations_observer_id, which was dropped.

Results

Measured on a Mac, warm cache, best of 3. Settings were as the API pool runs them: JIT off, no parallel workers for the new build.

• Old build: the statement before this change, in its steady state (cached generic plan, server-default parallel workers).
• Current: the same statement under the current custom-plan pool.
• New: this change, in custom mode.

Buffers are shared hits.

Dev data (5.9M observations, window ending 2026-10-20 11:00). Busy pair: the two busiest observers, at the same site. Quiet pair: two observers with about 200 receptions a week each. Region: four sites, none of them the busy pair's. IATA: the busy pair's site.

Window	Filter	Pair	Old build ms / buffers	Current ms / buffers	New ms / buffers
24 h	all	busy	47.8 / 130,141	73.9 / 418,136	29.0 / 85,005
24 h	all	quiet	0.05 / 157	0.04 / 157	0.05 / 157
24 h	region	busy	6.5 / 22,701	39.7 / 310,696	5.1 / 22,701
24 h	region	quiet	0.06 / 157	0.05 / 157	0.06 / 157
24 h	IATA	busy	48.4 / 130,141	77.0 / 418,136	31.3 / 85,005
24 h	IATA	quiet	0.02 / 41	0.02 / 41	0.02 / 41
7 d	all	busy	337.6 / 875,192	190.0 / 80,044 *	211.4 / 487,061
7 d	all	quiet	0.44 / 1,985	0.46 / 1,951	0.47 / 1,951
7 d	region	busy	57.3 / 150,612	47.1 / 38,583 *	61.1 / 68,161
7 d	region	quiet	0.46 / 1,985	0.49 / 1,951	0.49 / 1,951
7 d	IATA	busy	341.9 / 875,192	281.4 / 763,163	227.0 / 487,061
7 d	IATA	quiet	0.08 / 377	0.11 / 343	0.10 / 343
22 h page from the timing report	all	busy and mid	25.1 / 77,623	60.7 / 351,358	21.9 / 77,591

* The current build's 7 d plans that show fewer buffers are the hash joins over every route 0/1 packet. They count each index page once, but they scan the whole packet index whatever the observers heard. That is the plan that times out on the dev box. With warm caches on the Mac, the current build is 10 % faster on "7 d all busy"; the new plan reads only the packets the two observers heard.

Local production copy (29M observations, 3,558 observers). Plans ran on t_end, the read-only overlay that db/planguard/overlay/end_state.sql builds with the migrated index set, and the window ends at 2026-10-18 17:00. The two busiest observers at one site make the busy pair. The quiet pair is two observers with about 100 receptions a week each.

Window	Filter	Pair	Old build ms / buffers	Current ms / buffers	New ms / buffers
24 h	all	busy	74.2 / 176,362	55.5 / 102,721	43.3 / 106,958
24 h	all	quiet	0.07 / 254	0.06 / 254	0.07 / 254
24 h	region	busy	76.8 / 176,362	68.1 / 172,125	46.8 / 106,958
24 h	IATA	busy	75.2 / 176,362	66.4 / 172,125	44.1 / 106,958
7 d	all	busy	238.3 / 545,900	314.6 / 114,543 + temp spill	141.4 / 331,188
7 d	all	quiet	0.23 / 1,030	0.24 / 1,030	0.24 / 1,030
7 d	region	busy	240.2 / 545,900	215.2 / 533,133	148.7 / 331,188
7 d	IATA	busy	244.9 / 545,900	213.8 / 533,133	153.0 / 331,188
7 d	all	busy and quiet	119.6 / 274,923	143.0 / 274,900	108.2 / 274,923

The plan corpus entries pick their own pairs through resolvers, and those pairs differ from the ones in this table. Their busy pair is the two busiest observers overall, which sit at different sites. Their budgets come from the local tier on the same t_end. On those pairs the new statement reads the same buffers as the current build, within 0.5 %. For example, 24 h busy reads 130,894 B under both builds, and 7 d busy reads 372,197 B against 370,407 B. On the production copy the current build's plans mostly stay sane; they go wrong on dev-sized data and on the 7 d busy spill shown above.

Generic-mode plans of the new statement read the same or fewer buffers than the old build's generic plans in every row.

Overall:

• On every shape tried, the new statement reads at most the old build's buffers.
• The new statement is as fast as the old build or faster, with two exceptions. Quiet pairs match within timer noise. Dev "7 d region busy" takes 61 ms against the old build's 57 ms, while reading under half the buffers.
• 7 d all-regions on dev data now takes about 0.2 s.

Tests

• TestObserverComparisonMatchesTodayPostgres (new, added to the CI list) runs both the new statement and the old text over a skewed fixture: 2,000 packets of every route type and nine observers. It covers:

  • pairs: busy, quiet, mixed, reversed, the same observer twice, unknown observers;
  • windows: wide, one day, edges exactly on receptions, a 1 µs window, an empty future window;
  • IATA filters: nil, empty, one site, duplicates, no match, the empty-region {""}, lower case.

  It compares 2,112 parameter sets in custom and generic plan modes, with and without enable_seqscan = off, with JIT off. All rows match. The fixture drops idx_observations_observer_id. The test also checks that, in both modes, every observation scan uses idx_observations_observer, one per observer. It also checks that packets is read only by one packets_pkey probe per heard packet, under a nested loop, and never scanned whole. The fixture adds 100,000 unheard packets, so a whole-table scan of packets costs what it does at full size. With the old text the observation check fails, because the old text uses a single scan for both observers.

• The plan corpus has eight new entries for the statement: busy, quiet, mixed and unknown pairs; 24 h and 7 d windows; all sites, one site, a seven-site region and an empty region.

  • Inputs: observers and windows come from bounded resolvers over the observers in the newest 200,000 observation ids. The quiet pair is the two observers that heard least in the entry's own 7 d window. Each candidate is counted on idx_observations_observer, capped at 1,000 rows. The unknown entry uses a fixed non-zero id, derived from a hash, that a resolver confirms matches no observer.
  • Plan checks: each entry requires idx_observations_observer with an observer_id = condition on observations, and packets_pkey with a packet_hash = COALESCE(...) condition on packets.
  • Budgets and baselines: budgets are 1.5x the buffers measured by the local tier on t_end. Fixture and local baselines are included, the local ones also from t_end.
  • Old statement: on the fixture it fails one entry (a full scan of idx_packets_route_type). On t_end it fails every entry on the packets condition.

• go build, go vet, gofmt, go test ./..., the race step, every Postgres step in ci.yml, the fixture tier (684 runs, every planted regression detected), the planguard tests, gen-policy (no change: the statement stays custom-plan) and the swagger diff all pass.