Statement-timeout 503s, read-only scope restore, read-model restore logs

docs/api-contract.md

@@ -60,6 +60,8 @@ Retry-After: 5
 
 The body is the usual error envelope. Wait at least `Retry-After` seconds before retrying; a `deadline_exceeded` request will likely run as long again, so narrow it (a shorter window, a region) rather than repeat it. Clients are keyed by IP after the trusted-proxy step above.
 
+Separately from the guards, and on every server, a page whose database statement runs past its pool's statement limit (35 s; 120 s for the slow region and message statements) answers `503` with code `deadline_exceeded` and `Retry-After: 10`. This covers route evidence (`/routes/{iata}/{pathKey}/observations`) and the [background-data pages](#pages-that-wait-for-server-data) (region packets, region and scope messages, observer adverts). Other endpoints still answer `500` in that case.
+
 Each request falls in one family and one class:
 
 | Path | Condition | Family | Class |
@@ -582,7 +584,7 @@ Known routes are fully resolved multi-hop paths distilled from packet history.
 | `GET /routes` | Plain array, newest `lastSeen` first. Params: `iata`, `hopCount`, `cursor` (last item's `lastSeen`, epoch ms), `cursorId` (last item's `id`), `limit` (default 50). |
 | `GET /routes/search?from=&to=[&iatas=]` | Routes within one IATA between two node hash prefixes. `from`, `to` required (hex, 1–4 bytes; widths may differ). `iatas` optional: comma list, at most 100, omitted = every IATA; the single `iata` is still accepted. At most 500 results, shortest first. `hops` is trimmed to source…destination and `hopCount` = `hops.length`. |
 | `GET /routes/cross?fromHash=&toHash=[&iatas=]` | Routes that cross from one IATA into another. `fromHash`, `toHash` required. `iatas` optional: 2–100 codes, omitted = every IATA. The legacy `fromIata`+`toIata` pair is still accepted, but not together with `iatas`. At most 500 results, sorted by `totalHops`, then `crossHop.lastSeen` newest first. |
-| `GET /routes/{iata}/{pathKey}/observations` | Retained reports that match a saved route exactly. `pathKey` comes from a route response. Window: `range` (default `24h`, max `720h`) or `since`+`until` (max 30 days); `pageCursor` for the next page; `limit` default 50. |
+| `GET /routes/{iata}/{pathKey}/observations` | Retained reports that match a saved route exactly. `pathKey` comes from a route response. Window: `range` (default `24h`, max `720h`) or `since`+`until` (max 30 days); `pageCursor` for the next page; `limit` default 50. A lookup past the statement limit answers `503` + `Retry-After` ([Cost guards](#cost-guards)). |
 
 To page `/routes`, send the last item's `lastSeen` as `cursor` and its `id` as `cursorId`. Routes share a millisecond often (one batch of upserts), and `cursorId` returns the ones that didn't fit on the previous page. `cursor` alone still works but can skip those ties. `cursorId` must be a positive integer and requires `cursor`; otherwise `400`.
 

docs/configuration.md

@@ -217,6 +217,10 @@ MeshMapper imports. Every database session it opens is read-only
 `POST /api/v1/admin/accounts`, fails with SQLSTATE `25006` instead of changing data. The server
 logs a warning at startup saying so. The one thing it still runs is a read-only check of each
 read model (channel messages and packets by site) every 5 minutes; see [Read models](#read-models).
+When `meshmapper.scopes.enabled` is on, it also loads the writer's saved MeshMapper scope
+catalogues once at boot, reading only, so region and IATA scope lists (`/scopes?region=`,
+`/stats/scopes?region=`) match the writer's as of that boot. They don't refresh until the next
+restart.
 
 The REST API and `/ws` still serve what is in the database, but nothing new arrives. Because no
 migrations run, the database must already be at the schema the build expects. Its use is

docs/operations.md

@@ -619,7 +619,8 @@ Trust is held in memory by each server process; nothing in the database marks th
 exact. A process earns it at every start, once the boot reconcile has run, the backfill is
 complete and the one-time verification has passed. It drops trust:
 
-- after 3 reconciles in a row that each repaired at least one row. Two clean reconciles in a
+- after 3 reconciles in a row that each repaired at least one row. It logs
+  `read model untrusted: repair runs fixed rows the writers missed`. Two clean reconciles in a
   row restore it;
 - at once when ingest leaves a message without its sites: a failed site insert, a message
   stored without its sites, or any message insert error. It logs
@@ -628,6 +629,8 @@ complete and the one-time verification has passed. It drops trust:
   one interval;
 - when the stored state is no longer complete and verified, for example after a rebuild.
 
+Each restore after a drop logs `read model trusted again: its pages are served from it`.
+
 A reconcile that repairs rows logs `cmi reconcile repaired N pair(s): a writer bypassed the read model`.
 Something stored messages without recording their sites. On a single server running this
 release that should not happen. Running two server versions that both write to one database at
@@ -791,7 +794,8 @@ complete and the one-time verification has passed. It drops trust:
 - at once when a reconcile or the hourly audit repairs any row: something bypassed the
   writer. Two clean reconciles in a row restore it. A repair logs
   `packet_sites reconcile repaired N row(s): a writer bypassed the read model` or
-  `packet_sites audit repaired N row(s) below the watermark`;
+  `packet_sites audit repaired N row(s) below the watermark`, and the drop logs
+  `read model untrusted: repair runs fixed rows the writers missed`;
 - at once when an observation insert with its site row fails. Ingest never loses the
   observation: it logs `observation site row failed; storing the observation without it` and
   stores it without its row. Trust drops with
@@ -801,6 +805,8 @@ complete and the one-time verification has passed. It drops trust:
 - after 3 failed reconciles in a row, until one succeeds;
 - when the saved state is no longer complete and verified, for example after a rebuild.
 
+Each restore after a drop logs `read model trusted again: its pages are served from it`.
+
 On a single server running this release, repairs should not happen. Running two server versions
 that both write to one database at the same time is not supported; if it happens, the rows the
 older one leaves out are repaired within one reconcile interval, and trust returns two clean
@@ -1318,7 +1324,7 @@ scrape_configs:
 | `beacon_packet_sites_front_live` | gauge | `index` | Index entries written in the last two minutes that the last front sweep visited |
 | `beacon_packet_sites_bytes` | gauge | | Total size of `packet_sites`, heap and indexes, hourly |
 | `beacon_db_sequence_used_ratio` | gauge | `sequence`, `cycle` | Share of a sequence's usable values already handed out, 0 to 1, each minute. The usable range ends at the sequence's own limit or its column's type limit (`integer` 2,147,483,647), whichever is lower. `cycle="true"` marks a `CYCLE` sequence, whose ratio is its position, not used-up headroom. See [Sequence headroom](#sequence-headroom). |
-| `beacon_api_unavailable_total` | counter | `reason`, `model` | `503`s a page answered: `not_ready` while its read model or backfill is not ready, `slow_pool_busy` when `api_slow` had no free connection. `model` is `packet_sites`, `cmi` or `observation_payload_types`. |
+| `beacon_api_unavailable_total` | counter | `reason`, `model` | `503`s a page answered: `not_ready` while its read model or backfill is not ready, `slow_pool_busy` when `api_slow` had no free connection, `timeout` when its statement ran past its pool's statement limit or an enforced guard's deadline. `model` is `packet_sites`, `cmi` or `observation_payload_types`. |
 
 `guard` is `deadline`, `cap`, `admission` or `budget`. The Postgres counters are read once a
 minute; the index gauges first appear an hour after startup. The two `beacon_planguard_*`