fix: shared advisory-lock session and seven open bugs (#2169)

.github/workflows/e2e-nightly.yml

@@ -48,6 +48,12 @@ jobs:
           go-version: "1.27.2"
           cache: true
 
+      # A digest the docs pin is checked by nothing else, and three that did
+      # not exist shipped in the deployment guide (#2224). It asks the
+      # registries, so it runs here rather than in verify.
+      - name: Resolve the image digests the docs pin
+        run: make docs-images-check
+
       - name: Start E2E environment
         run: make e2e-up
 

.semgrep/go-advisory-lock.yml

@@ -0,0 +1,31 @@
+rules:
+  # A try-lock the platform holds across work goes onto the process's one lock
+  # session, internal/advisorylock (#2169). Each loop used to take its lock on
+  # a pooled connection of its own and hold it for the whole pass while the
+  # pass drew its queries from the same pool; with four such loops on a pool of
+  # five, the passes waited on the pool for their own queries, never released
+  # their locks, and starved the replica. connoauth's blocking per-call lock is
+  # the one session lock taken elsewhere, since a lock that waits cannot share
+  # a session. Test code takes what locks it likes, to stand in for another
+  # replica.
+  - id: advisory-try-lock-outside-advisorylock
+    languages: [go]
+    severity: ERROR
+    paths:
+      include:
+        - pkg
+        - internal
+        - cmd
+      exclude:
+        - "*_test.go"
+        - /internal/advisorylock
+        - /bench
+    pattern-regex: pg_try_advisory_lock\(
+    message: >-
+      Take a session advisory lock through internal/advisorylock: advisorylock.Try(ctx, db,
+      advisorylock.ID(key)) and defer the release it returns, and run the locked work on the pool.
+      A try-lock taken on a connection of its own pins that connection for as long as it is held,
+      and enough of them starve the pool their holders need to finish (#2169).
+    metadata:
+      category: reliability
+      issue: "#2169"

CLAUDE.md

@@ -203,7 +203,7 @@ mcp-data-platform/
 │   ├── connid/                     # Connection identity: the instance a connection is stored under, the name a call binds it by, the toolkit serving it, and which half of the config owns it — one Resolver, distinct types
 │   ├── connview/                   # Builds the list_connections view (configured + discovered)
 │   ├── contenttype/                # Media-type detection and normalization for every content write path
-│   ├── database/                   # Database utilities (migrate/ = golang-migrate runner + 207 embedded SQL migrations)
+│   ├── database/                   # Database utilities (migrate/ = golang-migrate runner + 209 embedded SQL migrations)
 │   ├── embedding/                  # Text embedding generation for memory vector search
 │   ├── indexjobs/                  # Postgres-backed, source-kind-agnostic background indexer
 │   ├── knowledge/                  # Unified read path for platform knowledge (federation/ = live toolkit registry adapter)
@@ -250,6 +250,7 @@ mcp-data-platform/
 │   ├── apigwmetrics/               # The api gateway's outbound HTTP instrumentation as an http.RoundTripper: the connection/status labels, and the persona the call was authorized under, read off the request context the tool call carries (#1615). Holds no gateway types, and was extracted when pkg/toolkits/apigateway reached its package-size budget
 │   ├── outbound/                   # The one transport chain every HTTP client the platform builds is made from (#1895): the User-Agent, a client span named `{kind} {method}` that continues the caller's trace and carries the W3C traceparent to the upstream (off per connection with `trace_propagation: false`), `http_client_requests_total{kind,connection,status_class}` and its histogram, and `egress_blocked_total{reason}` with one WARN when the egress guard under it refuses. `NewClient(Options)` or `Transport(base, opts)`; the recorder is the platform's one Metrics, installed by internal/platform/obs through `SetDefaultMetrics`, and `Metrics()` is how a retry loop counts beside the requests it issues. `.semgrep/go-outbound-client.yml` refuses an `http.Client` built anywhere else
 │   ├── opsobs/                     # The recorder the platform's own operations report through (#1898): installed by internal/platform/obs like outbound's default, read by the authenticator chain (auth_attempts_total), the OIDC JWKS fetch, the HTTP gate's fail-open, browser sign-in and the domain operations (`Start(ctx, Op*)` opens a span under the call and counts domain_operations_total{operation,result}); also the label vocabulary those series use (AuthMethod*, AuthReason*, Denial*, Dependency*, Connection*, Op*, Sink*), kept off pkg/observability's exported surface
+│   ├── advisorylock/               # The one session per process every PostgreSQL try-lock held across work is taken on (#2169): the background loops (catalog scan, maps, storage and webhook alerts, retention, capacity, the call catalog and audit sweeps, thumbnail sweeps, catalog import, resource reclassification), connlimit slots and the connoauth refresher. A lock on a pooled connection of its own pinned that connection for the whole pass while the pass drew from the same pool, and four such loops starved a pool of five. `Try(ctx, db, ID(key)|Pair(a,b))` returns a release; a key the process holds is refused, since PostgreSQL would grant it again; the session goes back to the pool with its last lock. `.semgrep/go-advisory-lock.yml` refuses a try-lock taken anywhere else
 │   ├── bgloop/                     # The one way the platform runs work in the background (#1897): `Run` (a timer, a wake channel, a stop channel), `Consume` (a channel of events), `Unit` (one claimed unit of work) and `Listen` (a LISTEN connection's state). Every iteration is counted by result, timed and stamped on success (`background_loop_*{loop}`), gets a span whose context its body logs with, and a skip under another replica's lock is `ErrSkipped`; loop names are constants in names.go. `.semgrep/go-background-loop.yml` refuses a ticker or an infinite loop around a select anywhere else
 │   ├── objectobs/                  # Every object operation on a bucket the platform owns (#1896): `Do` wraps one put/get/get_range/list/delete in a `storage.<operation>` client span and one `storage_operations_total{purpose,operation,result,reason}` observation, `Reason` reads a failure into access_denied/bucket_missing/quota_exceeded/not_found/other, and `WithPurpose` lets a caller writing through a shared client (the tile worker, an export, a script's output) report under its own purpose. Called by pkg/portal/s3adapter and the webhook objects; records through outbound.Metrics()
 │   ├── dbobs/                      # The platform's PostgreSQL pool opened through otelsql (#1896): a `postgres.<operation>` span per statement inside a traced request (none outside one), `db_client_operation_duration_seconds{operation}` and `db_client_slow_statements_total{operation}`, where the operation is the statement keyword, `vector_search` for a pgvector ranking, or the transaction step. The SQL text is on the span only with `OTEL_TRACES_INCLUDE_DB_STATEMENT=true`; the slow threshold is `MCP_PLATFORM_DB_SLOW_STATEMENT_THRESHOLD` (1s)

Makefile

@@ -53,7 +53,7 @@ GOVET := $(GO) vet
 GOFMT := gofmt
 GOLINT := golangci-lint
 
-.PHONY: all build test lint lint-full fmt clean install help docs-serve docs-build verify verify-release alert-rules-test builtin-pages-check \
+.PHONY: all build test lint lint-full fmt clean install help docs-serve docs-build docs-images-check verify verify-release alert-rules-test builtin-pages-check \
 	tools-check dead-code mutate patch-coverage patch-coverage-check coverage-summary doc-check acceptance acceptance-release acceptance-check acceptance-release-check release-tag-check schedule-lane schedule-lane-ui realdb-lane state-readers-check e2e-copy-check route-manifest-check posture-check preverify preverify-fast swagger swagger-check verify-checks verify-go verify-static verify-docker verify-ui vet-tags \
 	semgrep semgrep-diff semgrep-changed codeql sast osv embed-clean migrate-check \
 	frontend-install frontend-build frontend-build-content-viewer content-viewer-embed \
@@ -546,6 +546,10 @@ patch-coverage-check:
 	@echo "Checking patch coverage..."
 	@PATCH_COVERAGE_THRESHOLD=$(PATCH_COVERAGE_MIN) ./scripts/patch-coverage.sh
 
+## docs-images-check: Resolve every image the docs pin to a digest against its registry (network and Docker; run by the E2E Nightly)
+docs-images-check:
+	@./scripts/docs-image-digests.sh docs
+
 ## doc-check: Fail on orphaned docs, unresolved doc links, or unregistered tool refs; warn on undocumented changes
 doc-check:
 	@./scripts/doc-check.sh