[cjimti]

Description

Adds GET /api/v1/admin/endpoints and GET /api/v1/admin/endpoints/{name} (internal/admin/endpointapi) with the portal's Admin > Automations > Endpoints tab, max_concurrent_calls on api and graphql connections (internal/connlimit), exclusive: queue for managed scripts, background graphql_export and api_export runs (internal/exportasync, internal/platform/calltask), timeout cap reporting (internal/timeoutcap), structured upstream failures on graphql_query, and manage_table action=check. It covers six tickets, the whole of epic #2220. Two defects found along the way are fixed with it (listed after the tickets).

1. feat(portal): admin Endpoints page with every binding and its traffic #2208 endpoints admin.
   • Routes: GET /api/v1/admin/endpoints?range=day|week lists every binding ordered by name: its script (id, name, owner), address, methods, path, access (public or signed_in), open_reads, whether it is on, its limits, and who last changed it. GET /api/v1/admin/endpoints/{name} adds a series bucketed over the range, the top signed-in callers and the failed requests of the last 7 days.
   • Traffic: requests by status class, failures, p50 and p95 duration, distinct signed-in callers and the last request's time, read from each request's audit_logs row.
   • Refusals: requests turned away before the handler ran are counted by reason: bot_check, rate_limit, caller_rate_limit, body_limit, credential, origin. A replica counts in memory and adds its counts every 10s, one row per endpoint, hour and reason, into script_endpoint_refusals (migration 000207). A flood moves a counter instead of writing a row per request. The retention loop keeps 30 days (sweep endpoint_refusals).
   • Portal: an Endpoints tab beside Automations, Schedules and Runs (/portal/admin/automations/endpoints). A row opens the endpoint's page: a stacked chart of answered requests by status class beside the refused ones, the refusals by reason, the top callers (signed-in endpoints), the failed requests, Turn on/off, and Edit (the automation's page). "Bind an automation" binds one from the tab.
2. Per-connection concurrency limit with queueing for upstreams that cannot take parallel calls #2209 call limits.
   • The setting: max_concurrent_calls (1 to 32) on an api or graphql connection holds its upstream to that many calls at once. A value that is not a whole number is refused at save naming the key; before, a string was stored and read as no limit.
   • Across replicas: a slot is a PostgreSQL session advisory lock in the two-integer key form, (hash of kind and connection, slot n), held on a pooled database connection for the length of the call. A deployment with no database holds the limit per process.
   • Every caller: the slot is taken in the outbound transport (upstreamauth.NewHTTPClient) and given back when the response body is read or closed, so an interactive call, an export's page walk and a script's platform.call all count.
   • Over the limit: a call waits for a slot, polling at 20ms rising to 250ms, within its own deadline. A call whose time runs out while waiting fails naming the limit and how long it waited.
   • Reported: list_connections answers a top-level call_limits list (its connection entries are closed objects in older releases, so the field is not added to them). Metrics upstream_calls_inflight{kind,connection} and upstream_slot_wait_seconds{kind,connection}. The connection forms in Admin > Connections carry the field for both kinds.
   • Queued scripts: a script's exclusive is now off, refuse or queue (exclusive_mode, migration 000205, backfilled from the boolean). true still reads as refuse. With queue, a second run is accepted and stays pending until the open run ends; a run's queued flag and a partial unique index hold one queued script to one running run. The script page's setting reads "Overlapping runs: Allowed, Refused, Queued".
3. Report when a caller's timeout_seconds is capped by the connection's call_timeout #2210 timeout caps.
   • api_invoke_endpoint, api_export, graphql_query and graphql_export report timeout_requested_seconds, timeout_applied_seconds and timeout_capped_by (connection call_timeout or export max_timeout) whenever the caller's timeout_seconds was cut.
   • A call that then timed out says so in its error, naming the cap and that the connection's administrator owns it, for example "timed out after 1s (the connection's call_timeout; timeout_seconds 240 was capped)".
4. graphql_query: surface upstream status, body snippet and upstream_retryable when errors is absent #2211 graphql_query failures.
   • No errors array: a non-2xx answer reports upstream_status, upstream_body (the first 2 KiB, cut on a rune boundary), upstream_retryable and retry_after_seconds, through the same upstreamretry reading the api gateway uses. A query is read as a GET and a mutation as a POST, so a 503 is retryable for a query only.
   • No answer at all: a send the endpoint never answered (refused, reset, DNS, TLS, deadline) is a structured result with transport_error (naming the endpoint with any user info and query string removed), upstream_error: true and error.code: upstream_unavailable, where it used to be a bare error result.
   • graphql_export reports both the same way, and writes nothing.
5. Run long graphql_export / api_export page walks in the background and return a run id #2212 background exports.
   • Starting: graphql_export and api_export take async. A walk that runs in the background answers at once with status: running, a run_id, an idempotency_key (the caller's, or one minted) and the call that reads it again.
   • Reading it again: calling the same export with that idempotency_key answers with the run's progress while it works and with the export's result once it ends. A client without the Tasks extension reads it the same way, and two calls under one key share one walk.
   • Stored: call_tasks gains idempotency_key, background and progress (migration 000206), with a unique index holding one working run per caller, tool and key. Progress (pages and items) is written with the heartbeat the call-task runner already stamps.
   • Hearing about it later: platform_info's background_exports names the exports that ended since the caller was last briefed, each with its key.
6. platform.export register= validates the scratch target after writing, failing the run with its output already written #2213 register check.
   • Before the write: platform.export(..., register=...) asks manage_table action=check whether the connection and table name would be accepted before the file is written, so a refused target fails with "nothing was written". A failure after the write lists the written output on the run.
   • Statically: validate reports the same refusal for a literal connection.
   • The refusal names the way out: "Connections you can register a table on: ...", read from the live Trino connections at the moment of refusal.
   • manage_table action=check is a read: it answers the catalog, schema and query table a registration would use, writing nothing.

Defects fixed with it.

1. A deadline reached during the slot's lock query, rather than between two looks, reported "taking a slot failed" instead of the call's time running out (internal/connlimit/connlimit.go, TestShared_DeadlineDuringTheQuery).
2. Back from the Endpoints tab showed the Endpoints tab under the Automations address: the page is now keyed by the address it was opened at (ui/src/pages/scripts/AdminScriptRoutes.tsx).

Also.

• Size and surface budgets: the registration catalog writer moves to internal/platform/tableregister/regcatalog, the in-process transport to internal/handlertransport, and the cap arithmetic to internal/timeoutcap; ExclusiveMode lives in internal/runstate so pkg/script's exported surface stays within budget.
• Import ratchet: test/structure/testdata/allowed_internal_imports.txt gains 34 edges and drops one (internal/platform/tableregister -> pkg/catalog, now regcatalog's). The new ones are the six packages above reaching their dependencies, pkg/platform installing the limiter and the background runner, and both gateway toolkits reaching exportasync, timeoutcap and (graphql) upstreamretry.
• Route gate: TestEveryRouteReportsItsTemplate_RealDB excuses the two endpoint admin routes as it already does the bot check settings: both are mounted by endpointwire.Mount only once the script layer has started, which that test does not run.
• Docs: docs/server/graphql-gateway.md, docs/server/api-gateway.md, docs/server/configuration.md, docs/server/tools.md, docs/server/observability.md, docs/scripts/running.md, docs/scripts/endpoints.md, the writing-managed-scripts built-in page, CLAUDE.md, docs/llms.txt and docs/llms-full.txt.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• Test improvement (new or updated tests)
• Documentation update
• Stability/performance improvement

Related Issues

Closes #2208
Closes #2209
Closes #2210
Closes #2211
Closes #2212
Closes #2213
Closes #2220

Readers of this state

• GET /api/v1/admin/endpoints and /{name} (new; endpointapi.List, Detail, Endpoint, Traffic, Bucket, Caller, Script, StatusClasses).
  • Admin > Automations > Endpoints (ui/src/pages/scripts/endpoints/EndpointsTab.tsx) and the endpoint page (EndpointDetailPage.tsx) read every field; the types are ui/src/api/admin/types and the hooks ui/src/api/admin/hooks.
  • Every replica answers from audit_logs and script_endpoint_refusals, both shared; a replica's refusals reach the table within its 10s flush.
  • The MSW mock (ui/src/mocks/data/endpoints.ts, handlers/endpoints.ts) and the screenshot routes admin-endpoints and admin-endpoint-detail carry the shape.
• A script's exclusive is now a string (off, refuse, queue) where it was a boolean.
  • PUT /api/v1/portal/scripts/{id}/exclusive takes the string or a boolean, and the script reads back as the string from GET /api/v1/portal/scripts/{id}, manage_script get and list.
  • The script page (ui/src/pages/scripts/ScriptExclusiveSetting.tsx) reads and writes the three modes.
  • The scheduler and run_script read it at run creation; a queued run is held by the claim on every replica.
  • test/acceptance/issue_1986_test.go reads back the mode names.
• list_connections gains top-level call_limits. Read by the acceptance test; the connection entries are unchanged.
• Connection config gains max_concurrent_calls. The api and graphql connection forms (ApiGatewayConfigForm, GraphQLConfigForm) read and write it; GET /api/v1/admin/connection-instances/{kind}/{name} returns it as stored.
• platform_info gains top-level background_exports. Read by the acceptance test and by an agent at session start.
• Tool results gain fields: timeout_* on the four gateway tools; upstream_body, upstream_retryable, retry_after_seconds, transport_error and error on graphql_query and graphql_export; status, run_id, idempotency_key and progress on a background export. Every output schema is open to the platform's keys (MCPOutputSchemaMiddleware), and tools/list carries the new async input.

Testing

• Acceptance through the real surface on the local dev stack (two replicas behind the proxy), every criterion passing; transcripts in build/2208 through build/2213 and the build/1986 addendum.

  • feat(portal): admin Endpoints page with every binding and its traffic #2208: a public endpoint's three bot-check refusals and a signed-in endpoint's two answered GETs and one refused request show in the list with their automation and owner, and on the endpoint's page; an analyst's key is refused with the same status as every other admin route (401).
  • Per-connection concurrency limit with queueing for upstreams that cannot take parallel calls #2209: four 700ms calls, two through each replica, on a connection limited to 1, all succeed and take longer than four in a row (limit sent as an integer and as a float); a call with timeout_seconds 1 behind a held slot fails naming max_concurrent_calls and its wait; list_connections names the limit; a string limit is refused with 400; with exclusive: "queue" a second run stays pending and starts after the first finishes.
  • Report when a caller's timeout_seconds is capped by the connection's call_timeout #2210: graphql_query with call_timeout 5s and timeout_seconds 240 reports the cap; api_invoke_endpoint stopped at a 1s call_timeout names it.
  • graphql_query: surface upstream status, body snippet and upstream_retryable when errors is absent #2211: a 503 with Retry-After 7 from a platform endpoint used as the upstream carries its body and is retryable for a query, not a mutation; a connection to a closed port answers with transport_error and upstream_unavailable.
  • Run long graphql_export / api_export page walks in the background and return a run id #2212: a graphql_export page walk runs in the background and is read again by its key; an api_export started with async is joined, not walked twice, under its key.
  • platform.export register= validates the scratch target after writing, failing the run with its output already written #2213: manage_table check answers on a scratch connection and refuses one without a scratch target, naming the connection that has one; a draft registering on a connection with no scratch target fails with nothing written, and validate flags it.

• Real-Postgres: the advisory-lock limit across two pools, queued-run claims, background-export storage and its unique key, the refusal store and the endpoint traffic aggregates, and the SQL statement gate.

• make verify passed: the full Go race and coverage run (total 93.9%, patch 89.7%), the schedule lane, the real-database lane, the full vitest suite (3,231) and Playwright (296).

• Ran make test locally

• Tested manually

• Added new tests for changes (if applicable)

Checklist

• My code follows the project's style guidelines (make lint)
• I have read CONTRIBUTING.md
• I have updated documentation if needed
• This PR is focused and does not include unrelated changes

[codecov]

Codecov Report

❌ Patch coverage is 91.45374% with 97 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.79%. Comparing base (0a1601f) to head (191b1b3).

Files with missing lines	Patch %	Lines
pkg/toolkits/graphql/export.go	68.75%	15 Missing ⚠️
internal/httpserver/tablemounts.go	0.00%	13 Missing ⚠️
internal/admin/endpointapi/endpointapi.go	90.67%	11 Missing ⚠️
internal/admin/endpointapi/traffic.go	88.88%	9 Missing ⚠️
pkg/observability/metrics_outbound.go	42.85%	8 Missing ⚠️
internal/platform/calltask/background.go	91.86%	7 Missing ⚠️
internal/endpoint/epwire/epwire.go	33.33%	6 Missing ⚠️
internal/httpserver/endpointwire/endpointwire.go	25.00%	6 Missing ⚠️
internal/platform/tableregister/tooladapter.go	0.00%	6 Missing ⚠️
internal/exportasync/answer.go	92.85%	4 Missing ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2222      +/-   ##
==========================================
- Coverage   94.82%   94.79%   -0.03%     
==========================================
  Files        1361     1374      +13     
  Lines      110410   111373     +963     
==========================================
+ Hits       104693   105574     +881     
- Misses       5699     5781      +82     
  Partials       18       18              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.