Add administrator Core operational metrics

CONTRIBUTING.md

@@ -195,6 +195,21 @@ check targets remain; the separate `packages/codex-harness`, its build/check scr
 and its CI/`make check` gate are retired. Historical remote native probes are not
 current validation entrypoints.
 
+## Core operational metrics
+
+The administrator-only `/core/v1/admin/core-metrics` contract is documented in
+[core-metrics.md](contracts/agents-api/core-metrics.md). Keep this separate from
+Agent outcome and Sandbox capacity views. Instrument existing worker and job
+owners without changing scheduling, lease or retention behavior. Periodic pool
+pings and bounded in-process samples have explicit restart gaps; unknown values
+must remain null. Complete UTC buckets exclude the active partial bucket. Root
+Turn history is queried read-only from PostgreSQL with native timestamps.
+Count `execution_unavailable` at the existing HTTP error writer, once per rejected
+response; never record request/response bodies or infer this count from every
+503 or failed Turn. Builds inject the source commit with ldflags. No new monitoring
+service or storage system is required. Keep the frontend response shape aligned
+with the paired console contract.
+
 ## Architecture boundaries
 
 The following execution rules are retained from the source contributor guide.

contracts/agents-api/core-metrics.md

@@ -0,0 +1,99 @@
+# Core operational metrics
+
+`GET /core/v1/admin/core-metrics?range=1h|6h|24h|7d` is a deployment-administrator
+read. It implements the response shape agreed with Core Web PR #96
+(`53dc9d646bc6e3cc2cd9b8bbb353bc53c3513ecf`). It changes neither public `/v1`
+resources nor Agent or Sandbox metrics. Project API keys cannot call it.
+
+Only `range` is accepted, once; omission defaults to `1h`. Empty, repeated,
+unsupported or other query parameters return `400 invalid_request`. A missing
+metrics service returns `503 core_metrics_unavailable`. Partial measurement
+failures return the usual `200 core.metrics` envelope with `service.status` set
+to `degraded` and unavailable fields set to JSON null. No database or native
+error text, credentials, bodies, resource IDs or tenant labels are exposed.
+
+## Time and missing data
+
+The response's `range` uses UTC RFC 3339 boundaries. Its exclusive `end` is the
+most recent complete bucket boundary. The included interval is `[start,end)`;
+the current partial bucket is excluded from range aggregates and series.
+
+| Range | Bucket size | Buckets |
+| --- | --- | --- |
+| 1h | 60 seconds | 60 |
+| 6h | 300 seconds | 72 |
+| 24h | 900 seconds | 96 |
+| 7d | 7200 seconds | 84 |
+
+Current gauges and range aggregates are intentionally different: current worker,
+connection pool and process values are read when requested; queue gauges, database
+size and deployment maintenance are sampled every 30 seconds with bounded I/O.
+Samples older than 60 seconds are not reported as current. Queue, running and
+pool series report the highest **observed** value in each bucket, not a claim
+that all intermediate peaks were captured. Missing observations and the process's
+partial first bucket stay null. Successful periodic ping samples produce linear
+interpolated p50/p95; there is no request-triggered ping.
+
+A fixed-size in-process ring retains seven days of 30-second samples and
+rejection counts, plus two hours of padding for complete bucket alignment. Restart loses those measurements: no synthetic backfill occurs.
+The `execution.unavailable` count is null if the requested interval starts before
+this process's observation began; an entirely observed interval with no rejections
+is zero. PostgreSQL Turn history remains queryable across process restarts.
+An empty queue has a measured count of zero but no oldest age. No started Turns
+or successful ping samples means null percentiles, not zero latency.
+
+## Sources
+
+The envelope contains `object`, `range`, `service`, `execution`, `database`,
+`jobs` and `process`, matching the typed client from PR #96. All numeric values
+and `service.execution_owner` are nullable; lists of complete buckets are always
+present.
+
+- `service.revision` is a full source commit injected into `main.buildRevision`
+  by the standalone builder's `-ldflags`. Manual builds without a valid revision
+  report null. `started_at` records process initialization. `execution_owner`
+  reflects the execution worker's existing lease checks, with unknown ownership
+  represented as null. `maintenance` means the saved deployment is in maintenance;
+  measurement or job failures take precedence as `degraded`.
+- `execution.slots_in_use` is the worker's active Session reservation set. Its
+  configured capacity is four; environment input, Turns and file work share it.
+  It does not count native harness subprocesses. Worker-disabled installations
+  have zero configured execution slots.
+- `queued_turns` and `in_progress_turns` count root rows in `turns`, including
+  operational state retained for deleted Sessions. Native Subagent views and
+  pending Environment input reservations are not extra queued root Turns.
+  `waiting_for_daemon` is the queued subset whose Session device binding is
+  absent from the actual connected-device registry. `oldest_queued_seconds`
+  measures the oldest queued row's `created_at`.
+- `queue_wait_ms` uses `started_at - created_at`, in milliseconds, for Turns
+  started in the interval. Each bucket uses its own started Turns, with PostgreSQL
+  `percentile_cont`. `interrupted` counts failed Turns whose outcome error code is
+  `execution_interrupted`, using `completed_at` in the interval.
+- `unavailable` counts actual HTTP errors emitted with code
+  `execution_unavailable`, once per rejected response. Other 503 codes and errors
+  occurring after a stream has already started are not counted. The existing error
+  writer reports the code; no response/request body capture is involved.
+- `database.ping_ms` measures a periodic pool ping, including connection acquisition.
+  Pool `in_use`, `idle` and `max` come from `pgxpool.Stat()`. `size_bytes` is
+  `pg_database_size(current_database())`, not host disk usage. Failure to measure
+  one value does not turn it into zero.
+- `process.memory_bytes` is Go `runtime.MemStats.Alloc` (allocated heap bytes),
+  not RSS or container memory. `goroutines` is `runtime.NumGoroutine()`.
+
+## Background jobs
+
+The four bounded IDs are `scheduler`, `runtime_sampler`, `history_cleanup` and
+`audit_cleanup`. Each reports `status`, `last_run_at`, `processed` and `failed`.
+A not-yet-observed run is unknown; a disabled or stopped loop is stopped. The
+last-run time is the completion/observation of the last pass, not its next deadline.
+
+Scheduler processed counts selected Turn/environment work in that poll. Runtime
+sampling counts observed and failed targets from its existing sweep result.
+Cleanup processed counts confirmed removed rows; an unsuccessful cleanup reports
+unknown processed count. Cleanup/scheduler failure counts identify failed passes,
+not guessed numbers of lost rows or failed Turns. The actual scheduling, sampling,
+retention and execution lifecycles keep their existing owners and timing.
+
+No additional telemetry database, monitoring server, model-provider probe,
+message queue, object store, host disk measurement or scheduling mechanism is
+introduced. Metrics cannot authorize execution or change resource ownership.

contracts/agents-api/sandbox-manager.openapi.yaml

@@ -172,6 +172,169 @@ definitions:
           $ref: '#/definitions/store.RuntimeNode'
         type: array
     type: object
+  coremetrics.Database:
+    properties:
+      ping_ms:
+        $ref: '#/definitions/coremetrics.Latency'
+      pool:
+        $ref: '#/definitions/coremetrics.Pool'
+      series:
+        items:
+          $ref: '#/definitions/coremetrics.DatabaseBucket'
+        type: array
+      size_bytes:
+        type: integer
+        x-nullable: true
+    type: object
+  coremetrics.DatabaseBucket:
+    properties:
+      ping_p95_ms:
+        type: number
+        x-nullable: true
+      pool_in_use:
+        type: integer
+        x-nullable: true
+      start:
+        type: string
+    type: object
+  coremetrics.Execution:
+    properties:
+      connected_daemons:
+        type: integer
+        x-nullable: true
+      in_progress_turns:
+        type: integer
+        x-nullable: true
+      interrupted:
+        type: integer
+        x-nullable: true
+      oldest_queued_seconds:
+        type: number
+        x-nullable: true
+      queue_wait_ms:
+        $ref: '#/definitions/coremetrics.Latency'
+      queued_turns:
+        type: integer
+        x-nullable: true
+      series:
+        items:
+          $ref: '#/definitions/coremetrics.ExecutionBucket'
+        type: array
+      slots_in_use:
+        type: integer
+        x-nullable: true
+      slots_total:
+        type: integer
+        x-nullable: true
+      unavailable:
+        type: integer
+        x-nullable: true
+      waiting_for_daemon:
+        type: integer
+        x-nullable: true
+    type: object
+  coremetrics.ExecutionBucket:
+    properties:
+      in_progress:
+        type: integer
+        x-nullable: true
+      queue_wait_p95_ms:
+        type: number
+        x-nullable: true
+      queued:
+        type: integer
+        x-nullable: true
+      start:
+        type: string
+    type: object
+  coremetrics.Job:
+    properties:
+      failed:
+        type: integer
+        x-nullable: true
+      id:
+        type: string
+      last_run_at:
+        type: string
+        x-nullable: true
+      processed:
+        type: integer
+        x-nullable: true
+      status:
+        type: string
+    type: object
+  coremetrics.Latency:
+    properties:
+      p50:
+        type: number
+        x-nullable: true
+      p95:
+        type: number
+        x-nullable: true
+    type: object
+  coremetrics.Pool:
+    properties:
+      idle:
+        type: integer
+        x-nullable: true
+      in_use:
+        type: integer
+        x-nullable: true
+      max:
+        type: integer
+        x-nullable: true
+    type: object
+  coremetrics.Process:
+    properties:
+      goroutines:
+        type: integer
+        x-nullable: true
+      memory_bytes:
+        type: integer
+        x-nullable: true
+    type: object
+  coremetrics.Range:
+    properties:
+      end:
+        type: string
+      resolution_seconds:
+        type: integer
+      start:
+        type: string
+    type: object
+  coremetrics.ServiceState:
+    properties:
+      execution_owner:
+        type: boolean
+        x-nullable: true
+      revision:
+        type: string
+        x-nullable: true
+      started_at:
+        type: string
+        x-nullable: true
+      status:
+        type: string
+    type: object
+  coremetrics.View:
+    properties:
+      database:
+        $ref: '#/definitions/coremetrics.Database'
+      execution:
+        $ref: '#/definitions/coremetrics.Execution'
+      jobs:
+        items:
+          $ref: '#/definitions/coremetrics.Job'
+        type: array
+      object:
+        type: string
+      process:
+        $ref: '#/definitions/coremetrics.Process'
+      range:
+        $ref: '#/definitions/coremetrics.Range'
+      service:
+        $ref: '#/definitions/coremetrics.ServiceState'
+    type: object
   store.AdminAssetCounts:
     properties:
       agents:
@@ -2636,6 +2799,43 @@ paths:
       summary: Copy assets into another Project
       tags:
         - Core Administration
+  /core/v1/admin/core-metrics:
+    get:
+      description: Deployment administrator only. Complete UTC buckets; unknown measurements are null. Samples are process-local and are not backfilled after a restart.
+      parameters:
+        - description: Time range (default 1h)
+          enum:
+            - 1h
+            - 6h
+            - 24h
+            - 7d
+          in: query
+          name: range
+          type: string
+      produces:
+        - application/json
+      responses:
+        "200":
+          description: OK
+          schema:
+            $ref: '#/definitions/coremetrics.View'
+        "400":
+          description: Bad Request
+          schema:
+            $ref: '#/definitions/v1.ErrorResponse'
+        "401":
+          description: Unauthorized
+          schema:
+            $ref: '#/definitions/v1.ErrorResponse'
+        "503":
+          description: Service Unavailable
+          schema:
+            $ref: '#/definitions/v1.ErrorResponse'
+      security:
+        - DeploymentAdminAuth: []
+      summary: Retrieve Core operational metrics
+      tags:
+        - Core Administration
   /core/v1/admin/projects:
     get:
       parameters:

docs/api/web-management.md

@@ -88,3 +88,13 @@ Origin/Host requests are rejected; unavailable Core or rejected upstream redirec
 return 502. Console authentication uses its own error envelope. Management resource
 errors and deletion constraints are documented in the administrator reference and
 generated schema; do not interpret every empty or failed read as an absent resource.
+
+## Core metrics
+
+`GET /core/v1/admin/core-metrics?range=1h|6h|24h|7d` returns Core process, execution
+queue/slots, PostgreSQL and background-job measurements. It uses deployment
+administrator authentication, rejects arbitrary query filters and never grants
+Agent execution access. See the [exact measurement contract](../../contracts/agents-api/core-metrics.md)
+for complete buckets, null values, units and process-local retention. Frontend
+implementation is maintained separately; this backend change does not modify
+Agent metrics or the public Agent API.

scripts/build-agents-api-release.sh

@@ -55,7 +55,7 @@ if [[ "$go_version" != "$required_go" ]]; then
   printf 'Agents API release requires %s; found %s\n' "$required_go" "$go_version" >&2
   exit 1
 fi
-AGENTS_API_BUILD_DIR="$release_context/package/bin" \
+AGENTS_API_BUILD_REVISION="$source_revision" AGENTS_API_BUILD_DIR="$release_context/package/bin" \
   "$release_context/source/scripts/build-agents-api.sh"
 require_clean_source
 if [[ "$(git -C "$repo_root" rev-parse HEAD)" != "$source_revision" ]]; then

scripts/build-agents-api.sh

@@ -11,6 +11,12 @@ for directory in "$runtime_root" "$output_dir"; do
   fi
 done
 
+revision="${AGENTS_API_BUILD_REVISION:-$(git -C "$repo_root" rev-parse HEAD 2>/dev/null || true)}"
+if [[ -n "$revision" && ! "$revision" =~ ^[0-9a-f]{40}$ ]]; then
+  printf 'Invalid Agents API source revision\n' >&2
+  exit 1
+fi
+
 mkdir -p "$runtime_root/cache/agents-api-builds"
 build_context="$(mktemp -d "$runtime_root/cache/agents-api-builds/source.XXXXXX")"
 trap 'rm -rf "$build_context"' EXIT
@@ -31,7 +37,7 @@ tar -C "$repo_root" -cf - \
     artifact="agents-api-$command"
     if [[ "$command" == server ]]; then artifact=agents-api; fi
     if [[ "$command" == sandbox-node ]]; then artifact=parsar-sandbox-node; fi
-    go build -mod=readonly -trimpath -buildvcs=false \
+    go build -mod=readonly -trimpath -buildvcs=false -ldflags "-X main.buildRevision=$revision" \
       -o "$build_context/bin/$artifact" "./services/agents-api/cmd/$command"
   done
 )

scripts/build-core-distribution.sh

@@ -101,7 +101,7 @@ cp services/agents-api/deploy/codex/seccomp.json "$bundle/runtime/"
 cp LICENSE "$bundle/"
 cp -R site "$bundle/site"
 
-AGENTS_API_BUILD_DIR="$stage/core/bin" scripts/build-agents-api.sh
+AGENTS_API_BUILD_REVISION="$revision" AGENTS_API_BUILD_DIR="$stage/core/bin" scripts/build-agents-api.sh
 (
   cd services/agents-api/tools/microsandbox-provider
   GOWORK=off CGO_ENABLED=1 go build -mod=readonly -trimpath \