Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ Do not multiply entities without necessity. The long-term goal is minimal code,

- A new Sandbox Provider, Harness, model provider or vendor feature changes only its adapter. It adds no Core execution path, store table or column, migration, deployment or configuration field, API field or Web UI specific to one vendor or Harness. The [Sandbox Provider guide](docs/sandbox-provider.md) and [Harness onboarding](contracts/agents-api/harness-onboarding.md) describe how to add an adapter.
- When the protocol cannot express what an adapter needs, change the protocol. Never add an optional side interface for one implementation.
- Implementing the declared `CheckpointProvider` lifecycle in one vendor's Provider is an adapter change. A vendor-only pause interface, a Core path for that vendor, vendor receipts in the store or a vendor idle setting in the deployment is not.
- Implementing the declared checkpoint lifecycle in one vendor's Provider is an adapter change. A vendor-only pause interface, a Core path for that vendor, vendor receipts in the store or a vendor idle setting in the deployment is not.
- Fix shared lifecycle, admission, cancellation, reuse and performance problems in the common flow, never in a branch selected by Harness, Runtime or vendor name. Core preparation and execution never branch on operating system or Environment source; platform support requires native CI builds and automated tests.
- Each Harness runs its own model and tool loop through a maintained upstream SDK or native protocol, in the Environment's declared workspace directory. Its native history or configuration directory is never the workspace. Never build a second executor, a hand-written model/tool loop or a compatibility framework to fabricate parity. The public API and persistence never depend on one engine's native item types.

Expand Down
2 changes: 1 addition & 1 deletion contracts/agents-api/core.openapi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6714,7 +6714,7 @@ paths:
- Sandbox Manager
/core/v1/sandbox/runtime-observations:
get:
description: Core key only. Each observation is labelled with its owning Project ID. Uses the existing read-only Runtime sampler, with bounded concurrency and no execution or provisioning. A provider with a batch metrics read, such as E2B, samples the page's running sandboxes in one bounded request.
description: Core key only. Each observation is labelled with its owning Project ID. Uses the existing read-only Runtime sampler, with bounded concurrency and no execution or provisioning.
parameters:
- description: Last Session ID from the preceding page
in: query
Expand Down
2 changes: 1 addition & 1 deletion contracts/agents-api/runtime-observability-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ Only list rows carry `disk`: null, or `{usage_bytes, limit_bytes}` with the rule
| `unsupported` | `runtime_mode_not_observable` | `none` and `self_hosted` Sessions. |
| `unavailable` | `allocation_pending` | The managed allocation does not exist yet or is being created. |
| `unavailable` | `runtime_not_running` | The allocation is being cleaned up or is released, or the provider reports the Runtime absent, stopped or suspended. |
| `unavailable` | `source_not_configured` | No observation source serves the allocation's provider. |
| `unavailable` | `source_not_configured` | This Core has no managed installation identity. |
| `unavailable` | `sample_timeout` | The provider read exceeded its deadline. |
| `unavailable` | `sample_unavailable` | The provider could not produce a current sample. |

Expand Down
16 changes: 7 additions & 9 deletions contracts/agents-api/runtime-observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,11 @@ The resolver (`services/core/internal/deployment/observation.go`) reads the Sess

Managed Docker, microsandbox and E2B allocations are observed. `none` and `self_hosted` Sessions are `unsupported`; Core never attributes shared host statistics to an `environment:none` Session.

The allocation's persisted `provider_key` selects exactly one configured source, which verifies the allocation's labels or equivalent ownership data before it returns values. Before any provider read, the allocation state decides some rows: `creating` or no allocation yet gives `allocation_pending`, `cleanup_pending` or `released` gives `runtime_not_running`, and a provider key without a source gives `source_not_configured`. A provider read that exceeds its deadline gives `sample_timeout`, a not-running result `runtime_not_running`, and an unavailable result `sample_unavailable`. Any other error, an ownership mismatch or an invalid sample fails the read.
Every managed allocation is read through the deployment's selected Sandbox Provider, which verifies the allocation's installation (`provider_key`) and labels or equivalent ownership data before it returns values. Before any provider read, the allocation state decides some rows: `creating` or no allocation yet gives `allocation_pending`, `cleanup_pending` or `released` gives `runtime_not_running`, and a Core without an installation identity gives `source_not_configured`. A provider read that exceeds its deadline gives `sample_timeout`, a not-running result `runtime_not_running`, and an unavailable result `sample_unavailable`. Any other error, an ownership mismatch or an invalid sample fails the read.

The observation boundary is declared in `services/core/internal/runtimeobs/source.go`. Each registered `SourceResolver` declares supported `ResolveObservationSource`; registration validates this declaration without loading configuration or reading the database. Core resolves each provider key once per page, then validates the returned `Source` and uses that same immutable source for every read of the key on that page. An unconfigured resolver returns typed `ErrUnavailable`, which produces `sample_unavailable` without a provider type. Other resolution errors follow the provider-read error rules above.
`Observe` belongs to the [Sandbox Provider protocol](../../docs/sandbox-provider.md); `services/core/internal/runtimeobs/source.go` owns the observation types and the `Source` view of a Provider. Core loads the selected Provider and its registered kind once per page and uses that same immutable Provider for every read on that page, reading each running target with `Observe`. Without a selection the load returns typed `ErrUnavailable`, which produces `sample_unavailable` without a provider type. Other load errors follow the provider-read error rules above.

A source declares supported `ObservationProviderType`, which returns its immutable telemetry identity: a lowercase letter followed by at most 31 lowercase letters, digits or underscores. Empty identities are invalid. Provider registration and every resolved binding validate the identity and operation declarations before any sample or export. Reconfiguration affects later source resolutions; it cannot change the identity or provider selected for an in-flight page. Generation routers continue to resolve each allocation through its recorded deployment generation.

A source implements `Observe` and declares `ObserveBatch` in its provider operations. When `ObserveBatch` is declared supported, one call reads up to 100 targets of that provider; when it is declared unsupported, Core reads each target with `Observe`. A failed batch read is never retried target by target. The [Sandbox Provider guide](../../docs/sandbox-provider.md) describes the operation declarations.
An observation's provider type is that registered kind: `docker`, `microsandbox` or `e2b`. Reconfiguration affects later pages; it cannot change the provider type or Provider selected for an in-flight page. Generation routers continue to resolve each allocation through its recorded deployment generation.

## Sample semantics

Expand Down Expand Up @@ -56,7 +54,7 @@ Cumulative vCPU time, guest memory usage and the effective memory limit come fro

### E2B

One helper `observe` request reads a page of at most 100 allocations: E2B's batch metrics for the sandboxes named in the private receipts, and a labelled listing of the installation's running sandboxes that confirms each one. The [E2B helper](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/tools/e2b-provider/README.md) owns that request. It never connects to, renews or changes a sandbox and writes no receipts.
Core reads each allocation with one helper `observe` request: E2B's metrics for the sandbox named in its private receipt, and a listing of running sandboxes with the allocation's labels that confirms it. The [E2B helper](https://github.com/MiniMax-AI/OpenAgentCore/blob/main/services/core/tools/e2b-provider/README.md) owns that request. It never connects to, renews or changes a sandbox and writes no receipts.

| E2B value | Sample field |
| --- | --- |
Expand All @@ -65,11 +63,11 @@ One helper `observe` request reads a page of at most 100 allocations: E2B's batc
| `memUsed`, `memTotal` | Memory usage and limit |
| `diskUsed`, `diskTotal` | Disk usage and capacity, kept only when both are present and the total is nonzero |

E2B reports no cumulative CPU time, so CPU seconds stay null. `observed_at` is E2B's point time; a point up to 30 seconds ahead of Core's clock is recorded at Core's time, and a larger lead is `sample_unavailable`. A sandbox missing from the running listing is `runtime_not_running`. A missing or malformed point, an ambiguous listing and an E2B API failure, a rejected key included, are `sample_unavailable`; a malformed point affects only its own row.
E2B reports no cumulative CPU time, so CPU seconds stay null. `observed_at` is E2B's point time; a point up to 30 seconds ahead of Core's clock is recorded at Core's time, and a larger lead is `sample_unavailable`. A sandbox missing from the running listing is `runtime_not_running`. A missing or malformed point, an ambiguous listing and an E2B API failure, a rejected key included, are `sample_unavailable`.

## Read budgets

A current list read handles one page of up to 100 Sessions (default 20) with at most eight concurrent provider reads. Each provider read has two seconds, a batch read at least five, and the whole list request ten; beyond that the list returns 503. A single-Session read has two seconds. No provider call is retried within a request, and Core keeps no observation cache.
A current list read handles one page of up to 100 Sessions (default 20) with at most eight concurrent provider reads. Each provider read has two seconds and the whole list request ten; beyond that the list returns 503. A single-Session read has two seconds. No provider call is retried within a request, and Core keeps no observation cache.

## Durations

Expand Down Expand Up @@ -123,7 +121,7 @@ With an OTLP endpoint configured, Core exports every record, both `on_read` and
| `agents.session.tokens.input` | Gauge, tokens | Measured Session input tokens |
| `agents.session.tokens.output` | Gauge, tokens | Measured Session output tokens |
| `agents.runtime.sample` | Monotonic delta sum | One per validated result, unavailable and unsupported included |
| `agents.runtime.sample.duration` | Delta histogram, seconds | Provider read duration; a batch read counts once |
| `agents.runtime.sample.duration` | Delta histogram, seconds | Provider read duration |

CPU and memory points are exported only when the sample has `started_at`; a missing measurement produces no point. Attributes are `agents.tenant.id`, `agents.session.id`, `agents.environment.id`, `agents.runtime.allocation.id`, `agents.runtime.mode`, `agents.runtime.provider.type`, `agents.runtime.status`, `agents.runtime.reason`, `agents.runtime.collection.source` and nanosecond `agents.runtime.resolved_at_unix_nano`, `agents.runtime.observed_at_unix_nano` and `agents.runtime.compute.started_at_unix_nano`. The nanosecond times keep records joinable when a backend stores event time at lower precision. Provider keys, receipts, native identifiers, raw errors, paths and credentials are never attributes.

Expand Down
4 changes: 2 additions & 2 deletions contracts/agents-api/zh/runtime-observability-api.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Runtime 遥测 API"
source: contracts/agents-api/runtime-observability-api.md
source_hash: d6111447def530b07c392d457cce0cd553f92dafb2e7cc7f481da47f816fcab2
source_hash: 6eca80ffefcaf8e26901659e5251518f84d2c6c93349085b378d46f9ac49149d
---

Core 通过 `/core/v1` 下的只读管理员路由报告托管 Runtime 和沙箱节点所使用的信息:当前 Runtime 观测值、单个 Session 的已存储 Runtime 历史记录,以及沙箱节点的主机观测值和历史记录。读取操作绝不创建、唤醒、续期或更改计算资源,也绝不向历史记录添加样本。[Runtime observability](runtime-observability.md) 定义了 Core 如何采集和保留这些值;[Console API usage](../../../docs/zh/web/console-api-usage.md) 列出了读取这些值的 Web 页面。
Expand Down Expand Up @@ -147,7 +147,7 @@ Authorization: Bearer <Core key>
| `unsupported` | `runtime_mode_not_observable` | `none` 和 `self_hosted` Session。 |
| `unavailable` | `allocation_pending` | 托管分配尚不存在或正在创建。 |
| `unavailable` | `runtime_not_running` | 分配正在清理或已释放,或者提供方报告 Runtime 不存在、已停止或已暂停。 |
| `unavailable` | `source_not_configured` | 该分配的提供方未配置任何观测源。 |
| `unavailable` | `source_not_configured` | 此 Core 没有托管 installation 标识。 |
| `unavailable` | `sample_timeout` | 提供方读取超过其截止时间。 |
| `unavailable` | `sample_unavailable` | 提供方无法生成当前样本。 |

Expand Down
Loading
Loading