diff --git a/docs/spec/trace/extend_opentelemetry.md b/docs/spec/trace/extend_opentelemetry.md index dfb27c0..16c460a 100644 --- a/docs/spec/trace/extend_opentelemetry.md +++ b/docs/spec/trace/extend_opentelemetry.md @@ -24,12 +24,12 @@ The span carries the join pair (`acs.session.id` and `acs.request_id`), the tool ```json { - "traceId": "85c4878c9bda9e5b4f2fc6aae242d31f", - "spanId": "dea6530ad1681c0e", + "traceId": "13717f291e2368066d2fa3f2434ed2f6", + "spanId": "2a7efedbb7ee3881", "name": "gen_ai.tool.call", "kind": 1, - "startTimeUnixNano": "1791928279698000000", - "endTimeUnixNano": "1791928279749301083", + "startTimeUnixNano": "1792534253716796645", + "endTimeUnixNano": "1792534253780000000", "attributes": [ { "key": "acs.session.id", @@ -146,7 +146,7 @@ The span carries the join pair (`acs.session.id` and `acs.request_id`), the tool } ], "name": "acs.decision", - "timeUnixNano": "1791928279749295500", + "timeUnixNano": "1792534253780000000", "droppedAttributesCount": 0 } ], @@ -168,7 +168,7 @@ The span above is real output from the reference Guardian. This section explains ### Two streams, joined by a reader -No single process sees every outcome of a step. The Guardian sees each request it receives and the answer it gives. It exports one span per decided step, with the `acs.decision` event on it. It also reports the failures it can see itself: an error it returned instead of a decision, a decision it finished after the negotiated timeout, and a step still open when it shut down. +No single process sees every outcome of a step. The Guardian sees each request it receives and the answer it gives. It exports one span per step it trusted, with the `acs.decision` event on it, and records a request it rejected before trusting it apart, under names other than `acs.session.id` and `acs.request_id`. It also reports the failures it can see itself: an error it returned instead of a decision because it could not handle the step, a decision it finished after the negotiated timeout, and a step still open when it shut down. The host sees what the Guardian can't. An unreachable Guardian has nothing to record. A Guardian whose answer arrives after the host stopped waiting can't see what the host did in the meantime. The host adapter records both cases in its own audit log. A converter, `audit-to-otlp.ts`, turns that log into spans of the same OTLP/JSON shape, each with an `acs.decision.failure` event whose `acs.failure.reported_by` is `observed_agent`. @@ -224,7 +224,7 @@ sequenceDiagram A late decision gets both events on one span: the `acs.decision` with the verdict, then an `acs.decision.failure` with class `timeout`. When either party's stream holds a failure event for a pair, a reader doesn't count that pair's `acs.decision` as enforced. [Trace Events](./events.md) states that rule. -The Guardian never copies its error text into the trace. A failure event carries the error code, the timeout, the elapsed time and the request size when they're known, because error text can quote the request. A replay rejection gets no failure event, since its pair belongs to the earlier request. +The Guardian never copies its error text into the trace. A failure event carries the error code, the timeout, the elapsed time and the request size when they're known, because error text can quote the request. A request the Guardian rejected before trusting it, by a failed validation, signature, replay or timestamp check, gets no event under the pair it carried, since that pair can be a genuine request's. The reference Guardian records it as a separate span under its own names instead. If you only need to read a trace, you can stop here. The rest of this section is for running the pipeline or building your own. @@ -238,7 +238,7 @@ The collector path, set with `ACS_OTEL_ENDPOINT`, sends the same spans over OTLP The two Guardian outputs fail independently. The first failed write or send disables that output for the rest of the process and prints one line on stderr. The other output keeps running, and the decision goes out unchanged, as the [failure isolation](#failure-isolation) rule below requires. The [Guardian README](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/packages/guardian/README.md#troubleshooting) lists the messages you may see. -The committed [sample trace](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl) holds both streams in one file: five spans from the Guardian and one from the converter. Every value in it is synthetic. +The committed [sample trace](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl) holds both streams in one file: five spans from the Guardian, one of them a rejected request, and one from the converter. Every value in it is synthetic. ### What the labels don't prove diff --git a/reference-implementations/agt/README.md b/reference-implementations/agt/README.md index cdc6a86..15d772f 100644 --- a/reference-implementations/agt/README.md +++ b/reference-implementations/agt/README.md @@ -495,6 +495,7 @@ These are known limits of the tree as it stands. None is fixed. - **Memory and disk grow without bound.** The Guardian's session store never evicts a session. The envelope log, the audit log and the session-context log have no rotation and no size cap. A long-lived Guardian grows until something else stops it. - **Log writes sit on the decision path.** The Guardian writes two synchronous file appends per request in a single-threaded server. A slow disk blocks every in-flight decision. - **The wire is unauthenticated.** See [What this project is, and is not](#what-this-project-is-and-is-not). Loopback by default is the only protection. +- **A well-formed forgery is trusted.** The Guardian checks a request against the schema and nothing else, so a forged or replayed request that passes the schema is decided, recorded on the trace under the ids it carries, and appended to that session's hash chain. Signature, replay and timestamp checks arrive with PR #229. Until then, a trace's request pairs, and the chain, are only as trustworthy as access to the Guardian's port. - **The egress gate has three measured soft spots.** A shell command whose destination the extractor cannot parse falls through to `allow`, not `deny`. With the shipped allowlist, any command whose text contains an off-list URL is denied, whether or not it reaches that URL. A `WebFetch` URL reaches AGT's gate with no parsing, so four userinfo-style URL shapes pass on the fetch route and fail on the shell route. Closing the last one means editing a `.rego` file, which this project does not do. - **An `ask` never carries `ask_details`.** The Guardian's decision type has no such field. Every `ask` fails the response schema. A `defer` would fail the same way if any AGT verdict mapped to it. - **The handshake is not negotiated.** The Guardian returns constants and never reads the ClientHello. Neither side declares ACS profiles. diff --git a/reference-implementations/agt/packages/guardian/README.md b/reference-implementations/agt/packages/guardian/README.md index 4c604f7..3ced826 100644 --- a/reference-implementations/agt/packages/guardian/README.md +++ b/reference-implementations/agt/packages/guardian/README.md @@ -56,7 +56,7 @@ Set `ACS_OTEL_FILE`, `ACS_OTEL_ENDPOINT`, or both. With neither set, no exporter ACS_OTEL_FILE=.acs/otel-traces.jsonl bun run guardian ``` -Each decided step becomes one span, named by [`trace/otel-mapping.json`](../../../../specification/v0.1.0/trace/otel-mapping.json): `gen_ai.tool.call` for `steps/toolCallRequest` and `gen_ai.tool.result` for `steps/toolCallResult`. The span opens when the request is recorded and closes when the response is. It carries each attribute the mapping lists for it when the envelope supplies a value. An attribute the envelope omits, such as `acs.capability`, is absent from the span. +Each step the Guardian trusted becomes one span, named by [`trace/otel-mapping.json`](../../../../specification/v0.1.0/trace/otel-mapping.json): `gen_ai.tool.call` for `steps/toolCallRequest` and `gen_ai.tool.result` for `steps/toolCallResult`. The span opens when the request is recorded and closes when the response is. It carries each attribute the mapping lists for it when the envelope supplies a value. An attribute the envelope omits, such as `acs.capability`, is absent from the span. Every span also carries `acs.session.id`, and `acs.request_id` whenever the envelope has one, so a session can be rebuilt from the trace alone. When an argument or output carries a Provenance object, the span carries `acs.provenance.origin` as an array, one element per Provenance object in wire order. `acs.provenance.source_id` is the matching array, with `""` where an object has no `source_id`, and it is left out when no object has one. Both stop at 64 objects. `acs.provenance.lineage_depth` is not emitted: `provenance.json` defines no such field. @@ -64,9 +64,13 @@ Every string attribute is cut at 4096 characters, a long `acs.reasoning` include The decision rides on the span as an `acs.decision` event. The event always carries `acs.decision`, carries `acs.reasoning` when the response has one, and carries `acs.reason_codes`, `acs.policy.ids`, `acs.policy.versions`, `acs.policy.rule_ids`, `acs.cited_provenance_ids` and `acs.chain_hash` only when the response carries them. `acs.evaluator` appears when the Guardian's response carries `metadata.evaluator`. This Guardian sends it once PR #229 is merged. -This Guardian does not send `chain_hash`, `cited_provenance_ids`, or `policy_version` on its responses yet. `acs.chain_hash` and `acs.cited_provenance_ids` are absent from its spans, and `acs.policy.versions` carries `""` for each policy reference. The three policy arrays always have the same length as `policy_references`, with `""` where a reference has no value. `handshake/hello`, methods this Guardian does not dispatch, and unparseable bodies produce no span. +This Guardian does not send `chain_hash`, `cited_provenance_ids`, or `policy_version` on its responses yet. `acs.chain_hash` and `acs.cited_provenance_ids` are absent from its spans, and `acs.policy.versions` carries `""` for each policy reference. The three policy arrays always have the same length as `policy_references`, with `""` where a reference has no value. A valid `handshake/hello`, a valid request for a method this Guardian does not dispatch, and an unparseable body produce no span. -A step that got no decision is recorded as an `acs.decision.failure` event on the same span, with `acs.failure.class`, `acs.failure.posture` and `acs.failure.reported_by`, plus the error code, the timeout, the elapsed time and the request size when known. The Guardian emits it for an error it answered instead of a decision, for a decision it finished after the negotiated timeout, and for a step still open when it shut down. It carries the error code but never the error text, because that text can quote the request. A replay rejection (`-32005`) gets no event, because its pair belongs to the earlier request. The converter below produces the same event, reported by the observed agent, from the host adapter's audit log. +A step that got no decision is recorded as an `acs.decision.failure` event on the same span, with `acs.failure.class`, `acs.failure.posture` and `acs.failure.reported_by`, plus the error code, the timeout, the elapsed time and the request size when known. The Guardian emits it for an error it answered instead of a decision, for a decision it finished after the negotiated timeout, and for a step still open when it shut down. It carries the error code but never the error text, because that text can quote the request. The converter below produces the same event, reported by the observed agent, from the host adapter's audit log. + +A request the Guardian cannot vouch for is recorded apart from the steps it governed, as a `guardian.rejected_request` span. Such a request can carry any session and request id, a genuine request's included. So the span carries no `acs.session.id`, `acs.request_id`, step attribute or `acs.decision` event. It records the ids the request claimed as `guardian.rejected.claimed_session_id` and `guardian.rejected.claimed_request_id`, with `guardian.rejected.reason` and, for an error answer, `guardian.rejected.error_code`. These names belong to this Guardian, not to the ACS mapping. + +The reasons are `envelope_invalid`, for an envelope that fails validation, and `signature_invalid`, `replay_detected` or `timestamp_out_of_window`, for a step answered with that error by whichever check produced it. The last is `guardian_fault`, for a request the Guardian could not check at all, a missing schema directory for example. The host got an error for that one, so its span also carries an `acs.decision.failure` with class `guardian_error`. This Guardian verifies no signature, replay or timestamp yet (PR #229), so a forged or replayed request that passes the schema is still recorded as a step. The file holds one OTLP `ExportTraceServiceRequest` per line, the shape the Collector's file exporter writes, with `service.name` set to `acs-reference-guardian` (the converter below uses `acs-reference-audit-converter`, and both are labels, not attribution). Spans are batched and flushed when the Guardian closes on SIGINT or SIGTERM. A SIGKILL loses the last batch, and so does an output that has not finished three seconds into shutdown: the Guardian reports it on stderr and exits instead of waiting. [`otel-collector.yaml`](../../otel-collector.yaml) at the root of this tree receives the endpoint output and writes the same file shape. @@ -116,7 +120,7 @@ It prints how many spans reached the file and exits non-zero when that is fewer ### Sample trace -[`samples/trace/acs-trace-sample.jsonl`](../../samples/trace/acs-trace-sample.jsonl) is what this export looks like, so you can read it without running the Guardian. It holds six steps: an allowed tool call and its result, linked by `acs.request_id_ref`, a denied call carrying reasoning, reason codes, policy references and provenance, a call the Guardian answered with an error, a decision that finished after the timeout, and a step the host could not deliver, converted from its audit log. The decisions come from the stock policy, and the one late decision is delayed on purpose. Every value in it is synthetic. +[`samples/trace/acs-trace-sample.jsonl`](../../samples/trace/acs-trace-sample.jsonl) is what this export looks like, so you can read it without running the Guardian. It holds five steps and one rejected request: an allowed tool call and its result, linked by `acs.request_id_ref`, a denied call carrying reasoning, reason codes, policy references and provenance, a forged request that reuses the allowed call's ids and fails the schema, recorded as a `guardian.rejected_request` span so the allowed call keeps the only span under its pair, a decision that finished after the timeout, and a step the host could not deliver, converted from its audit log. The decisions come from the stock policy, and the one late decision is delayed on purpose. Every value in it is synthetic. ```bash bun run sample:trace diff --git a/reference-implementations/agt/packages/guardian/src/otel-mapping.ts b/reference-implementations/agt/packages/guardian/src/otel-mapping.ts index 05d9443..0d9b900 100644 --- a/reference-implementations/agt/packages/guardian/src/otel-mapping.ts +++ b/reference-implementations/agt/packages/guardian/src/otel-mapping.ts @@ -26,6 +26,11 @@ const PROVENANCE_COLUMN_LIMIT = 64; * earlier request, so #228 forbids a failure event for it: one would mark * that earlier, legitimate decision as not enforced. */ export const REPLAY_DETECTED_CODE = -32005; +/** JSON-RPC SIGNATURE_INVALID and TIMESTAMP_OUT_OF_WINDOW (Specification + * §17.1). Defined here beside REPLAY_DETECTED_CODE so the Guardian's checks + * and the trace's rejection mapping read one number for each. */ +export const SIGNATURE_INVALID_CODE = -32004; +export const TIMESTAMP_OUT_OF_WINDOW_CODE = -32006; // The repository's own mapping, five directories up from this file, the // same relationship SCHEMA_ROOT in validate-envelope.ts relies on. @@ -48,6 +53,46 @@ export const DECISION_FAILURE_EVENT = "acs.decision.failure"; * so a query on status text finds both sides of the same failure. */ export const GUARDIAN_ERROR_STATUS = "The Guardian returned an error instead of a decision."; +/** + * A request this Guardian cannot vouch for, recorded apart from the steps it + * governed. Trace Events forbids an acs.decision or acs.decision.failure + * under the pair of a request rejected before it was trusted, because such a + * request can carry any pair, a genuine request's included. Issue #283 ruled + * that it is still recorded, under names outside `step_to_span`, so a forged + * or replayed request stays visible without being attributed to the request + * whose ids it carries. These names belong to this reference Guardian, not + * to the ACS mapping. + */ +export const REJECTED_SPAN_NAME = "guardian.rejected_request"; +export const REJECTED_CLAIMED_SESSION_ID = "guardian.rejected.claimed_session_id"; +export const REJECTED_CLAIMED_REQUEST_ID = "guardian.rejected.claimed_request_id"; +export const REJECTED_REASON = "guardian.rejected.reason"; +export const REJECTED_ERROR_CODE = "guardian.rejected.error_code"; +export const REJECTED_STATUS = "The Guardian could not vouch for this request, so it carries no ACS ids or events."; + +/** Why a request was rejected before it was trusted. `guardian_fault` is the + * Guardian failing to check the request at all (a missing schema directory, + * say): its ids are just as unverified, so it is recorded the same way. */ +export type RejectionReason = + | "envelope_invalid" + | "guardian_fault" + | "signature_invalid" + | "replay_detected" + | "timestamp_out_of_window"; + +/** + * The §17.1 codes that mean the Guardian rejected a request before trusting + * it. dispatch reports its own rejections to the exporter directly. This is + * the backstop: a step answered with one of these codes is recorded as a + * rejected request even if the check that produced it forgot to report, so + * a rejected request's ids never land on a step span. + */ +export const PRE_TRUST_REJECTION_CODES: ReadonlyMap = new Map([ + [SIGNATURE_INVALID_CODE, "signature_invalid"], + [REPLAY_DETECTED_CODE, "replay_detected"], + [TIMESTAMP_OUT_OF_WINDOW_CODE, "timestamp_out_of_window"], +]); + /** The event's attribute keys: the three required ones first, then the * ones carried when known. */ export const FAILURE_ATTRIBUTES = { diff --git a/reference-implementations/agt/packages/guardian/src/otel-trace-exporter.ts b/reference-implementations/agt/packages/guardian/src/otel-trace-exporter.ts index b28bb85..be3b026 100644 --- a/reference-implementations/agt/packages/guardian/src/otel-trace-exporter.ts +++ b/reference-implementations/agt/packages/guardian/src/otel-trace-exporter.ts @@ -66,13 +66,20 @@ import { PROVENANCE_ATTRIBUTE_SOURCES, provenanceColumn, provenanceObjects, - REPLAY_DETECTED_CODE, + PRE_TRUST_REJECTION_CODES, + REJECTED_CLAIMED_REQUEST_ID, + REJECTED_CLAIMED_SESSION_ID, + REJECTED_ERROR_CODE, + REJECTED_REASON, + REJECTED_SPAN_NAME, + REJECTED_STATUS, requestParts, resultParts, STEP_ATTRIBUTE_SOURCES, type DecisionFailure, type FailureClass, type OtelMapping, + type RejectionReason, } from "./otel-mapping.ts"; import { createFileSpanExporter } from "./otlp-file-exporter.ts"; @@ -86,8 +93,10 @@ export { GUARDIAN_ERROR_STATUS, loadOtelMapping, METHODS_WITHOUT_SPAN, + REJECTED_SPAN_NAME, STEP_ATTRIBUTE_SOURCES, type DecisionFailure, + type RejectionReason, type FailureClass, type FailurePosture, type FailureReporter, @@ -149,16 +158,51 @@ export type StepSpan = { readonly method: string; readonly startedAt: number; readonly payloadBytes?: number; + /** The session and request id the request claimed. They go onto the span + * as acs.session.id and acs.request_id only when it ends as a trusted step + * (see `finish`), and under the rejected names otherwise. */ + readonly claimed: Attributes; + /** Every other step attribute, read from the request and held back the + * same way, so a span that ends as a rejected request carries none of + * them (`acs.request_id_ref` on a forged result names a genuine call). */ + readonly held: Attributes; + /** Set when the request was rejected before its span opened. */ + readonly rejected?: RejectionReason; }; +/** + * What `start` is told about the request besides its envelope. `trusted` is + * whether the Guardian accepted the envelope before acting on it. A request + * it did not accept is recorded as a rejected request, apart from the steps + * it governed, with `reason` saying why (see REJECTED_SPAN_NAME). `trusted` + * is required, not defaulted, so a caller that forgets it fails to compile + * rather than exporting an untrusted request's ids as a step. + * + * Trusted here means accepted by every check this Guardian runs. This + * reference Guardian checks the envelope against the schema and nothing + * else: it verifies no signature, replay or timestamp yet (PR #229), so a + * forgery or a replay that passes the schema is still recorded as a step. + * + * `receivedAt` is `performance.now()` when the request arrived, so the time + * the Guardian spent checking it counts toward a late decision. + */ +export type StepStart = { payloadBytes?: number; receivedAt?: number } & ( + | { trusted: true } + | { trusted: false; reason: RejectionReason } +); + /** The Guardian-side writer role, called at the envelope log's own two * points: `start` where the request is recorded, `end` where the response * is, with the handle `start` returned. */ export type OtelTraceExporter = { /** Null when this request produces no span, and `end` then does nothing. - * `payloadBytes` is the request body's size, for the failure event. */ - start(envelope: unknown, method: string | null, payloadBytes?: number): StepSpan | null; - end(stepSpan: StepSpan | null, response: unknown): void; + * `payloadBytes` is the request body's size, for the failure event. An + * untrusted request produces a rejected-request span (see StepStart). */ + start(envelope: unknown, method: string | null, step: StepStart): StepSpan | null; + /** `rejection` is set when the Guardian rejected the request before + * trusting it after the span opened (a signature, replay or timestamp + * check inside dispatch), whatever shape its answer took. */ + end(stepSpan: StepSpan | null, response: unknown, rejection?: RejectionReason): void; /** Pushes every ended span through to the exporters now rather than on * the batch timer. Tests call it; shutdown does too. */ flush(): Promise; @@ -279,6 +323,22 @@ function isolated({ label, exporter }: Output, report: (label: string, error: un }; } +/** A rejected request's attributes: the ids it claimed, under names a reader + * cannot mistake for the ACS pair, and why it was rejected. */ +function rejectedAttributes(claimed: Attributes, reason: RejectionReason, errorCode?: number): Attributes { + const attributes: Attributes = { [REJECTED_REASON]: reason }; + if (claimed["acs.session.id"] !== undefined) { + attributes[REJECTED_CLAIMED_SESSION_ID] = claimed["acs.session.id"]; + } + if (claimed["acs.request_id"] !== undefined) { + attributes[REJECTED_CLAIMED_REQUEST_ID] = claimed["acs.request_id"]; + } + if (errorCode !== undefined) { + attributes[REJECTED_ERROR_CODE] = errorCode; + } + return attributes; +} + export function createOtelTraceExporter({ filePath, endpoint, @@ -407,7 +467,32 @@ export function createOtelTraceExporter({ }); } - function begin(envelope: unknown, method: string | null, payloadBytes: number | undefined, tracer: Tracer, mapping: OtelMapping): StepSpan | null { + function begin(envelope: unknown, method: string | null, step: StepStart, tracer: Tracer, mapping: OtelMapping): StepSpan | null { + const parts = requestParts(envelope); + // Held back, not set: the pair, and every other step attribute, goes on + // only when the span ends as a trusted step, because a check that runs + // after this point can still reject the request. + const claimed: Attributes = {}; + collect(claimed, Object.keys(CORRELATION_ATTRIBUTE_SOURCES), CORRELATION_ATTRIBUTE_SOURCES, parts); + const startedAt = step.receivedAt ?? performance.now(); + // The span's own start is the arrival too, so its duration agrees with + // the elapsed time the failure event reports. + const startTime = step.receivedAt; + // A rejected request is recorded whatever its method, handled or not: + // its span needs no step_to_span entry. + if (!step.trusted) { + const rejected: StepSpan = { + span: tracer.startSpan(REJECTED_SPAN_NAME, { attributes: rejectedAttributes(claimed, step.reason), root: true, startTime }), + method: method ?? "", + startedAt, + payloadBytes: step.payloadBytes, + claimed, + held: {}, + rejected: step.reason, + }; + openSpans.add(rejected); + return rejected; + } if (method === null || !handled.has(method)) { return null; } @@ -415,9 +500,7 @@ export function createOtelTraceExporter({ if (entry === undefined) { return null; } - const parts = requestParts(envelope); const attributes: Attributes = {}; - collect(attributes, Object.keys(CORRELATION_ATTRIBUTE_SOURCES), CORRELATION_ATTRIBUTE_SOURCES, parts); collect( attributes, [...entry.required_attributes, ...(entry.conditional_attributes ?? []), ...(entry.optional_attributes ?? [])], @@ -437,17 +520,46 @@ export function createOtelTraceExporter({ } } const stepSpan: StepSpan = { - span: tracer.startSpan(entry.span_name, { attributes, root: true }), + span: tracer.startSpan(entry.span_name, { root: true, startTime }), method, - startedAt: performance.now(), - payloadBytes, + startedAt, + payloadBytes: step.payloadBytes, + claimed, + held: attributes, }; openSpans.add(stepSpan); return stepSpan; } - function finish(stepSpan: StepSpan, envelope: unknown, mapping: OtelMapping): void { + /** Ends a span as a rejected request: renamed if it opened as a step, the + * claimed ids under the rejected names, and no ACS pair, step attribute or + * decision event. A Guardian fault is still a decision failure for the + * host, which got an error, so it keeps that event, on a span with no pair. */ + function finishRejected(stepSpan: StepSpan, reason: RejectionReason, errorCode?: number): void { + const { span } = stepSpan; + span.updateName(REJECTED_SPAN_NAME); + span.setAttributes(rejectedAttributes(stepSpan.claimed, reason, errorCode)); + span.setStatus({ code: SpanStatusCode.ERROR, message: REJECTED_STATUS }); + if (reason === "guardian_fault") { + span.addEvent(DECISION_FAILURE_EVENT, failureAttributes(stepSpan, "guardian_error", { errorCode })); + } + span.end(); + } + + function finish(stepSpan: StepSpan, envelope: unknown, mapping: OtelMapping, rejection: RejectionReason | undefined): void { const { span } = stepSpan; + const error = asObject(asObject(envelope)?.error); + const errorCode = asNumber(error?.code); + // Rejected before the span opened, rejected by a check after it opened, + // or answered with a code that only a rejection carries: each ends as a + // rejected request, and no held attribute reaches it. + const reason = + stepSpan.rejected ?? rejection ?? (errorCode === undefined ? undefined : PRE_TRUST_REJECTION_CODES.get(errorCode)); + if (reason !== undefined) { + finishRejected(stepSpan, reason, errorCode); + return; + } + span.setAttributes({ ...stepSpan.claimed, ...stepSpan.held }); const decision = resultParts(envelope); if (decision === undefined) { // A JSON-RPC error: the step was answered but not decided. The first @@ -455,12 +567,8 @@ export function createOtelTraceExporter({ // carried: the message can quote the request (a schema error names its // argument keys), and #228 forbids copying response or error bytes into // the trace, so the status message is a fixed sentence too. - const error = asObject(asObject(envelope)?.error); - const errorCode = asNumber(error?.code); span.setStatus({ code: SpanStatusCode.ERROR, message: GUARDIAN_ERROR_STATUS }); - if (errorCode !== REPLAY_DETECTED_CODE) { - span.addEvent(DECISION_FAILURE_EVENT, failureAttributes(stepSpan, "guardian_error", { errorCode })); - } + span.addEvent(DECISION_FAILURE_EVENT, failureAttributes(stepSpan, "guardian_error", { errorCode })); } else { const attributes: Attributes = {}; collect(attributes, new Set([...mapping.decisionAttributes, ...DECISION_DETAIL_NAMES]), DECISION_ATTRIBUTE_SOURCES, decision); @@ -484,18 +592,18 @@ export function createOtelTraceExporter({ let shutdownPromise: Promise | undefined; return { - start(envelope, method, payloadBytes): StepSpan | null { + start(envelope, method, step): StepSpan | null { if (disabled || stopped || tracer === undefined || mapping === undefined) { return null; } try { - return begin(envelope, method, payloadBytes, tracer, mapping); + return begin(envelope, method, step, tracer, mapping); } catch (error) { fail(error); return null; } }, - end(stepSpan, response): void { + end(stepSpan, response, rejection): void { if (stepSpan === null) { return; } @@ -506,7 +614,7 @@ export function createOtelTraceExporter({ return; } try { - finish(stepSpan, response, mapping); + finish(stepSpan, response, mapping, rejection); } catch (error) { fail(error); } @@ -532,6 +640,11 @@ export function createOtelTraceExporter({ // so the step is exported at all. for (const stepSpan of openSpans) { try { + if (stepSpan.rejected !== undefined) { + finishRejected(stepSpan, stepSpan.rejected); + continue; + } + stepSpan.span.setAttributes({ ...stepSpan.claimed, ...stepSpan.held }); stepSpan.span.addEvent( DECISION_FAILURE_EVENT, failureAttributes(stepSpan, "other", { detail: "the Guardian shut down before it answered this step" }), diff --git a/reference-implementations/agt/packages/guardian/src/server.ts b/reference-implementations/agt/packages/guardian/src/server.ts index 922c11b..c00aeac 100644 --- a/reference-implementations/agt/packages/guardian/src/server.ts +++ b/reference-implementations/agt/packages/guardian/src/server.ts @@ -94,6 +94,7 @@ import { checkResponse } from "./check-response.ts"; import { buildServerHello, type ServerHello } from "./handshake.ts"; import { createEnvelopeLogSink, NULL_ENVELOPE_LOG_SINK, type EnvelopeLogSink } from "./envelope-log-sink.ts"; import { createOtelTraceExporter, NULL_OTEL_TRACE_EXPORTER, type OtelTraceExporter } from "./otel-trace-exporter.ts"; +import type { RejectionReason } from "./otel-mapping.ts"; import { appendContextEntry, createMemorySessionContextStore, @@ -568,13 +569,30 @@ async function handleAcsRequest( // Recorded before validation, so an envelope that fails the schema is // visible to the Inspector rather than invisible. + const receivedAt = performance.now(); const method = extractMethod(raw); envelopeLog.write("request", raw, method); - const stepSpan = traceExporter.start(raw, method, Buffer.byteLength(body.text)); + // Validated once, here, before the span opens, because the span has to + // know whether the Guardian trusts this request: Trace Events forbids an + // acs.decision under the ids of a request rejected before it was trusted. + // A rejected request is recorded apart, as a rejected-request span (issue + // #283). dispatch acts on the same result rather than validating again. A + // signature, replay or timestamp rejection answered from inside dispatch + // is caught by its error code when the span ends (PRE_TRUST_REJECTION_CODES). + const validation = validateForDispatch(raw); + const payloadBytes = Buffer.byteLength(body.text); + const stepSpan = traceExporter.start( + raw, + method, + validation.kind === "valid" + ? { trusted: true, payloadBytes, receivedAt } + : { trusted: false, reason: validation.kind === "invalid" ? "envelope_invalid" : "guardian_fault", payloadBytes, receivedAt }, + ); let response: JsonRpcSuccess | JsonRpcFailure; + let rejection: RejectionReason | undefined; try { - response = await dispatch(raw, bridge, mapping, onDecisionFailure, sessionContextStore); + ({ response, rejection } = await dispatch(raw, validation, bridge, mapping, onDecisionFailure, sessionContextStore)); } catch (error) { // The outer net. Deliberately a bare JSON-RPC error, not an ACS `deny`: // this route is `dispatch` rethrowing entirely past denyOnInvalidEnvelope @@ -616,7 +634,7 @@ async function handleAcsRequest( } envelopeLog.write("response", response, method); - traceExporter.end(stepSpan, response); + traceExporter.end(stepSpan, response, rejection); return response; } @@ -712,44 +730,72 @@ async function readCappedBody(req: Request): Promise { : { withinLimit: true, text: text + decoder.decode() }; } +/** + * validateEnvelope's outcome, held rather than thrown so handleAcsRequest can + * read it before the trace span opens and dispatch can act on it after. + * `failed` is anything validateEnvelope threw that is not a validation + * verdict (a missing schema directory, say). dispatch rethrows it, so it + * still reaches the outer net exactly as it did when dispatch validated. + */ +type Validation = + | { kind: "valid"; envelope: AcsRequestEnvelope } + | { kind: "invalid"; error: EnvelopeValidationError } + | { kind: "failed"; error: unknown }; + +function validateForDispatch(raw: unknown): Validation { + try { + return { kind: "valid", envelope: validateEnvelope(raw) }; + } catch (error) { + return error instanceof EnvelopeValidationError ? { kind: "invalid", error } : { kind: "failed", error }; + } +} + +/** + * What dispatch answered, and whether it rejected the request before trusting + * it. `rejection` is how a check inside dispatch tells the trace exporter so, + * whatever the response looks like: a signature, replay or timestamp check + * (PR #229) answering with an error, or with an honoured deny the way an + * invalid envelope is answered. The exporter then records the request apart + * from the steps instead of under the ids it claimed. + */ +type Dispatched = { response: JsonRpcSuccess | JsonRpcFailure; rejection?: RejectionReason }; + async function dispatch( raw: unknown, + validation: Validation, bridge: PolicyBridge, mapping: Mapping, onDecisionFailure: "proceed" | "deny" | undefined, sessionContextStore: SessionContextStore, -): Promise { +): Promise { const rpcId = extractId(raw); - let envelope: AcsRequestEnvelope; - try { - // validateEnvelope checks the general request-envelope.json shape for - // every method, plus -- only for steps/toolCallRequest -- the - // hook-specific payload schema. Failure is a THROWN typed error, never - // a decision itself -- validateEnvelope stays total to its own contract - // (see its doc comment) -- but the catch below (denyOnInvalidEnvelope) - // turns a steps/* failure into an honoured ACS `deny` decision rather - // than a bare JSON-RPC error, since there is an identifiable step to - // answer for. A handshake failure and an undispatched method are not - // steps/*, so they always fall through to the JSON-RPC error unchanged. - envelope = validateEnvelope(raw); - } catch (error) { - if (error instanceof EnvelopeValidationError) { - if (isStepMethod(raw)) { - const denial = denyOnInvalidEnvelope(raw, { reasonCode: "envelope_invalid", message: error.message }); - const decisionResponse = asDecisionResponse(rpcId, denial); - if (decisionResponse) { - return decisionResponse; - } + // validateEnvelope checked the general request-envelope.json shape for + // every method, plus -- only for steps/toolCallRequest -- the hook-specific + // payload schema (validateForDispatch, above). An invalid steps/* envelope + // becomes an honoured ACS `deny` decision rather than a bare JSON-RPC error, + // since there is an identifiable step to answer for. A handshake failure + // and an undispatched method are not steps/*, so they always fall through + // to the JSON-RPC error unchanged. + if (validation.kind === "failed") { + throw validation.error; + } + if (validation.kind === "invalid") { + const { error } = validation; + if (isStepMethod(raw)) { + const denial = denyOnInvalidEnvelope(raw, { reasonCode: "envelope_invalid", message: error.message }); + const decisionResponse = asDecisionResponse(rpcId, denial); + if (decisionResponse) { + return { response: decisionResponse, rejection: "envelope_invalid" }; } - return errorResponse(rpcId, ENVELOPE_INVALID_CODE, error.message, { pointer: error.pointer }); } - throw error; + return { response: errorResponse(rpcId, ENVELOPE_INVALID_CODE, error.message, { pointer: error.pointer }), rejection: "envelope_invalid" }; } + const envelope = validation.envelope; if (envelope.method === HANDSHAKE_METHOD) { const env = onDecisionFailure === undefined ? undefined : { ACS_ON_DECISION_FAILURE: onDecisionFailure }; - return successResponse(envelope.id, buildServerHello(env)); + return { response: successResponse(envelope.id, buildServerHello(env)) }; } // Two gated branches, one per ACS method this Guardian assembles a snapshot @@ -773,21 +819,21 @@ async function dispatch( // shape. Anything neither predicate answers falls past both, to // METHOD_NOT_DISPATCHED_CODE below. if (isToolCallRequest(envelope)) { - return await evaluateStep(raw, envelope, assemblePreToolCallSnapshot, bridge, mapping, sessionContextStore); + return { response: await evaluateStep(raw, envelope, assemblePreToolCallSnapshot, bridge, mapping, sessionContextStore) }; } // The result gate, beside the request gate rather than merged into it. if (isToolCallResult(envelope)) { - return await evaluateStep(raw, envelope, assemblePostToolCallSnapshot, bridge, mapping, sessionContextStore); + return { response: await evaluateStep(raw, envelope, assemblePostToolCallSnapshot, bridge, mapping, sessionContextStore) }; } // A well-formed envelope (it passed validateEnvelope: the method matched // the required prefix pattern) naming a method this Guardian has no // handler for -- distinct from a malformed envelope, which never reaches // here because validateEnvelope already threw above. - return errorResponse(rpcId, METHOD_NOT_DISPATCHED_CODE, `method not dispatched by this Guardian: ${envelope.method}`, { + return { response: errorResponse(rpcId, METHOD_NOT_DISPATCHED_CODE, `method not dispatched by this Guardian: ${envelope.method}`, { method: envelope.method, - }); + }) }; } /** diff --git a/reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts b/reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts index 0d69ca9..4908cdc 100644 --- a/reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts +++ b/reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts @@ -20,11 +20,15 @@ import { METHODS_WITHOUT_SPAN, NULL_OTEL_TRACE_EXPORTER, OTEL_SERVICE_NAME, + REJECTED_SPAN_NAME, STEP_ATTRIBUTE_SOURCES, type CreateOtelTraceExporterOptions, type OtelTraceExporter, } from "../src/otel-trace-exporter.ts"; +/** What the Guardian tells `start` about a request it accepted. */ +const TRUSTED = { trusted: true } as const; + /** The live mapping, read the same way the exporter reads it, so every * expected name below comes from the file and not from a literal that could * drift from it. */ @@ -114,7 +118,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { const requestId = crypto.randomUUID(); const request = toolCallEnvelope("rm -rf /", { id: 7, requestId, sessionId }); - const step = exporter.start(request, "steps/toolCallRequest"); + const step = exporter.start(request, "steps/toolCallRequest", TRUSTED); exporter.end(step, decisionResponse(7, DENY)); await exporter.flush(); @@ -132,7 +136,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { it("records the decision as the mapping's event on that span, with acs.decision and no evaluator the response did not send", async () => { const { exporter, spans } = memoryExporter(); - exporter.end(exporter.start(toolCallEnvelope("rm -rf /", { id: 7 }), "steps/toolCallRequest"), decisionResponse(7, DENY)); + exporter.end(exporter.start(toolCallEnvelope("rm -rf /", { id: 7 }), "steps/toolCallRequest", TRUSTED), decisionResponse(7, DENY)); await exporter.flush(); const [event] = spans()[0]?.events ?? []; @@ -153,7 +157,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { it("takes acs.evaluator from the response when it carries one", async () => { const { exporter, spans } = memoryExporter(); exporter.end( - exporter.start(toolCallEnvelope("ls", { id: 7 }), "steps/toolCallRequest"), + exporter.start(toolCallEnvelope("ls", { id: 7 }), "steps/toolCallRequest", TRUSTED), decisionResponse(7, { decision: "allow", metadata: { evaluator: "agent", model_id: "m-1" } }), ); await exporter.flush(); @@ -166,7 +170,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { it("lines up policy columns in wire order, with \"\" where a reference lacks the field", async () => { const { exporter, spans } = memoryExporter(); exporter.end( - exporter.start(toolCallEnvelope("ls", { id: 7 }), "steps/toolCallRequest"), + exporter.start(toolCallEnvelope("ls", { id: 7 }), "steps/toolCallRequest", TRUSTED), decisionResponse(7, { decision: "deny", reasoning: "x", @@ -187,7 +191,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { it("pads a missing rule_id with \"\" in its own position", async () => { const { exporter, spans } = memoryExporter(); exporter.end( - exporter.start(toolCallEnvelope("ls", { id: 7 }), "steps/toolCallRequest"), + exporter.start(toolCallEnvelope("ls", { id: 7 }), "steps/toolCallRequest", TRUSTED), decisionResponse(7, { decision: "deny", reasoning: "x", @@ -210,7 +214,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { it("emits all three policy columns at the length of policy_references", async () => { const { exporter, spans } = memoryExporter(); exporter.end( - exporter.start(toolCallEnvelope("ls", { id: 7 }), "steps/toolCallRequest"), + exporter.start(toolCallEnvelope("ls", { id: 7 }), "steps/toolCallRequest", TRUSTED), decisionResponse(7, { decision: "deny", reasoning: "x", @@ -227,8 +231,8 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { it("carries acs.chain_hash only when the response does", async () => { const { exporter, spans } = memoryExporter(); - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest"), decisionResponse(1, { decision: "allow", chain_hash: "a".repeat(64) })); - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 2 }), "steps/toolCallRequest"), decisionResponse(2, { decision: "allow" })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest", TRUSTED), decisionResponse(1, { decision: "allow", chain_hash: "a".repeat(64) })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 2 }), "steps/toolCallRequest", TRUSTED), decisionResponse(2, { decision: "allow" })); await exporter.flush(); const [withHash, withoutHash] = spans(); @@ -240,8 +244,8 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { const { exporter, spans } = memoryExporter(); // handshake/hello has no step_to_span entry. steps/sessionStart has one, // and is not in the ServerHello's methods_evaluated. - const hello = exporter.start(makeEnvelope("handshake/hello", {}, { id: 1 }), "handshake/hello"); - const sessionStart = exporter.start(makeEnvelope("steps/sessionStart", {}, { id: 2 }), "steps/sessionStart"); + const hello = exporter.start(makeEnvelope("handshake/hello", {}, { id: 1 }), "handshake/hello", TRUSTED); + const sessionStart = exporter.start(makeEnvelope("steps/sessionStart", {}, { id: 2 }), "steps/sessionStart", TRUSTED); expect(hello).toBeNull(); expect(sessionStart).toBeNull(); exporter.end(hello, { jsonrpc: "2.0", id: 1, result: { negotiated_version: "0.1.0" } }); @@ -257,7 +261,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { it("ends a span answered with a JSON-RPC error in error status, with a guardian_error failure event and no decision event", async () => { const { exporter, spans } = memoryExporter(); exporter.end( - exporter.start(toolCallEnvelope("ls", { id: 9 }), "steps/toolCallRequest"), + exporter.start(toolCallEnvelope("ls", { id: 9 }), "steps/toolCallRequest", TRUSTED), { jsonrpc: "2.0", id: 9, error: { code: -32020, message: "evaluation failed" } }, ); await exporter.flush(); @@ -276,15 +280,20 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { }); // The pairing is the handle, not the id: an envelope with no JSON-RPC id - // at all still gets its span, closed by its own response. + // at all still gets its span, closed by its own response. The schema + // requires an id, so such an envelope is always one the Guardian rejected. it("pairs request and response by handle, so a request with no JSON-RPC id still gets its span", async () => { const { exporter, spans } = memoryExporter(); - const step = exporter.start(toolCallEnvelope("ls", { id: null }), "steps/toolCallRequest"); + const step = exporter.start(toolCallEnvelope("ls", { id: null }), "steps/toolCallRequest", { + trusted: false, + reason: "envelope_invalid", + }); expect(step).not.toBeNull(); exporter.end(step, { jsonrpc: "2.0", id: null, error: { code: -32010, message: "invalid" } }); await exporter.flush(); expect(spans()).toHaveLength(1); + expect(spans()[0]?.name).toBe(REJECTED_SPAN_NAME); expect(spans()[0]?.status.code).toBe(SpanStatusCode.ERROR); }); @@ -298,10 +307,10 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { onError: (error) => errors.push(error), }); - const step = exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest"); + const step = exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest", TRUSTED); expect(() => exporter.end(step, decisionResponse(1, DENY))).not.toThrow(); await exporter.flush(); - expect(() => exporter.start(toolCallEnvelope("ls", { id: 2 }), "steps/toolCallRequest")).not.toThrow(); + expect(() => exporter.start(toolCallEnvelope("ls", { id: 2 }), "steps/toolCallRequest", TRUSTED)).not.toThrow(); await exporter.flush(); await exporter.shutdown(); @@ -321,7 +330,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { methodsEvaluated: METHODS_EVALUATED, onError: (error) => errors.push(error), }); - expect(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest")).toBeNull(); + expect(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest", TRUSTED)).toBeNull(); await exporter.shutdown(); expect(errors).toHaveLength(1); } finally { @@ -346,7 +355,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { }); try { for (const id of [1, 2, 3]) { - exporter.end(exporter.start(toolCallEnvelope("ls", { id }), "steps/toolCallRequest"), decisionResponse(id, { decision: "allow" })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id }), "steps/toolCallRequest", TRUSTED), decisionResponse(id, { decision: "allow" })); await exporter.flush(); } await exporter.shutdown(); @@ -376,7 +385,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { }); const exporter = createOtelTraceExporter({ endpoint: `http://127.0.0.1:${collector.port}/v1/traces`, methodsEvaluated: METHODS_EVALUATED }); try { - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest"), decisionResponse(1, { decision: "allow" })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest", TRUSTED), decisionResponse(1, { decision: "allow" })); await exporter.flush(); await exporter.shutdown(); @@ -403,7 +412,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { }, { id: 1 }, ); - exporter.end(exporter.start(request, "steps/toolCallRequest"), decisionResponse(1, { decision: "allow" })); + exporter.end(exporter.start(request, "steps/toolCallRequest", TRUSTED), decisionResponse(1, { decision: "allow" })); await exporter.flush(); const attributes = spans()[0]?.attributes; @@ -424,7 +433,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { }, { id: 1 }, ); - exporter.end(exporter.start(request, "steps/toolCallRequest"), decisionResponse(1, { decision: "allow" })); + exporter.end(exporter.start(request, "steps/toolCallRequest", TRUSTED), decisionResponse(1, { decision: "allow" })); await exporter.flush(); const attributes = spans()[0]?.attributes; @@ -434,7 +443,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { it("carries no provenance attribute when nothing in the payload has one", async () => { const { exporter, spans } = memoryExporter(); - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest"), decisionResponse(1, { decision: "allow" })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest", TRUSTED), decisionResponse(1, { decision: "allow" })); await exporter.flush(); const attributes = spans()[0]?.attributes ?? {}; @@ -452,7 +461,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { }, { id: 1 }, ); - exporter.end(exporter.start(request, "steps/toolCallResult"), decisionResponse(1, { decision: "allow" })); + exporter.end(exporter.start(request, "steps/toolCallResult", TRUSTED), decisionResponse(1, { decision: "allow" })); await exporter.flush(); const [span] = spans(); @@ -470,8 +479,8 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { const { exporter, spans } = memoryExporter(); const first = crypto.randomUUID(); const second = crypto.randomUUID(); - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1, requestId: first }), "steps/toolCallRequest"), decisionResponse(1, { decision: "allow" })); - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 2, requestId: second }), "steps/toolCallRequest"), decisionResponse(2, { decision: "allow" })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1, requestId: first }), "steps/toolCallRequest", TRUSTED), decisionResponse(1, { decision: "allow" })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 2, requestId: second }), "steps/toolCallRequest", TRUSTED), decisionResponse(2, { decision: "allow" })); await exporter.flush(); const [spanA, spanB] = spans(); try { @@ -501,7 +510,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { const path = join(nested, "traces.jsonl"); const exporter = createOtelTraceExporter({ filePath: path, methodsEvaluated: METHODS_EVALUATED }); try { - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest"), decisionResponse(1, { decision: "allow" })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest", TRUSTED), decisionResponse(1, { decision: "allow" })); await exporter.shutdown(); expect((statSync(nested).mode & 0o777).toString(8)).toBe("700"); @@ -528,7 +537,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { }, }; const exporter = createOtelTraceExporter({ spanExporter: slow, methodsEvaluated: METHODS_EVALUATED }); - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest"), decisionResponse(1, { decision: "allow" })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest", TRUSTED), decisionResponse(1, { decision: "allow" })); const first = exporter.shutdown(); const second = exporter.shutdown(); @@ -571,7 +580,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { ); try { const { exporter, spans } = memoryExporter({ mappingPath }); - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest"), decisionResponse(1, { decision: "allow" })); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 1 }), "steps/toolCallRequest", TRUSTED), decisionResponse(1, { decision: "allow" })); await exporter.flush(); const [span] = spans(); @@ -613,7 +622,7 @@ describe("createOtelTraceExporter -- what one exchange becomes", () => { }); it("NULL_OTEL_TRACE_EXPORTER writes nothing and never throws", async () => { - expect(NULL_OTEL_TRACE_EXPORTER.start(toolCallEnvelope("ls"), "steps/toolCallRequest")).toBeNull(); + expect(NULL_OTEL_TRACE_EXPORTER.start(toolCallEnvelope("ls"), "steps/toolCallRequest", TRUSTED)).toBeNull(); expect(() => NULL_OTEL_TRACE_EXPORTER.end(null, decisionResponse(1, DENY))).not.toThrow(); await NULL_OTEL_TRACE_EXPORTER.flush(); await NULL_OTEL_TRACE_EXPORTER.shutdown(); @@ -742,24 +751,95 @@ describe("Guardian trace export wiring", () => { } }); - // The span opens before validation, so the envelope that fails the schema - // is visible in the trace too -- as a deny, and with no chain hash, because - // the step never reached the chain. - it("gives a schema-invalid steps/* envelope a span with a deny event and no acs.chain_hash", async () => { + // Trace Events (PR #273): a Guardian MUST NOT emit acs.decision, or + // acs.decision.failure, under the pair of a request it rejects before + // trusting it, because such a request can carry any pair, a genuine + // request's included. Issue #283 ruled that it is still recorded, apart, + // under names outside step_to_span, so the attack stays visible. + it("records a schema-invalid steps/* envelope as a rejected request, and still answers it with a deny", async () => { const { exporter, spans } = memoryExporter(); const guardian = await startGuardian({ ...baseOptions, traceExporter: exporter }); try { const bad = toolCallEnvelope("rm -rf /", { id: 12 }); delete (bad.params as Record).acs_version; - await postRaw(guardian.url, bad); + const answer = await postRaw(guardian.url, bad); await exporter.flush(); + // The host still gets its honoured deny. Only the trace changes. + expect((answer.result as { decision?: string } | undefined)?.decision).toBe("deny"); const [span] = spans(); expect(spans()).toHaveLength(1); - const attributes = span?.events[0]?.attributes; - expect(attributes?.["acs.decision"]).toBe("deny"); - expect(attributes?.["acs.reason_codes"]).toEqual(["envelope_invalid"]); - expect(attributes?.["acs.chain_hash"]).toBeUndefined(); + expect(span?.name).toBe(REJECTED_SPAN_NAME); + expect(span?.attributes["acs.session.id"]).toBeUndefined(); + expect(span?.attributes["acs.request_id"]).toBeUndefined(); + expect(span?.attributes["guardian.rejected.reason"]).toBe("envelope_invalid"); + expect(span?.events).toEqual([]); + } finally { + await guardian.close(); + } + }); + + // The forgery here fails the schema. One that passes it is still recorded + // as a step: this Guardian verifies no signature, replay or timestamp yet. + for (const order of ["genuine first", "forgery first"] as const) { + it(`keeps a genuine request the only span under its pair when a schema-invalid request forges it (${order})`, async () => { + const { exporter, spans } = memoryExporter(); + const guardian = await startGuardian({ ...baseOptions, traceExporter: exporter }); + try { + const requestId = crypto.randomUUID(); + const sessionId = crypto.randomUUID(); + const genuine = toolCallEnvelope("ls -la", { id: 1, requestId, sessionId }); + const forged = toolCallEnvelope("rm -rf /", { id: 2, requestId, sessionId }); + delete (forged.params as Record).acs_version; + for (const envelope of order === "genuine first" ? [genuine, forged] : [forged, genuine]) { + await postRaw(guardian.url, envelope); + } + await exporter.flush(); + + const underPair = spans().filter( + (span) => span.attributes["acs.request_id"] === requestId && span.attributes["acs.session.id"] === sessionId, + ); + expect(underPair).toHaveLength(1); + expect(underPair[0]?.events.map((event) => event.attributes?.["acs.decision"])).toEqual(["allow"]); + const rejected = spans().filter((span) => span.name === REJECTED_SPAN_NAME); + expect(rejected).toHaveLength(1); + expect(rejected[0]?.attributes["guardian.rejected.claimed_request_id"]).toBe(requestId); + } finally { + await guardian.close(); + } + }); + } + + it("records a schema-invalid envelope for a method it does not dispatch as a rejected request too", async () => { + const { exporter, spans } = memoryExporter(); + const guardian = await startGuardian({ ...baseOptions, traceExporter: exporter }); + try { + const bad = makeEnvelope("steps/sessionStart", {}, { id: 44 }); + delete (bad.params as Record).acs_version; + await postRaw(guardian.url, bad); + await exporter.flush(); + + expect(spans().map((span) => span.name)).toEqual([REJECTED_SPAN_NAME]); + } finally { + await guardian.close(); + } + }); + + it("records an invalid envelope answered with a bare error as a rejected request, with no decision failure", async () => { + const { exporter, spans } = memoryExporter(); + const guardian = await startGuardian({ ...baseOptions, traceExporter: exporter }); + try { + // No request_id and no JSON-RPC id: denyOnInvalidEnvelope cannot address + // a decision, so the Guardian answers with a JSON-RPC error instead. + const bad = toolCallEnvelope("ls", { id: null }); + delete (bad.params as Record).request_id; + await postRaw(guardian.url, bad); + await exporter.flush(); + + const [span] = spans(); + expect(span?.name).toBe(REJECTED_SPAN_NAME); + expect(span?.events).toEqual([]); + expect(span?.attributes["guardian.rejected.error_code"]).toBe(-32010); } finally { await guardian.close(); } @@ -954,31 +1034,38 @@ describe("Guardian trace export wiring", () => { }, 15_000); // server.ts builds the exporter from the ServerHello, so the posture on a - // failure event is the one this Guardian declared to the host. An envelope - // with neither an id nor a request_id earns a deny that cannot be - // addressed to it, so the Guardian answers a bare JSON-RPC error: a - // guardian_error on the span, through the file path that production uses. + // failure event is the one this Guardian declared to the host. The failure + // here is a decision that finished after the ServerHello's own timeout, on + // a request the Guardian trusted, through the file path production uses. + // An invalid envelope would not do: a request rejected before it is + // trusted gets no failure event at all (see the schema-invalid case above). it("reports the ServerHello's posture on a failure event: a Guardian declaring deny says fail_closed", async () => { const dir = mkdtempSync(join(tmpdir(), "acs-otel-posture-")); const path = join(dir, "traces.jsonl"); - const guardian = await startGuardian({ ...baseOptions, onDecisionFailure: "deny", otelTraceFile: path }); + const late: PolicyBridge = { + async evaluate() { + await Bun.sleep(buildServerHello().timeout_config.default_ms + 200); + return { decision: "allow" }; + }, + }; + const guardian = await startGuardian({ ...baseOptions, bridge: late, onDecisionFailure: "deny", otelTraceFile: path }); try { - const response = await postRaw(guardian.url, { jsonrpc: "2.0", method: "steps/toolCallRequest", params: {} }); - expect(response.error).toBeDefined(); + const response = await postRaw(guardian.url, toolCallEnvelope("ls -la", { id: 1 })); + expect(response.result).toBeDefined(); } finally { await guardian.close(); } try { const [span] = otlpSpans(otlpFile(path)); const failure = span?.events?.find((event) => event.name === "acs.decision.failure"); - expect(otlpAttribute(failure, "acs.failure.class")).toBe("guardian_error"); + expect(otlpAttribute(failure, "acs.failure.class")).toBe("timeout"); expect(otlpAttribute(failure, "acs.failure.posture")).toBe("fail_closed"); expect(otlpAttribute(failure, "acs.failure.reported_by")).toBe("guardian"); } finally { unlinkSync(path); rmdirSync(dir); } - }); + }, 15_000); // A collector that is down must cost the endpoint and nothing else. The // real OTLPTraceExporter against a closed port, not the throwing stub: a @@ -1094,7 +1181,7 @@ describe("createOtelTraceExporter -- review fixes", () => { // to the same rule, since it lands in the same backend. it("copies no JSON-RPC error text into the failure event or the span status, and keeps the code", async () => { const { exporter, spans } = memoryExporter(); - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 3 }), "steps/toolCallRequest"), { + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 3 }), "steps/toolCallRequest", TRUSTED), { jsonrpc: "2.0", id: 3, error: { code: -32020, message: `evaluation failed: ${SENTINEL}` }, @@ -1108,18 +1195,110 @@ describe("createOtelTraceExporter -- review fixes", () => { expect(span?.events[0]?.attributes?.["acs.failure.detail"]).toBeUndefined(); }); - // #228: a Guardian MUST NOT emit the failure event for a request it rejects - // as a replay, because that pair belongs to the earlier request. - it("emits no failure event for REPLAY_DETECTED (-32005)", async () => { + // #228 and PR #273: a request rejected before it was trusted gets no + // acs.decision.failure under its pair, because the pair may be a genuine + // request's. That holds even when the check ran after the span opened, as + // a signature, replay or timestamp check inside dispatch would: the span + // ends as a rejected request, renamed and without the pair. + for (const [code, reason] of [ + [-32004, "signature_invalid"], + [-32005, "replay_detected"], + [-32006, "timestamp_out_of_window"], + ] as const) { + it(`records a step answered with ${code} as a rejected request, without its pair`, async () => { + const { exporter, spans } = memoryExporter(); + const requestId = crypto.randomUUID(); + exporter.end(exporter.start(toolCallEnvelope("ls", { id: 4, requestId }), "steps/toolCallRequest", TRUSTED), { + jsonrpc: "2.0", + id: 4, + error: { code, message: "rejected" }, + }); + await exporter.flush(); + + const [span] = spans(); + expect(span?.name).toBe(REJECTED_SPAN_NAME); + expect(span?.events).toEqual([]); + expect(span?.attributes["acs.request_id"]).toBeUndefined(); + expect(span?.attributes["acs.session.id"]).toBeUndefined(); + expect(span?.attributes["guardian.rejected.claimed_request_id"]).toBe(requestId); + expect(span?.attributes["guardian.rejected.reason"]).toBe(reason); + expect(span?.attributes["guardian.rejected.error_code"]).toBe(code); + }); + } + + it("records a request the Guardian did not trust as a rejected request, with the reason it was given", async () => { const { exporter, spans } = memoryExporter(); - exporter.end(exporter.start(toolCallEnvelope("ls", { id: 4 }), "steps/toolCallRequest"), { + const requestId = crypto.randomUUID(); + const handle = exporter.start(toolCallEnvelope("ls", { id: 5, requestId }), "steps/toolCallRequest", { + trusted: false, + reason: "guardian_fault", + }); + expect(handle).not.toBeNull(); + exporter.end(handle, { jsonrpc: "2.0", id: 5, error: { code: -32020, message: "schemas missing" } }); + await exporter.flush(); + + const [span] = spans(); + expect(span?.name).toBe(REJECTED_SPAN_NAME); + expect(span?.attributes["acs.request_id"]).toBeUndefined(); + expect(span?.attributes["guardian.rejected.claimed_request_id"]).toBe(requestId); + expect(span?.attributes["guardian.rejected.reason"]).toBe("guardian_fault"); + expect(span?.status.code).toBe(SpanStatusCode.ERROR); + // The host got an error, so the step went ungoverned: still a decision + // failure, recorded on a span and event that carry no pair. + expect(span?.events.map((event) => event.attributes?.["acs.failure.class"])).toEqual(["guardian_error"]); + expect(span?.events[0]?.attributes?.["acs.request_id"]).toBeUndefined(); + }); + + it("carries none of a rejected result's step attributes, so acs.request_id_ref cannot name the genuine call", async () => { + const { exporter, spans } = memoryExporter(); + const genuineCall = crypto.randomUUID(); + const forgedResult = makeEnvelope( + "steps/toolCallResult", + { tool: { name: "run_shell" }, exit_status: "success", outputs: [{ value: "ok" }], request_id_ref: genuineCall }, + { id: 8 }, + ); + exporter.end(exporter.start(forgedResult, "steps/toolCallResult", TRUSTED), { jsonrpc: "2.0", - id: 4, + id: 8, error: { code: -32005, message: "replay" }, }); await exporter.flush(); - expect(spans()[0]?.events.map((event) => event.name)).toEqual([]); + const [span] = spans(); + expect(span?.name).toBe(REJECTED_SPAN_NAME); + expect(span?.attributes["acs.request_id_ref"]).toBeUndefined(); + expect(span?.attributes["gen_ai.tool.name"]).toBeUndefined(); + expect(Object.keys(span?.attributes ?? {}).every((key) => key.startsWith("guardian.rejected."))).toBe(true); + }); + + it("records a step dispatch rejected as a rejected request even when it answered with a deny", async () => { + const { exporter, spans } = memoryExporter(); + const handle = exporter.start(toolCallEnvelope("ls", { id: 9 }), "steps/toolCallRequest", TRUSTED); + exporter.end(handle, decisionResponse(9, { decision: "deny", reason_codes: ["signature_invalid"] }), "signature_invalid"); + await exporter.flush(); + + const [span] = spans(); + expect(span?.name).toBe(REJECTED_SPAN_NAME); + expect(span?.events).toEqual([]); + expect(span?.attributes["acs.request_id"]).toBeUndefined(); + expect(span?.attributes["guardian.rejected.reason"]).toBe("signature_invalid"); + }); + + it("counts the time before the span opened toward a late decision", async () => { + const { exporter, spans } = memoryExporter({ timeoutConfig: { default_ms: 50 } }); + const handle = exporter.start(toolCallEnvelope("ls", { id: 6 }), "steps/toolCallRequest", { + trusted: true, + receivedAt: performance.now() - 100, + }); + exporter.end(handle, decisionResponse(6, { decision: "allow" })); + await exporter.flush(); + + const [span] = spans(); + const failure = span?.events.find((event) => event.name === "acs.decision.failure"); + expect(failure?.attributes?.["acs.failure.class"]).toBe("timeout"); + // The span starts at arrival too, so its own duration covers the 100 ms. + const durationMs = (span?.duration[0] ?? 0) * 1000 + (span?.duration[1] ?? 0) / 1e6; + expect(durationMs).toBeGreaterThanOrEqual(90); }); // #228: acs.provenance.source_id is emitted only when at least one attached @@ -1130,7 +1309,7 @@ describe("createOtelTraceExporter -- review fixes", () => { tool: { name: "run_shell" }, arguments: { a: { value: "x", provenance: provenance("tool_output") }, b: { value: "y", provenance: provenance("user_input") } }, }); - exporter.end(exporter.start(envelope, "steps/toolCallRequest"), decisionResponse(1, DENY)); + exporter.end(exporter.start(envelope, "steps/toolCallRequest", TRUSTED), decisionResponse(1, DENY)); await exporter.flush(); const attributes = spans()[0]?.attributes; @@ -1153,7 +1332,7 @@ describe("createOtelTraceExporter -- review fixes", () => { { tool: { name: "run_shell" }, arguments: args }, { sessionId: "x".repeat(10_000) }, ); - exporter.end(exporter.start(envelope, "steps/toolCallRequest"), decisionResponse(1, DENY)); + exporter.end(exporter.start(envelope, "steps/toolCallRequest", TRUSTED), decisionResponse(1, DENY)); await exporter.flush(); expect(spans()).toHaveLength(1); @@ -1171,7 +1350,7 @@ describe("createOtelTraceExporter -- review fixes", () => { process.env.OTEL_SPAN_EVENT_COUNT_LIMIT = "0"; try { const { exporter, spans } = memoryExporter(); - exporter.end(exporter.start(toolCallEnvelope("ls"), "steps/toolCallRequest"), decisionResponse(1, DENY)); + exporter.end(exporter.start(toolCallEnvelope("ls"), "steps/toolCallRequest", TRUSTED), decisionResponse(1, DENY)); await exporter.flush(); expect(spans()).toHaveLength(1); @@ -1199,7 +1378,7 @@ describe("createOtelTraceExporter -- review fixes", () => { exportTimeoutMs: 100, onError: (error) => errors.push(error), }); - exporter.end(exporter.start(toolCallEnvelope("ls"), "steps/toolCallRequest"), decisionResponse(1, DENY)); + exporter.end(exporter.start(toolCallEnvelope("ls"), "steps/toolCallRequest", TRUSTED), decisionResponse(1, DENY)); const started = performance.now(); await exporter.shutdown(); @@ -1228,7 +1407,7 @@ describe("createOtelTraceExporter -- review fixes", () => { for (const mappingPath of [undefined, path]) { const { exporter, spans } = memoryExporter({ mappingPath }); exporter.end( - exporter.start(toolCallEnvelope("ls"), "steps/toolCallRequest"), + exporter.start(toolCallEnvelope("ls"), "steps/toolCallRequest", TRUSTED), decisionResponse(1, { ...DENY, reason_codes: ["blocked"], chain_hash: "a".repeat(64) }), ); await exporter.flush(); @@ -1314,7 +1493,7 @@ describe("the Guardian's exporter does not depend on its environment", () => { // Three spans ended in one synchronous turn: with a queue of one, the // SDK's default-from-environment processor drops the ones that arrive // while the first export is in flight. - for (const id of [1, 2, 3]) exporter.end(exporter.start(envelope(id), "steps/toolCallRequest"), allow(id)); + for (const id of [1, 2, 3]) exporter.end(exporter.start(envelope(id), "steps/toolCallRequest", TRUSTED), allow(id)); await exporter.shutdown(); const spans = readFileSync(path, "utf8") .split("\n") @@ -1340,7 +1519,7 @@ describe("the Guardian's exporter does not depend on its environment", () => { const path = join(dir, "trace.jsonl"); const exporter = createOtelTraceExporter({ filePath: path, methodsEvaluated: METHODS_EVALUATED }); const request = envelope(1); - exporter.end(exporter.start(request, "steps/toolCallRequest"), allow(1)); + exporter.end(exporter.start(request, "steps/toolCallRequest", TRUSTED), allow(1)); await exporter.shutdown(); const resourceSpans = JSON.parse(readFileSync(path, "utf8").split("\n")[0]!).resourceSpans[0]; expect(resourceSpans.resource.attributes).toEqual([{ key: "service.name", value: { stringValue: OTEL_SERVICE_NAME } }]); @@ -1357,7 +1536,7 @@ describe("the Guardian's exporter does not depend on its environment", () => { const collector = startCollector(); try { const exporter = createOtelTraceExporter({ endpoint: collector.url, methodsEvaluated: METHODS_EVALUATED }); - exporter.end(exporter.start(envelope(1), "steps/toolCallRequest"), allow(1)); + exporter.end(exporter.start(envelope(1), "steps/toolCallRequest", TRUSTED), allow(1)); await exporter.shutdown(); expect(collector.received.map((r) => r.headers.get("content-encoding"))).toEqual([null]); } finally { @@ -1373,7 +1552,7 @@ describe("the Guardian's exporter does not depend on its environment", () => { const collector = startCollector(); try { const exporter = createOtelTraceExporter({ endpoint: collector.url, methodsEvaluated: METHODS_EVALUATED }); - exporter.end(exporter.start(envelope(1), "steps/toolCallRequest"), allow(1)); + exporter.end(exporter.start(envelope(1), "steps/toolCallRequest", TRUSTED), allow(1)); await exporter.shutdown(); expect(collector.received.map((r) => r.headers.get("x-acs-probe"))).toEqual(["1"]); } finally { @@ -1415,7 +1594,7 @@ describe("the Guardian's exporter does not depend on its environment", () => { const endpoint = "http://127.0.0.1:" + collector.port + "/v1/traces"; keepLoopbackOffProxy(endpoint, process.env); const exporter = createOtelTraceExporter({ endpoint, methodsEvaluated: ["steps/toolCallRequest"], onError: () => {} }); - exporter.end(exporter.start(${JSON.stringify(envelope(1))}, "steps/toolCallRequest"), ${JSON.stringify(allow(1))}); + exporter.end(exporter.start(${JSON.stringify(envelope(1))}, "steps/toolCallRequest", { trusted: true }), ${JSON.stringify(allow(1))}); await exporter.shutdown(); console.log(JSON.stringify({ collected })); collector.stop(true); diff --git a/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl b/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl index 437a8f2..bb22651 100644 --- a/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl +++ b/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl @@ -1,2 +1,2 @@ -{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"acs-reference-guardian"}}],"droppedAttributesCount":0},"scopeSpans":[{"scope":{"name":"acs-reference-guardian"},"spans":[{"traceId":"9785c34288d27dbd92fcf12e53ac3315","spanId":"31258fb4b343b292","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065946937000000","endTimeUnixNano":"1792065947036723875","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}},{"key":"acs.capability","value":{"stringValue":"process.exec"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"allow"}}],"name":"acs.decision","timeUnixNano":"1792065947036676541","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"112037a108c3893225309fa193536b77","spanId":"46883c95863bb8fb","name":"gen_ai.tool.result","kind":1,"startTimeUnixNano":"1792065947038000000","endTimeUnixNano":"1792065947088187667","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b02"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}},{"key":"acs.exit_status","value":{"stringValue":"success"}},{"key":"acs.request_id_ref","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"allow"}}],"name":"acs.decision","timeUnixNano":"1792065947088182292","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"981a929f7cea4c999642a38dd3d822b1","spanId":"3b199edb6364b94d","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065947089000000","endTimeUnixNano":"1792065947141424750","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b03"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}},{"key":"acs.provenance.origin","value":{"arrayValue":{"values":[{"stringValue":"user_input"},{"stringValue":"agent_generated"}]}}},{"key":"acs.provenance.source_id","value":{"arrayValue":{"values":[{"stringValue":"sample-user-1"},{"stringValue":""}]}}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"deny"}},{"key":"acs.reasoning","value":{"stringValue":"This command was blocked because it matches a destructive-shell-command pattern. Policy: destructive_shell_command_blocked, from AGT's stock bundle (agt_stock). Matched at offset 0."}},{"key":"acs.reason_codes","value":{"arrayValue":{"values":[{"stringValue":"destructive_shell_command_blocked"}]}}},{"key":"acs.policy.ids","value":{"arrayValue":{"values":[{"stringValue":"agt_stock"}]}}},{"key":"acs.policy.versions","value":{"arrayValue":{"values":[{"stringValue":""}]}}},{"key":"acs.policy.rule_ids","value":{"arrayValue":{"values":[{"stringValue":"destructive_shell_command_blocked"}]}}}],"name":"acs.decision","timeUnixNano":"1792065947141419958","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"95c92494c004b23662900ae8c17ea04c","spanId":"02d46f81d8206a0f","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065947141000000","endTimeUnixNano":"1792065947141270000","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b04"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.failure.class","value":{"stringValue":"guardian_error"}},{"key":"acs.failure.posture","value":{"stringValue":"fail_open"}},{"key":"acs.failure.reported_by","value":{"stringValue":"guardian"}},{"key":"acs.failure.error_code","value":{"intValue":-32010}},{"key":"acs.failure.elapsed_ms","value":{"intValue":0}},{"key":"acs.failure.payload_bytes","value":{"intValue":340}}],"name":"acs.decision.failure","timeUnixNano":"1792065947141268125","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":2,"message":"The Guardian returned an error instead of a decision."},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"e5703a6e87d75324de28fd7cc555724b","spanId":"e813670f52f6a811","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065947142000000","endTimeUnixNano":"1792065949203024291","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b05"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"allow"}}],"name":"acs.decision","timeUnixNano":"1792065949202988875","droppedAttributesCount":0},{"attributes":[{"key":"acs.failure.class","value":{"stringValue":"timeout"}},{"key":"acs.failure.posture","value":{"stringValue":"fail_open"}},{"key":"acs.failure.reported_by","value":{"stringValue":"guardian"}},{"key":"acs.failure.timeout_ms","value":{"intValue":1500}},{"key":"acs.failure.elapsed_ms","value":{"intValue":2061}},{"key":"acs.failure.payload_bytes","value":{"intValue":367}}],"name":"acs.decision.failure","timeUnixNano":"1792065949203022666","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257}]}]}]} -{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"acs-reference-audit-converter"}}],"droppedAttributesCount":0},"scopeSpans":[{"scope":{"name":"acs-reference-audit-converter"},"spans":[{"traceId":"9b082079ac3695104b2da1df27340e16","spanId":"99e8d283763d16d3","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065949205000000","endTimeUnixNano":"1792065949205000000","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b06"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.failure.class","value":{"stringValue":"unreachable"}},{"key":"acs.failure.posture","value":{"stringValue":"fail_open"}},{"key":"acs.failure.reported_by","value":{"stringValue":"observed_agent"}},{"key":"acs.failure.detail","value":{"stringValue":"The host could not reach the Guardian."}}],"name":"acs.decision.failure","timeUnixNano":"1792065949205000000","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":2,"message":"The host could not reach the Guardian."},"links":[],"droppedLinksCount":0,"flags":257}]}]}]} +{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"acs-reference-guardian"}}],"droppedAttributesCount":0},"scopeSpans":[{"scope":{"name":"acs-reference-guardian"},"spans":[{"traceId":"7752aef7c63150fcc88f856974d22f5b","spanId":"7bd24ca6a5dd31f0","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792534253539454676","endTimeUnixNano":"1792534253649000000","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}},{"key":"acs.capability","value":{"stringValue":"process.exec"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"allow"}}],"name":"acs.decision","timeUnixNano":"1792534253649000000","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"180a78a6f4b17aa91f6a952195321255","spanId":"003c1193df7f552a","name":"gen_ai.tool.result","kind":1,"startTimeUnixNano":"1792534253650630391","endTimeUnixNano":"1792534253716000000","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b02"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}},{"key":"acs.exit_status","value":{"stringValue":"success"}},{"key":"acs.request_id_ref","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"allow"}}],"name":"acs.decision","timeUnixNano":"1792534253716000000","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"13717f291e2368066d2fa3f2434ed2f6","spanId":"2a7efedbb7ee3881","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792534253716796645","endTimeUnixNano":"1792534253780000000","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b03"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}},{"key":"acs.provenance.origin","value":{"arrayValue":{"values":[{"stringValue":"user_input"},{"stringValue":"agent_generated"}]}}},{"key":"acs.provenance.source_id","value":{"arrayValue":{"values":[{"stringValue":"sample-user-1"},{"stringValue":""}]}}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"deny"}},{"key":"acs.reasoning","value":{"stringValue":"This command was blocked because it matches a destructive-shell-command pattern. Policy: destructive_shell_command_blocked, from AGT's stock bundle (agt_stock). Matched at offset 0."}},{"key":"acs.reason_codes","value":{"arrayValue":{"values":[{"stringValue":"destructive_shell_command_blocked"}]}}},{"key":"acs.policy.ids","value":{"arrayValue":{"values":[{"stringValue":"agt_stock"}]}}},{"key":"acs.policy.versions","value":{"arrayValue":{"values":[{"stringValue":""}]}}},{"key":"acs.policy.rule_ids","value":{"arrayValue":{"values":[{"stringValue":"destructive_shell_command_blocked"}]}}}],"name":"acs.decision","timeUnixNano":"1792534253780000000","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"2cadee9c2b0236f4cf64ecf54933d071","spanId":"d2666e080ce88014","name":"guardian.rejected_request","kind":1,"startTimeUnixNano":"1792534253779859236","endTimeUnixNano":"1792534253781000000","attributes":[{"key":"guardian.rejected.reason","value":{"stringValue":"envelope_invalid"}},{"key":"guardian.rejected.claimed_session_id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"guardian.rejected.claimed_request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01"}},{"key":"guardian.rejected.error_code","value":{"intValue":-32010}}],"droppedAttributesCount":0,"events":[],"droppedEventsCount":0,"status":{"code":2,"message":"The Guardian could not vouch for this request, so it carries no ACS ids or events."},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"f92f9aa4700acfd946b59a2fe19e489e","spanId":"2e4aa2bc8b522e06","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792534253780950247","endTimeUnixNano":"1792534255851000000","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b05"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"allow"}}],"name":"acs.decision","timeUnixNano":"1792534255851000000","droppedAttributesCount":0},{"attributes":[{"key":"acs.failure.class","value":{"stringValue":"timeout"}},{"key":"acs.failure.posture","value":{"stringValue":"fail_open"}},{"key":"acs.failure.reported_by","value":{"stringValue":"guardian"}},{"key":"acs.failure.timeout_ms","value":{"intValue":1500}},{"key":"acs.failure.elapsed_ms","value":{"intValue":2070}},{"key":"acs.failure.payload_bytes","value":{"intValue":367}}],"name":"acs.decision.failure","timeUnixNano":"1792534255851000000","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257}]}]}]} +{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"acs-reference-audit-converter"}}],"droppedAttributesCount":0},"scopeSpans":[{"scope":{"name":"acs-reference-audit-converter"},"spans":[{"traceId":"8ad7d2634fb9d194b1414a0033d90467","spanId":"8572a4b873e3e58e","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792534255853000000","endTimeUnixNano":"1792534255853000000","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b06"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.failure.class","value":{"stringValue":"unreachable"}},{"key":"acs.failure.posture","value":{"stringValue":"fail_open"}},{"key":"acs.failure.reported_by","value":{"stringValue":"observed_agent"}},{"key":"acs.failure.detail","value":{"stringValue":"The host could not reach the Guardian."}}],"name":"acs.decision.failure","timeUnixNano":"1792534255853000000","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":2,"message":"The host could not reach the Guardian."},"links":[],"droppedLinksCount":0,"flags":257}]}]}]} diff --git a/reference-implementations/agt/scripts/make-sample-trace.ts b/reference-implementations/agt/scripts/make-sample-trace.ts index de4d5e4..daf1759 100644 --- a/reference-implementations/agt/scripts/make-sample-trace.ts +++ b/reference-implementations/agt/scripts/make-sample-trace.ts @@ -42,17 +42,26 @@ export type Scenario = { capability?: string; /** The `request_id_ref` a result carried, if any. */ requestIdRef?: string; - /** The decision the Guardian answered with, or none when it answered an - * error or was never reached. */ + /** The decision the Guardian answered with, or none when it was never + * reached. */ decision?: "allow" | "deny"; }; +/** + * A forged request that reuses the allowed call's session and request id and + * fails the schema (it carries no JSON-RPC id). The Guardian rejects it + * before trusting it, so it is recorded as a `guardian.rejected_request` + * span with the ids it claimed under rejected names, and the allowed call + * keeps the only span under its pair (Trace Events, issue #283). It is not a + * step, so it is not in SCENARIOS, which are keyed by the pair on a span. + */ +export const REJECTED = { name: "forged", claimedRequestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01", reason: "envelope_invalid" } as const; + /** Every step the sample holds, keyed by the request id its span carries. */ export const SCENARIOS: readonly Scenario[] = [ { name: "allowed", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01", capability: "process.exec", decision: "allow" }, { name: "result", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b02", requestIdRef: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01", decision: "allow" }, { name: "denied", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b03", decision: "deny" }, - { name: "error", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b04" }, { name: "late", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b05", decision: "allow" }, { name: "unreachable", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b06" }, ]; @@ -114,8 +123,8 @@ function requests(): Record[] { }, 3, ), - // No JSON-RPC id: the Guardian answers with an error, not a decision. - envelope("steps/toolCallRequest", requestIdOf("error"), bash("pwd"), null), + // No JSON-RPC id, so it fails the schema, and the allowed call's pair. + envelope("steps/toolCallRequest", REJECTED.claimedRequestId, bash("pwd"), null), envelope("steps/toolCallRequest", requestIdOf("late"), bash(`echo ${SLOW_MARKER}`), 5), ]; } diff --git a/reference-implementations/agt/test/sample-trace.test.ts b/reference-implementations/agt/test/sample-trace.test.ts index dbc86ab..8ca2fda 100644 --- a/reference-implementations/agt/test/sample-trace.test.ts +++ b/reference-implementations/agt/test/sample-trace.test.ts @@ -4,7 +4,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { CONVERTER_SERVICE_NAME } from "../packages/guardian/src/audit-to-otlp.ts"; import { OTEL_SERVICE_NAME } from "../packages/guardian/src/otel-trace-exporter.ts"; -import { generateSampleTrace, SAMPLE_PATH, SCENARIOS } from "../scripts/make-sample-trace.ts"; +import { generateSampleTrace, REJECTED, SAMPLE_PATH, SCENARIOS } from "../scripts/make-sample-trace.ts"; /** * The committed sample trace (issue #266), held to the mapping it claims to @@ -100,6 +100,10 @@ const CONVERTER_EXEMPT: Readonly> = { }; const SPANS = spansIn(SAMPLE_PATH); +/** The spans for steps the Guardian governed. A rejected request is recorded + * under a name outside step_to_span (issue #283), so the mapping's step + * rules do not apply to it; its own test is below. */ +const STEP_SPANS = SPANS.filter((span) => span.name !== "guardian.rejected_request"); function spanFor(requestId: string): OtlpSpan | undefined { return SPANS.find((span) => attributes(span)["acs.request_id"] === requestId); @@ -111,9 +115,9 @@ function failureOf(span: OtlpSpan | undefined): Record | undefi } describe("the committed sample trace matches the mapping", () => { - it("names every span with a step_to_span value", () => { - expect(SPANS.length).toBeGreaterThan(0); - for (const span of SPANS) { + it("names every step span with a step_to_span value", () => { + expect(STEP_SPANS.length).toBeGreaterThan(0); + for (const span of STEP_SPANS) { expect({ name: span.name, mapped: span.name in STEP_BY_SPAN_NAME }).toEqual({ name: span.name, mapped: true }); } }); @@ -128,7 +132,7 @@ describe("the committed sample trace matches the mapping", () => { it("carries every required attribute, apart from the named converter exemptions", () => { const always = MAPPING.properties.step_span_attributes.properties.required_attributes.default; - for (const span of SPANS) { + for (const span of STEP_SPANS) { const present = attributes(span); const fromConverter = failureOf(span)?.["acs.failure.reported_by"] === "observed_agent"; for (const attribute of [...always, ...STEP_BY_SPAN_NAME[span.name]!.required_attributes]) { @@ -182,7 +186,7 @@ describe("the committed sample trace matches the mapping", () => { expect(denied["acs.provenance.source_id"]).toContain(""); }); - it("uses only failure classes, postures and reporters the mapping lists, and covers the four failures", () => { + it("uses only failure classes, postures and reporters the mapping lists, and covers the failures it holds", () => { const allowed = MAPPING.properties.decision_failure_event.properties.attribute_values.default; const seen = new Set(); for (const span of SPANS) { @@ -197,7 +201,22 @@ describe("the committed sample trace matches the mapping", () => { } seen.add(`${failure["acs.failure.class"]}/${failure["acs.failure.reported_by"]}`); } - expect([...seen].sort()).toEqual(["guardian_error/guardian", "timeout/guardian", "unreachable/observed_agent"]); + // The sample holds no step the Guardian answered with an error. + expect([...seen].sort()).toEqual(["timeout/guardian", "unreachable/observed_agent"]); + }); + + it("holds one span per step, and keeps the forged request apart from the call whose ids it claimed", () => { + for (const scenario of SCENARIOS) { + const underPair = SPANS.filter((span) => attributes(span)["acs.request_id"] === scenario.requestId); + expect({ scenario: scenario.name, spans: underPair.length }).toEqual({ scenario: scenario.name, spans: 1 }); + } + const rejected = SPANS.filter((span) => span.name === "guardian.rejected_request"); + expect(rejected).toHaveLength(1); + const claimed = attributes(rejected[0]); + expect(claimed["guardian.rejected.claimed_request_id"]).toBe(REJECTED.claimedRequestId); + expect(claimed["guardian.rejected.reason"]).toBe(REJECTED.reason); + expect(claimed["acs.request_id"]).toBeUndefined(); + expect(rejected[0]?.events ?? []).toEqual([]); }); });