diff --git a/README.md b/README.md index 1e42c727..053f7587 100644 --- a/README.md +++ b/README.md @@ -294,7 +294,9 @@ as synchronous file appends on the decision path, with no batching. Trace export partial: set `ACS_OTEL_FILE` or `ACS_OTEL_ENDPOINT` and the Guardian exports OTLP spans for the two live methods, `steps/toolCallRequest` and `steps/toolCallResult`, as the [Guardian README](reference-implementations/agt/packages/guardian/README.md#export-traces) -describes. +describes. A [sample trace](reference-implementations/agt/samples/trace/acs-trace-sample.jsonl) +shows that output without running anything, and [Produce a trace](reference-implementations/agt/README.md#produce-a-trace) +walks you from a fresh clone to a trace file of your own. That exporter does not make the reference ACS-Trace conformant, and this tree claims no Trace profile. It covers two methods, emits no OCSF, and leaves out a required attribute diff --git a/docs/spec/trace/extend_opentelemetry.md b/docs/spec/trace/extend_opentelemetry.md index b2afa5cb..daa25127 100644 --- a/docs/spec/trace/extend_opentelemetry.md +++ b/docs/spec/trace/extend_opentelemetry.md @@ -16,6 +16,152 @@ A typical agent session produces a tree of spans: This hierarchy is deterministic — implementations that follow the mapping produce comparable traces across vendors and Guardians. +## Example: a denied tool call + +This is one span the reference Guardian exported, a `steps/toolCallRequest` the stock policy denied, copied from the committed [sample trace](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl). Every value in it is synthetic. It's the OTLP/JSON shape the exporter writes, so attribute values are typed (`stringValue`, `arrayValue`). + +The span carries the join pair (`acs.session.id` and `acs.request_id`), the tool name, and the two Provenance columns. `acs.provenance.source_id` has `""` where the second object has no `source_id`. The decision rides on the span as the `acs.decision` event, with the verdict, the reasoning, the reason codes and the three policy arrays. `acs.policy.versions` is `[""]`, because this Guardian doesn't send a policy version. + +```json +{ + "traceId": "85c4878c9bda9e5b4f2fc6aae242d31f", + "spanId": "dea6530ad1681c0e", + "name": "gen_ai.tool.call", + "kind": 1, + "startTimeUnixNano": "1791928279698000000", + "endTimeUnixNano": "1791928279749301083", + "attributes": [ + { + "key": "acs.session.id", + "value": { + "stringValue": "5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01" + } + }, + { + "key": "acs.request_id", + "value": { + "stringValue": "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b03" + } + }, + { + "key": "gen_ai.tool.name", + "value": { + "stringValue": "Bash" + } + }, + { + "key": "acs.provenance.origin", + "value": { + "arrayValue": { + "values": [ + { + "stringValue": "user_input" + }, + { + "stringValue": "agent_generated" + } + ] + } + } + }, + { + "key": "acs.provenance.source_id", + "value": { + "arrayValue": { + "values": [ + { + "stringValue": "sample-user-1" + }, + { + "stringValue": "" + } + ] + } + } + } + ], + "droppedAttributesCount": 0, + "events": [ + { + "attributes": [ + { + "key": "acs.decision", + "value": { + "stringValue": "deny" + } + }, + { + "key": "acs.reasoning", + "value": { + "stringValue": "This command was blocked because it matches a destructive-shell-command pattern. Policy: destructive_shell_command_blocked, from AGT's stock bundle (agt_stock). Matched at offset 0." + } + }, + { + "key": "acs.reason_codes", + "value": { + "arrayValue": { + "values": [ + { + "stringValue": "destructive_shell_command_blocked" + } + ] + } + } + }, + { + "key": "acs.policy.ids", + "value": { + "arrayValue": { + "values": [ + { + "stringValue": "agt_stock" + } + ] + } + } + }, + { + "key": "acs.policy.versions", + "value": { + "arrayValue": { + "values": [ + { + "stringValue": "" + } + ] + } + } + }, + { + "key": "acs.policy.rule_ids", + "value": { + "arrayValue": { + "values": [ + { + "stringValue": "destructive_shell_command_blocked" + } + ] + } + } + } + ], + "name": "acs.decision", + "timeUnixNano": "1791928279749295500", + "droppedAttributesCount": 0 + } + ], + "droppedEventsCount": 0, + "status": { + "code": 0 + }, + "links": [], + "droppedLinksCount": 0, + "flags": 257 +} +``` + +To produce a trace like this yourself, follow [Produce a trace](https://github.com/GenAI-Security-Project/agent-control-standard/blob/main/reference-implementations/agt/README.md#produce-a-trace) in the reference implementation's README. + ## Provenance attributes When a hook payload carries Provenance, the resulting span MUST carry `acs.provenance.origin` and SHOULD carry `acs.provenance.source_id` when populated. The set of provenance ids cited by a Guardian decision (the response envelope's `cited_provenance_ids`) MAY be expressed as OTel span links keyed by `provenance_id`, letting tools that index span links reconstruct the lineage graph without parsing payloads. diff --git a/reference-implementations/agt/README.md b/reference-implementations/agt/README.md index bb69941b..91b91ad8 100644 --- a/reference-implementations/agt/README.md +++ b/reference-implementations/agt/README.md @@ -193,6 +193,111 @@ echo '{"session_id":"demo","hook_event_name":"PreToolUse","tool_name":"Bash","to The shim never runs the command. It only asks the Guardian. +## Produce a trace + +This walks you from a fresh clone to an OpenTelemetry trace file of your own, in about ten minutes. You don't need Claude Code or a collector for it. Every command runs from `reference-implementations/agt` unless the step says otherwise. + +If you only want to see what the output looks like, open [`samples/trace/acs-trace-sample.jsonl`](samples/trace/acs-trace-sample.jsonl). It's a committed run of the same exporter. + +**1. Get the code.** You need `git` and [Bun](https://bun.sh) 1.3 (CI runs 1.3.11). If someone gave you a commit to use, check it out, so your results match theirs. + +```bash +git clone https://github.com/GenAI-Security-Project/agent-control-standard.git +cd agent-control-standard +git checkout # skip this line to use the latest integration +cd reference-implementations/agt +bun install --frozen-lockfile +mkdir -p .acs && chmod 700 .acs # owner-only, even if .acs already existed +export NO_PROXY=127.0.0.1,localhost no_proxy=127.0.0.1,localhost # keep local traffic off any HTTP_PROXY +``` + +Run every later step in a terminal where you've run the `export` line. Without it, an `HTTP_PROXY` in your shell sends the shim's requests and the Guardian's spans to the proxy. + +**2. Start the Guardian with trace export on.** Leave it running in its own terminal. + +```bash +ACS_OTEL_FILE=.acs/otel-traces.jsonl bun run guardian +``` + +**3. Send it some steps.** In a second terminal, pipe hook payloads through the Claude Code shim. The shim asks the Guardian and prints its answer, and it never runs the command. These three give you an allowed call, a denied call and a tool result. + +```bash +echo '{"session_id":"trace-demo","hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"ls -la"}}' \ + | bun run hosts/claude-code/acs-hook.ts +echo '{"session_id":"trace-demo","hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"rm -rf /"}}' \ + | bun run hosts/claude-code/acs-hook.ts +echo '{"session_id":"trace-demo","hook_event_name":"PostToolUse","tool_name":"Bash","tool_input":{"command":"ls -la"},"tool_response":{"stdout":"README.md"}}' \ + | bun run hosts/claude-code/acs-hook.ts +``` + +Use made-up session ids and commands, because the trace records what you send. + +**4. Stop the Guardian** with Ctrl-C in its terminal. Spans are written in batches, and stopping the Guardian writes the last one. + +**5. Read the trace.** `.acs/otel-traces.jsonl` holds one OTLP export request per line. This prints one line per span: its name, then the decision, or the failure class when no decision arrived. + +```bash +bun -e 'const clean = (s) => String(s ?? "").replace(/[\u0000-\u001f\u007f-\u009f]/g, "?"); let bad = 0; for (const l of (await Bun.file(process.argv[1]).text()).split("\n").filter(Boolean)) { try { const req = JSON.parse(l); for (const r of req.resourceSpans) for (const s of r.scopeSpans) for (const p of s.spans) { const v = (e, k) => e.attributes?.find((a) => a.key === k)?.value.stringValue; console.log(clean(p.name), (p.events ?? []).map((e) => clean(v(e, "acs.decision") ?? v(e, "acs.failure.class"))).join(" ")) } } catch { bad++ } } if (bad) console.log(`${bad} line(s) were not readable and were skipped`)' .acs/otel-traces.jsonl +``` + +To read the collector's file in step 7, run the same command with `.acs/otel-collector-traces.jsonl` at the end. + +You should see two `gen_ai.tool.call` spans, one `allow` and one `deny`, and one `gen_ai.tool.result` span with `allow`. The spans carry the attributes [`trace/otel-mapping.json`](../../specification/v0.1.0/trace/otel-mapping.json) lists, and the [Guardian README](packages/guardian/README.md#export-traces) says which ones appear when. + +**6. Capture the failures only the host sees.** With the Guardian stopped, send one more step. The shim can't reach the Guardian, so it proceeds under the posture the Guardian declared when the session started, which is `proceed` for the stock Guardian, and records that in `.acs/audit.jsonl`. The converter turns that record into a span in the same format. + +```bash +echo '{"session_id":"trace-demo","hook_event_name":"PreToolUse","tool_name":"Bash","tool_input":{"command":"ls"}}' \ + | bun run hosts/claude-code/acs-hook.ts +bun run packages/guardian/src/audit-to-otlp.ts .acs/audit.jsonl .acs/otel-audit.jsonl +``` + +The converter prints how many spans it wrote. Its spans carry the same session and request ids as the Guardian's, so you can read both files as one trace. + +Run the converter after the agent session has ended. It refuses to write to a file that already exists, so it can't append into the audit log or the Guardian's trace. Delete `.acs/otel-audit.jsonl` first if you're running it again. If its summary shows a non-zero replay count, those were steps that proceeded without a decision that this trace doesn't show yet ([issue #274](https://github.com/GenAI-Security-Project/agent-control-standard/issues/274)). + +**7. Check it with a real collector (optional).** If you want an independent check that the output is valid OTLP, install [otelcol-contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/releases) 0.162.0, which the repository was tested with, and send the spans to it, not to a file. Check that nothing else is listening first: `lsof -nP -iTCP:4318 -sTCP:LISTEN` should print nothing. + +```bash +umask 077 +otelcol-contrib --config otel-collector.yaml # wait for "Everything is ready" +NO_PROXY=127.0.0.1,localhost no_proxy=127.0.0.1,localhost \ + ACS_OTEL_ENDPOINT=http://127.0.0.1:4318/v1/traces bun run guardian # in another terminal +``` + +To run the collector in Docker instead, from `reference-implementations/agt`: + +```bash +docker run --rm --user "$(id -u):$(id -g)" -p 127.0.0.1:4318:4318 \ + -v "$PWD/otel-collector.yaml":/etc/otelcol-contrib/config.yaml:ro \ + -v "$PWD/.acs":/work/.acs -w /work \ + otel/opentelemetry-collector-contrib:0.162.0@sha256:39923a8e431bd1f57be82411999d389fcfe40857492e4365456d97a4c1f74be6 \ + --config /etc/otelcol-contrib/config.yaml \ + --set=receivers::otlp::protocols::http::endpoint=0.0.0.0:4318 +``` + +With Podman, use the same command with `podman` in place of `docker`, plus `--userns=keep-id --umask 077`, and `:z` on both mounts on an SELinux host. + +Start the collector before the Guardian. The Guardian stops sending to an endpoint after its first failed send, and prints `disabled after failure` once. If you see that, or you restart the collector, stop the Guardian, restart the collector, then start the Guardian again. + +Repeat step 3. The collector writes what it received to `.acs/otel-collector-traces.jsonl`. That file can still be empty right after you repeat step 3, because spans go out in batches about every five seconds. + +Stop the collector with Ctrl-C or `docker stop`, never `kill -9` or `docker kill`. A hard kill can leave half a line, and the next batch the restarted collector writes is joined onto it. The step 5 reader skips that combined line and tells you, but the spans in it are lost. + +The collector appends across restarts. Natively under `umask 077` the file is `0600`. In a container it's `0644`, and the `0700` `.acs` directory is what keeps other users out ([issue #71](https://github.com/GenAI-Security-Project/agent-control-standard/issues/71) tracks file modes). + +The `export NO_PROXY` line from step 1 keeps the spans off any `HTTP_PROXY`. The Guardian also adds a loopback endpoint to `NO_PROXY` itself. `OTEL_EXPORTER_OTLP_*HEADERS` variables still add headers to every send, so unset them if you don't mean them. + +If `.acs/otel-collector-traces.jsonl` doesn't appear within about ten seconds of step 3, the collector isn't running this config. + +Verified on macOS (arm64): natively with otelcol-contrib 0.162.0, and with the Docker command above on Docker Desktop for Mac. Rootless Docker, Podman, SELinux hosts and musl-based Linux haven't been verified. + +**Run it again.** Every output here except the converter's appends, so a second run on the same `.acs` doubles every count, and the converter refuses to run at all. Delete it first: `rm -rf .acs && mkdir -p .acs && chmod 700 .acs`. + +**What this is, and isn't.** The exporter covers the two hook methods this Guardian evaluates, `steps/toolCallRequest` and `steps/toolCallResult`. It isn't an ACS-Trace conformance claim. Traces driven through the shim also lack `acs.capability` and `acs.request_id_ref`, because the shim doesn't send them yet ([issue #274](https://github.com/GenAI-Security-Project/agent-control-standard/issues/274)). The committed sample drives the Guardian directly, so it has both. Spans also lack `acs.evaluator` until the Guardian sends `metadata.evaluator` (PR #229). + +The trace files, the audit log and the collector's port can be written by any process running as you, the observed agent included, and in a container by other containers on the same Docker network, so the output isn't tamper-evident. `service.name` (`acs-reference-guardian`, `acs-reference-audit-converter`) is a label each writer sets for itself, not authentication, and loopback isn't authentication either. These labels belong to the reference implementation, not to the ACS specification. + ## View the contract The contract has four layers in this tree. Read them in this order. diff --git a/reference-implementations/agt/otel-collector.yaml b/reference-implementations/agt/otel-collector.yaml index 533bbb24..6542e3f5 100644 --- a/reference-implementations/agt/otel-collector.yaml +++ b/reference-implementations/agt/otel-collector.yaml @@ -1,11 +1,30 @@ # An OpenTelemetry Collector that receives the Guardian's spans and writes -# them as OTLP/JSON, one export request per line -- the same shape -# ACS_OTEL_FILE writes directly. Run it with the contrib distribution, which -# is where the `file` exporter lives: -# otelcol-contrib --config otel-collector.yaml -# then start the Guardian with ACS_OTEL_ENDPOINT=http://127.0.0.1:4318/v1/traces -# Tested with otelcol-contrib 0.162.0. In a container, run it as your own -# user (--user "$(id -u):$(id -g)") so it can write into .acs/. +# them as OTLP/JSON, one export request per line, the same shape +# ACS_OTEL_FILE writes directly. Tested with otelcol-contrib 0.162.0, which +# is where the `file` exporter lives. The walkthrough in README.md +# ("Produce a trace", step 7) has the full sequence. In short: +# +# Native: +# otelcol-contrib --config otel-collector.yaml +# +# Docker (run from reference-implementations/agt, after +# mkdir -p .acs && chmod 700 .acs): +# docker run --rm --user "$(id -u):$(id -g)" -p 127.0.0.1:4318:4318 \ +# -v "$PWD/otel-collector.yaml":/etc/otelcol-contrib/config.yaml:ro \ +# -v "$PWD/.acs":/work/.acs -w /work \ +# otel/opentelemetry-collector-contrib:0.162.0@sha256:39923a8e431bd1f57be82411999d389fcfe40857492e4365456d97a4c1f74be6 \ +# --config /etc/otelcol-contrib/config.yaml \ +# --set=receivers::otlp::protocols::http::endpoint=0.0.0.0:4318 +# +# Podman: the same, plus --userns=keep-id --umask 077, and :z on both +# mounts on an SELinux host. +# +# The endpoint below is loopback and literal on purpose. Inside a container +# the receiver must listen on 0.0.0.0, and that override lives only on the +# container command line above, where -p 127.0.0.1:... keeps the host side +# on loopback. Never set it for a native run: it listens on every interface. +# Loopback is not authentication: any local process can send to this port, +# and in a container so can other containers on the same Docker network. receivers: otlp: protocols: @@ -15,9 +34,12 @@ receivers: exporters: file: path: .acs/otel-collector-traces.jsonl - # A fresh checkout has no .acs/ yet, and without this the exporter - # refuses to start. + # Keep what earlier runs wrote: without this a restart truncates the file. + append: true + # A fresh checkout has no .acs/ yet. 0700 applies only when the collector + # creates it. The walkthrough creates it first and sets 0700 itself. create_directory: true + directory_permissions: "0700" # basic, not detailed: detailed prints every attribute, reasoning text # and provenance source ids included, to a stdout that often ends up in a # log system nobody planned for. @@ -25,6 +47,9 @@ exporters: verbosity: basic service: + telemetry: + metrics: + level: none pipelines: traces: receivers: [otlp] diff --git a/reference-implementations/agt/package.json b/reference-implementations/agt/package.json index 6b920dcb..2f5f915d 100644 --- a/reference-implementations/agt/package.json +++ b/reference-implementations/agt/package.json @@ -11,7 +11,8 @@ "conformance": "bash scripts/run-conformance.sh", "watch:upstream": "bash scripts/run-upstream-watch.sh", "guardian": "bun run packages/guardian/src/main.ts", - "inspector": "bun run packages/inspector/src/main.ts" + "inspector": "bun run packages/inspector/src/main.ts", + "sample:trace": "bun run scripts/make-sample-trace.ts" }, "dependencies": { "agent-control-specification": "0.3.1-beta.0" diff --git a/reference-implementations/agt/packages/guardian/README.md b/reference-implementations/agt/packages/guardian/README.md index a46a56f3..a3d2d753 100644 --- a/reference-implementations/agt/packages/guardian/README.md +++ b/reference-implementations/agt/packages/guardian/README.md @@ -50,7 +50,7 @@ Every throw on the request path is caught. Nothing escapes the fetch handler, be ## Export traces -Set `ACS_OTEL_FILE`, `ACS_OTEL_ENDPOINT`, or both. With neither set, no exporter exists and the Guardian runs exactly as before. This is an exporter for the two live methods, not an ACS-Trace conformance claim. +Set `ACS_OTEL_FILE`, `ACS_OTEL_ENDPOINT`, or both. With neither set, no exporter exists and the Guardian runs exactly as before. This is an exporter for the two live methods, not an ACS-Trace conformance claim. To go from a fresh clone to a trace file of your own, follow [Produce a trace](../../README.md#produce-a-trace) in the tree README. ```bash ACS_OTEL_FILE=.acs/otel-traces.jsonl bun run guardian @@ -60,19 +60,19 @@ Each decided step becomes one span, named by [`trace/otel-mapping.json`](../../. Every span also carries `acs.session.id`, and `acs.request_id` whenever the envelope has one, so a session can be rebuilt from the trace alone. When an argument or output carries a Provenance object, the span carries `acs.provenance.origin` as an array, one element per Provenance object in wire order. `acs.provenance.source_id` is the matching array, with `""` where an object has no `source_id`, and it is left out when no object has one. Both stop at 64 objects. `acs.provenance.lineage_depth` is not emitted: `provenance.json` defines no such field. -Every string attribute is cut at 4096 characters, a long `acs.reasoning` included, so one large request cannot grow the export queue without bound. The sampler and span limits are set in code, so an `OTEL_TRACES_SAMPLER` or `OTEL_SPAN_*` variable in the environment does not change what is exported. +Every string attribute is cut at 4096 characters, a long `acs.reasoning` included, so one large request cannot grow the export queue without bound. The sampler and span limits are set in code, so `OTEL_TRACES_*`, `OTEL_SPAN_*`, `OTEL_ATTRIBUTE_*` and `OTEL_BSP_*` variables in the environment do not change what is exported, and the endpoint output is never compressed. `OTEL_EXPORTER_OTLP_*HEADERS` still add headers to the endpoint's requests, and a loopback endpoint is added to `NO_PROXY` so an `HTTP_PROXY` cannot route it. -The decision rides on the span as an `acs.decision` event. The event always carries `acs.decision`, carries `acs.reasoning` when the response has one, and carries `acs.reason_codes`, `acs.policy.ids`, `acs.policy.versions`, `acs.policy.rule_ids`, `acs.cited_provenance_ids` and `acs.chain_hash` only when the response carries them. `acs.evaluator` appears when the Guardian's response carries `metadata.evaluator`. This Guardian sends it once #229 is merged. +The decision rides on the span as an `acs.decision` event. The event always carries `acs.decision`, carries `acs.reasoning` when the response has one, and carries `acs.reason_codes`, `acs.policy.ids`, `acs.policy.versions`, `acs.policy.rule_ids`, `acs.cited_provenance_ids` and `acs.chain_hash` only when the response carries them. `acs.evaluator` appears when the Guardian's response carries `metadata.evaluator`. This Guardian sends it once PR #229 is merged. -This Guardian does not send `chain_hash`, `cited_provenance_ids`, or `policy_version` on its responses yet, so those attributes are absent from its spans today. The three policy arrays always have the same length as `policy_references`, with `""` where a reference has no value. `handshake/hello`, methods this Guardian does not dispatch, and unparseable bodies produce no span. +This Guardian does not send `chain_hash`, `cited_provenance_ids`, or `policy_version` on its responses yet. `acs.chain_hash` and `acs.cited_provenance_ids` are absent from its spans, and `acs.policy.versions` carries `""` for each policy reference. The three policy arrays always have the same length as `policy_references`, with `""` where a reference has no value. `handshake/hello`, methods this Guardian does not dispatch, and unparseable bodies produce no span. A step that got no decision is recorded as an `acs.decision.failure` event on the same span, with `acs.failure.class`, `acs.failure.posture` and `acs.failure.reported_by`, plus the error code, the timeout, the elapsed time and the request size when known. The Guardian emits it for an error it answered instead of a decision, for a decision it finished after the negotiated timeout, and for a step still open when it shut down. It carries the error code but never the error text, because that text can quote the request. A replay rejection (`-32005`) gets no event, because its pair belongs to the earlier request. The converter below produces the same event, reported by the observed agent, from the host adapter's audit log. -The file holds one OTLP `ExportTraceServiceRequest` per line, the shape the Collector's file exporter writes, with `service.name` set to `acs-reference-guardian`. Spans are batched and flushed when the Guardian closes on SIGINT or SIGTERM. A SIGKILL loses the last batch, and so does an output that has not finished three seconds into shutdown: the Guardian reports it on stderr and exits instead of waiting. [`otel-collector.yaml`](../../otel-collector.yaml) at the root of this tree receives the endpoint output and writes the same file shape. +The file holds one OTLP `ExportTraceServiceRequest` per line, the shape the Collector's file exporter writes, with `service.name` set to `acs-reference-guardian` (the converter below uses `acs-reference-audit-converter`, and both are labels, not attribution). Spans are batched and flushed when the Guardian closes on SIGINT or SIGTERM. A SIGKILL loses the last batch, and so does an output that has not finished three seconds into shutdown: the Guardian reports it on stderr and exits instead of waiting. [`otel-collector.yaml`](../../otel-collector.yaml) at the root of this tree receives the endpoint output and writes the same file shape. Export never touches a decision. Each output fails on its own: the first failed write or send disables that output for the rest of the process and prints one line on stderr, the other output keeps running, and the response goes out unchanged. -Use synthetic sessions for lab runs. The envelope log holds raw envelopes before any redaction, a span's `acs.reasoning` repeats the Guardian's explanation of them, and `acs.provenance.source_id` goes out as the host sent it: by `provenance.json` it can be a URL or a file path, so a signed URL or a home-directory path would too ([#71](https://github.com/GenAI-Security-Project/agent-control-standard/issues/71)). +Use synthetic sessions for lab runs. The envelope log holds raw envelopes before any redaction, a span's `acs.reasoning` repeats the Guardian's explanation of them, and `acs.provenance.source_id` goes out as the host sent it: by `provenance.json` it can be a URL or a file path, so a signed URL or a home-directory path would too ([issue #71](https://github.com/GenAI-Security-Project/agent-control-standard/issues/71)). ### Convert the host audit log @@ -82,7 +82,7 @@ The failures only the host sees, a Guardian it could not reach or one that answe bun run packages/guardian/src/audit-to-otlp.ts .acs/audit.jsonl .acs/otel-audit.jsonl ``` -Run it after the Guardian has stopped. Two writers on one file can interleave lines, so give the converter a file of its own or wait. Running it twice over the same log writes every span twice. +Run it after the agent session has ended, into a file that does not exist when it starts. It refuses an existing output path, so it doesn't append into the audit log, the Guardian's trace file, or its own earlier run, and a wrong audit path or mapping leaves no file behind. `acs.failure.posture` is the deployment's posture, not what became of the step. A refused request shows `fail_open` under a `proceed` deployment even though the host blocked it. The detail sentence says so, and the error code is in `acs.failure.error_code`. The detail is always a sentence the converter writes, never the host's message, which quotes the Guardian's error text. @@ -95,6 +95,16 @@ Entries with `outcome: "ungoverned"` are skipped: those are steps no gate asked It prints how many spans reached the file and exits non-zero when that is fewer than it built. +### Sample trace + +[`samples/trace/acs-trace-sample.jsonl`](../../samples/trace/acs-trace-sample.jsonl) is what this export looks like, so you can read it without running the Guardian. It holds six steps: an allowed tool call and its result, linked by `acs.request_id_ref`, a denied call carrying reasoning, reason codes, policy references and provenance, a call the Guardian answered with an error, a decision that finished after the timeout, and a step the host could not deliver, converted from its audit log. The decisions come from the stock policy, and the one late decision is delayed on purpose. Every value in it is synthetic. + +```bash +bun run sample:trace +``` + +That command regenerates it. `test/sample-trace.test.ts` checks the sample against `otel-mapping.json` and fails when the exporter's output no longer matches it, so a change to the exporter or the mapping comes with a regenerated sample. The converted span has no `gen_ai.tool.name`, because the audit log does not record the tool (issue #274). Like the exporter, the sample covers two hook methods and is not an ACS-Trace conformance claim. + ## Use it in-process The tests and the conformance harness start a Guardian in the same process. diff --git a/reference-implementations/agt/packages/guardian/src/audit-to-otlp.ts b/reference-implementations/agt/packages/guardian/src/audit-to-otlp.ts index 90ded720..33f96935 100644 --- a/reference-implementations/agt/packages/guardian/src/audit-to-otlp.ts +++ b/reference-implementations/agt/packages/guardian/src/audit-to-otlp.ts @@ -34,12 +34,14 @@ * * Reads the log with plain JSON.parse and loose field reads, importing * nothing from the host adapter: the file is the contract, the same way the - * Inspector reads it. Run it after the Guardian has stopped. Two writers on - * one file can interleave lines, and running it twice over the same log - * writes every span twice. + * Inspector reads it. Run it after the agent session has ended, into a file + * that does not exist when it starts: it refuses an existing output path, so + * it does not append into the audit log, the Guardian's trace file, or its + * own earlier run. */ import { createHash } from "node:crypto"; -import { readFileSync } from "node:fs"; +import { closeSync, mkdirSync, openSync, readFileSync } from "node:fs"; +import { dirname } from "node:path"; import { SpanStatusCode, type Attributes } from "@opentelemetry/api"; import { ExportResultCode, type ExportResult } from "@opentelemetry/core"; import { resourceFromAttributes } from "@opentelemetry/resources"; @@ -56,7 +58,7 @@ import { decisionFailureAttributes, GUARDIAN_ERROR_STATUS, loadOtelMapping, - OTEL_SERVICE_NAME, + SPAN_LIMITS, type FailureClass, } from "./otel-trace-exporter.ts"; @@ -192,18 +194,44 @@ function countingExporter(inner: SpanExporter): { exporter: SpanExporter; writte }; } +/** The converter's `service.name` and tracer name. Its own, not the + * Guardian's: these spans are derived after the fact from the host's audit + * log, and `acs.failure.reported_by` on each event already says which party + * claims the failure. A label, not attribution (see the Guardian README). */ +export const CONVERTER_SERVICE_NAME = "acs-reference-audit-converter"; + export async function convertAuditLog(auditPath: string, otelPath: string, mappingPath?: string): Promise { + // Everything that can fail without writing goes first, so a wrong audit + // path or a bad mapping leaves no output behind to block the rerun. const mapping = loadOtelMapping(mappingPath); + const auditText = readFileSync(auditPath, "utf8"); + // Exclusive create: an output path that exists when the converter starts is + // refused before anything is written. That one check covers the log itself, + // a hardlink to it, a symlink, the Guardian's own trace file, and a second + // run over the same output, each of which the append-only exporter would + // otherwise corrupt or double. It does not stop another process swapping + // the path mid-run. Nothing here defends against a same-user writer. + mkdirSync(dirname(otelPath), { recursive: true, mode: 0o700 }); + try { + closeSync(openSync(otelPath, "wx", 0o600)); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "EEXIST") { + throw new Error(`refusing to write ${otelPath}: it already exists, perhaps from an earlier or failed run. Give the converter a new file, or delete this one first.`); + } + throw error; + } const output = countingExporter(createFileSpanExporter(otelPath)); // One span per export, not batched: a batch processor's bounded queue // overflowed inside this synchronous loop and dropped spans silently. // Sampler set explicitly, so no OTEL_TRACES_SAMPLER can drop them either. const provider = new BasicTracerProvider({ - resource: resourceFromAttributes({ "service.name": OTEL_SERVICE_NAME }), + resource: resourceFromAttributes({ "service.name": CONVERTER_SERVICE_NAME }), sampler: new AlwaysOnSampler(), + // Frozen shared limits, copied: no OTEL_* variable may truncate or drop output. + spanLimits: { ...SPAN_LIMITS }, spanProcessors: [new SimpleSpanProcessor(output.exporter)], }); - const tracer = provider.getTracer(OTEL_SERVICE_NAME); + const tracer = provider.getTracer(CONVERTER_SERVICE_NAME); const summary: ConversionSummary = { converted: 0, written: 0, @@ -214,7 +242,7 @@ export async function convertAuditLog(auditPath: string, otelPath: string, mappi skippedUnreadable: 0, }; - for (const line of readFileSync(auditPath, "utf8").split("\n")) { + for (const line of auditText.split("\n")) { if (line.trim() === "") continue; let entry: JsonObject | undefined; try { @@ -306,7 +334,13 @@ if (import.meta.main) { console.error("usage: bun run packages/guardian/src/audit-to-otlp.ts "); process.exit(2); } - const summary = await convertAuditLog(auditPath, otelPath); + let summary: ConversionSummary; + try { + summary = await convertAuditLog(auditPath, otelPath); + } catch (error) { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); + } const noMethod = Object.entries(summary.skippedNoMethod) .map(([kind, count]) => `${count} ${kind}`) .join(", "); diff --git a/reference-implementations/agt/packages/guardian/src/otel-trace-exporter.ts b/reference-implementations/agt/packages/guardian/src/otel-trace-exporter.ts index 6a10917d..9969b931 100644 --- a/reference-implementations/agt/packages/guardian/src/otel-trace-exporter.ts +++ b/reference-implementations/agt/packages/guardian/src/otel-trace-exporter.ts @@ -55,7 +55,7 @@ import { ExportResultCode } from "@opentelemetry/core"; import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"; import { JsonTraceSerializer } from "@opentelemetry/otlp-transformer"; import { resourceFromAttributes } from "@opentelemetry/resources"; -import { AlwaysOnSampler, BasicTracerProvider, BatchSpanProcessor, type SpanExporter } from "@opentelemetry/sdk-trace-base"; +import { AlwaysOnSampler, BasicTracerProvider, BatchSpanProcessor, type SpanExporter, type SpanLimits } from "@opentelemetry/sdk-trace-base"; /** How long the OTLP/HTTP exporter keeps retrying one batch against an * endpoint that does not answer before it reports the failure. */ @@ -89,6 +89,19 @@ const REPLAY_DETECTED_CODE = -32005; * semantic-conventions package: one constant does not earn a dependency. */ export const OTEL_SERVICE_NAME = "acs-reference-guardian"; +/** Every span limit, set in code so no OTEL_SPAN_* or OTEL_ATTRIBUTE_* + * variable changes what is exported: the SDK reads the environment for any + * limit left unset. Shared with the audit converter, so the two writers of + * the same file shape cannot drift. Frozen, because both providers hold it. */ +export const SPAN_LIMITS: Readonly = Object.freeze({ + attributeValueLengthLimit: ATTRIBUTE_VALUE_LENGTH_LIMIT, + attributeCountLimit: 128, + eventCountLimit: 128, + attributePerEventCountLimit: 128, + linkCountLimit: 128, + attributePerLinkCountLimit: 128, +}); + // The repository's own mapping, five directories up from this file -- the // same relationship SCHEMA_ROOT in validate-envelope.ts relies on. const OTEL_MAPPING_PATH = fileURLToPath( @@ -562,6 +575,34 @@ export function createFileSpanExporter(path: string): SpanExporter { }; } +/** + * Adds a loopback endpoint's host to NO_PROXY and no_proxy, so an + * HTTP_PROXY in the Guardian's environment cannot route spans meant for this + * machine through a proxy: request content (reasoning text, provenance ids) + * would reach the proxy, and the collector would never see it. Bun's HTTP + * client honors these variables per request, and an explicit agent does not + * opt out, so the environment is the only lever. A non-loopback endpoint is + * left to the operator's proxy settings. + * + * The change is process-wide and permanent: it is written to process.env and + * never undone, so anything else in this process (a host adapter embedded + * with the Guardian) and any child process it spawns inherits it. That is + * acceptable because only loopback hosts are ever added, and traffic to this + * machine should not use a proxy for any of them. + */ +export function keepLoopbackOffProxy(endpoint: string): void { + const host = new URL(endpoint).hostname.replace(/^\[|\]$/g, ""); + if (host !== "127.0.0.1" && host !== "localhost" && host !== "::1") { + return; + } + for (const key of ["NO_PROXY", "no_proxy"]) { + const entries = (process.env[key] ?? "").split(",").map((entry) => entry.trim()).filter((entry) => entry !== ""); + if (!entries.includes(host)) { + process.env[key] = [...entries, host].join(","); + } + } +} + /** One place spans go, with the name its failure is reported under: the * file's path, the endpoint's URL, or "span exporter" for an injected one. */ type Output = { label: string; exporter: SpanExporter }; @@ -689,7 +730,14 @@ export function createOtelTraceExporter({ // seconds rather than the 10 s default: long enough for a collector that // is restarting, short enough that a flush or a shutdown behind a dead // one does not hold the process for ten. - construct(endpoint, () => new OTLPTraceExporter({ url: endpoint, timeoutMillis: ENDPOINT_TIMEOUT_MS })); + // compression set in code: OTEL_EXPORTER_OTLP_COMPRESSION would otherwise + // gzip the body. Headers from OTEL_EXPORTER_OTLP_*HEADERS are merged by + // the SDK under any set here and cannot be refused, so the README lists + // them as honored. + construct(endpoint, () => { + keepLoopbackOffProxy(endpoint); + return new OTLPTraceExporter({ url: endpoint, timeoutMillis: ENDPOINT_TIMEOUT_MS, compression: "none" as NonNullable[0]>["compression"] }); + }); } if (spanExporter !== undefined) { construct("span exporter", () => spanExporter); @@ -709,16 +757,18 @@ export function createOtelTraceExporter({ // away or drop decision events: the SDK reads OTEL_TRACES_SAMPLER and // the span limits from the environment for anything left unset. sampler: new AlwaysOnSampler(), - spanLimits: { - attributeValueLengthLimit: ATTRIBUTE_VALUE_LENGTH_LIMIT, - attributeCountLimit: 128, - eventCountLimit: 128, - attributePerEventCountLimit: 128, - linkCountLimit: 128, - attributePerLinkCountLimit: 128, - }, + // Copied, because the SDK may write to the object it is given. + spanLimits: { ...SPAN_LIMITS }, spanProcessors: outputs.map( - (output) => new BatchSpanProcessor(isolated(output, reportOutput), { exportTimeoutMillis: exportTimeoutMs }), + (output) => + new BatchSpanProcessor(isolated(output, reportOutput), { + // The SDK defaults, written out: anything left unset is read from + // OTEL_BSP_*, and a queue of one drops spans. + maxQueueSize: 2048, + maxExportBatchSize: 512, + scheduledDelayMillis: 5000, + exportTimeoutMillis: exportTimeoutMs, + }), ), }); tracer = provider.getTracer(OTEL_SERVICE_NAME); diff --git a/reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts b/reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts index b3ca0da5..581bf4c9 100644 --- a/reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts +++ b/reference-implementations/agt/packages/guardian/test/otel-trace-exporter.test.ts @@ -1,5 +1,6 @@ -import { describe, expect, it, spyOn } from "bun:test"; -import { existsSync, mkdtempSync, readFileSync, rmdirSync, statSync, unlinkSync, writeFileSync } from "node:fs"; +import { afterEach, describe, expect, it, spyOn } from "bun:test"; +import { existsSync, mkdtempSync, readFileSync, rmdirSync, rmSync, statSync, unlinkSync, writeFileSync } from "node:fs"; +import { createServer } from "node:net"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { SpanStatusCode } from "@opentelemetry/api"; @@ -10,6 +11,7 @@ import { FAILURE_DETAIL_BY_KIND } from "../src/audit-to-otlp.ts"; import { buildServerHello } from "../src/handshake.ts"; import { createFileSpanExporter, + keepLoopbackOffProxy, createOtelTraceExporter, DECISION_ATTRIBUTE_SOURCES, loadOtelMapping, @@ -1236,3 +1238,219 @@ describe("createOtelTraceExporter -- review fixes", () => { } }); }); + +// Issue #274: nothing in the lab's shell changes what the Guardian exports. +describe("the Guardian's exporter does not depend on its environment", () => { + const saved: Record = {}; + const remember = (key: string): void => { + if (!(key in saved)) saved[key] = process.env[key]; + }; + const setEnv = (values: Record): void => { + for (const [key, value] of Object.entries(values)) { + remember(key); + process.env[key] = value; + } + }; + const unsetEnv = (...keys: string[]): void => { + for (const key of keys) { + remember(key); + delete process.env[key]; + } + }; + /** A local stand-in for the collector that records what it receives. */ + const startCollector = () => { + const received: { headers: Headers }[] = []; + const server = Bun.serve({ + port: 0, + hostname: "127.0.0.1", + fetch(request) { + // Copied now: Bun invalidates the Request once the handler returns, + // and a header read after that comes back empty. + received.push({ headers: new Headers(request.headers) }); + return new Response("{}", { headers: { "content-type": "application/json" } }); + }, + }); + return { server, received, url: `http://127.0.0.1:${server.port}/v1/traces` }; + }; + afterEach(() => { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) { + // Bun's HTTP client latches a proxy variable when it is set and does + // not see it deleted, so every later fetch in this process would go + // to the dead proxy. Assigning an empty value first clears it. + // Seen on Bun 1.3.14 (the suite was also confirmed on 1.3.11). + process.env[key] = ""; + delete process.env[key]; + } else { + process.env[key] = value; + } + } + }); + + const envelope = (id: number) => ({ + jsonrpc: "2.0", + method: "steps/toolCallRequest", + id, + params: { + acs_version: "0.1.0", + request_id: crypto.randomUUID(), + timestamp: new Date().toISOString(), + metadata: { agent_id: "a", session_id: crypto.randomUUID() }, + payload: { tool: { name: "run_shell" }, capability: "process.exec", arguments: { command: { value: "ls" } } }, + }, + }); + const allow = (id: number) => ({ jsonrpc: "2.0", id, result: { type: "final", acs_version: "0.1.0", request_id: "r", decision: "allow" } }); + + it("keeps every span when OTEL_BSP_MAX_QUEUE_SIZE is 1", async () => { + setEnv({ OTEL_BSP_MAX_QUEUE_SIZE: "1" }); + const dir = mkdtempSync(join(tmpdir(), "acs-bsp-")); + try { + const path = join(dir, "trace.jsonl"); + const exporter = createOtelTraceExporter({ filePath: path, methodsEvaluated: METHODS_EVALUATED }); + // Three spans ended in one synchronous turn: with a queue of one, the + // SDK's default-from-environment processor drops the ones that arrive + // while the first export is in flight. + for (const id of [1, 2, 3]) exporter.end(exporter.start(envelope(id), "steps/toolCallRequest"), allow(id)); + await exporter.shutdown(); + const spans = readFileSync(path, "utf8") + .split("\n") + .filter(Boolean) + .flatMap((line) => JSON.parse(line).resourceSpans[0].scopeSpans[0].spans); + expect(spans.length).toBe(3); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, 10_000); + + it("keeps full ids, every attribute, and its own service.name under hostile limit and resource variables", async () => { + // A guard, not a red test: the Guardian already set its limits in code. + // It pins that the Task 1 refactor to the shared SPAN_LIMITS kept them. + setEnv({ + OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT: "8", + OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT: "1", + OTEL_SERVICE_NAME: "zzz", + OTEL_RESOURCE_ATTRIBUTES: "service.name=evil", + }); + const dir = mkdtempSync(join(tmpdir(), "acs-attr-")); + try { + const path = join(dir, "trace.jsonl"); + const exporter = createOtelTraceExporter({ filePath: path, methodsEvaluated: METHODS_EVALUATED }); + const request = envelope(1); + exporter.end(exporter.start(request, "steps/toolCallRequest"), allow(1)); + await exporter.shutdown(); + const resourceSpans = JSON.parse(readFileSync(path, "utf8").split("\n")[0]!).resourceSpans[0]; + expect(resourceSpans.resource.attributes).toEqual([{ key: "service.name", value: { stringValue: OTEL_SERVICE_NAME } }]); + const span = resourceSpans.scopeSpans[0].spans[0]; + expect(span.droppedAttributesCount ?? 0).toBe(0); + expect(span.attributes.find((a: { key: string }) => a.key === "acs.request_id").value.stringValue).toBe(request.params.request_id); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, 10_000); + + it("sends uncompressed JSON even when OTEL_EXPORTER_OTLP_COMPRESSION is gzip", async () => { + setEnv({ OTEL_EXPORTER_OTLP_COMPRESSION: "gzip", OTEL_EXPORTER_OTLP_TRACES_COMPRESSION: "gzip" }); + const collector = startCollector(); + try { + const exporter = createOtelTraceExporter({ endpoint: collector.url, methodsEvaluated: METHODS_EVALUATED }); + exporter.end(exporter.start(envelope(1), "steps/toolCallRequest"), allow(1)); + await exporter.shutdown(); + expect(collector.received.map((r) => r.headers.get("content-encoding"))).toEqual([null]); + } finally { + collector.server.stop(true); + } + }, 10_000); + + it("still sends headers from OTEL_EXPORTER_OTLP_TRACES_HEADERS, which the README lists as honored", async () => { + // The SDK merges environment headers under any set in code, so they + // cannot be refused. This pins today's behavior: if an SDK update changes + // it, the README's sentence has to change with it. + setEnv({ OTEL_EXPORTER_OTLP_TRACES_HEADERS: "x-acs-probe=1" }); + const collector = startCollector(); + try { + const exporter = createOtelTraceExporter({ endpoint: collector.url, methodsEvaluated: METHODS_EVALUATED }); + exporter.end(exporter.start(envelope(1), "steps/toolCallRequest"), allow(1)); + await exporter.shutdown(); + expect(collector.received.map((r) => r.headers.get("x-acs-probe"))).toEqual(["1"]); + } finally { + collector.server.stop(true); + } + }, 10_000); + + it("sends to a loopback endpoint directly when HTTP_PROXY is set", async () => { + // A proxy that only counts connections and never answers. + let proxyConnections = 0; + const proxy = createServer((socket) => { + proxyConnections += 1; + socket.destroy(); + }); + await new Promise((resolve) => proxy.listen(0, "127.0.0.1", resolve)); + const proxyUrl = `http://127.0.0.1:${(proxy.address() as { port: number }).port}`; + setEnv({ HTTP_PROXY: proxyUrl, http_proxy: proxyUrl }); + unsetEnv("NO_PROXY", "no_proxy"); + const collector = startCollector(); + try { + const exporter = createOtelTraceExporter({ endpoint: collector.url, methodsEvaluated: METHODS_EVALUATED }); + exporter.end(exporter.start(envelope(1), "steps/toolCallRequest"), allow(1)); + await exporter.shutdown(); + expect({ proxyConnections, collected: collector.received.length }).toEqual({ proxyConnections: 0, collected: 1 }); + } finally { + collector.server.stop(true); + proxy.close(); + } + }, 10_000); +}); + +describe("keepLoopbackOffProxy", () => { + const saved: Record = { + NO_PROXY: process.env.NO_PROXY, + no_proxy: process.env.no_proxy, + }; + const reset = (value: string | undefined): void => { + for (const key of ["NO_PROXY", "no_proxy"]) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + }; + afterEach(() => { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + }); + + it("adds localhost for a localhost endpoint", () => { + reset(undefined); + keepLoopbackOffProxy("http://localhost:4318/v1/traces"); + expect([process.env.NO_PROXY, process.env.no_proxy]).toEqual(["localhost", "localhost"]); + }); + + it("adds ::1 without brackets for an IPv6 loopback endpoint", () => { + reset(undefined); + keepLoopbackOffProxy("http://[::1]:4318/v1/traces"); + expect([process.env.NO_PROXY, process.env.no_proxy]).toEqual(["::1", "::1"]); + }); + + it("does not duplicate the entry when called twice", () => { + reset(undefined); + keepLoopbackOffProxy("http://127.0.0.1:4318/v1/traces"); + keepLoopbackOffProxy("http://127.0.0.1:4318/v1/traces"); + expect([process.env.NO_PROXY, process.env.no_proxy]).toEqual(["127.0.0.1", "127.0.0.1"]); + }); + + it("preserves existing entries", () => { + reset("example.internal"); + keepLoopbackOffProxy("http://localhost:4318/v1/traces"); + expect([process.env.NO_PROXY, process.env.no_proxy]).toEqual(["example.internal,localhost", "example.internal,localhost"]); + }); + + it("leaves both variables exactly as they were for a non-loopback endpoint", () => { + reset(undefined); + keepLoopbackOffProxy("http://collector.example.com:4318/v1/traces"); + expect("NO_PROXY" in process.env).toBe(false); + expect("no_proxy" in process.env).toBe(false); + reset("example.internal"); + keepLoopbackOffProxy("http://collector.example.com:4318/v1/traces"); + expect([process.env.NO_PROXY, process.env.no_proxy]).toEqual(["example.internal", "example.internal"]); + }); +}); diff --git a/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl b/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl new file mode 100644 index 00000000..437a8f22 --- /dev/null +++ b/reference-implementations/agt/samples/trace/acs-trace-sample.jsonl @@ -0,0 +1,2 @@ +{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"acs-reference-guardian"}}],"droppedAttributesCount":0},"scopeSpans":[{"scope":{"name":"acs-reference-guardian"},"spans":[{"traceId":"9785c34288d27dbd92fcf12e53ac3315","spanId":"31258fb4b343b292","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065946937000000","endTimeUnixNano":"1792065947036723875","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}},{"key":"acs.capability","value":{"stringValue":"process.exec"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"allow"}}],"name":"acs.decision","timeUnixNano":"1792065947036676541","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"112037a108c3893225309fa193536b77","spanId":"46883c95863bb8fb","name":"gen_ai.tool.result","kind":1,"startTimeUnixNano":"1792065947038000000","endTimeUnixNano":"1792065947088187667","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b02"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}},{"key":"acs.exit_status","value":{"stringValue":"success"}},{"key":"acs.request_id_ref","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"allow"}}],"name":"acs.decision","timeUnixNano":"1792065947088182292","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"981a929f7cea4c999642a38dd3d822b1","spanId":"3b199edb6364b94d","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065947089000000","endTimeUnixNano":"1792065947141424750","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b03"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}},{"key":"acs.provenance.origin","value":{"arrayValue":{"values":[{"stringValue":"user_input"},{"stringValue":"agent_generated"}]}}},{"key":"acs.provenance.source_id","value":{"arrayValue":{"values":[{"stringValue":"sample-user-1"},{"stringValue":""}]}}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"deny"}},{"key":"acs.reasoning","value":{"stringValue":"This command was blocked because it matches a destructive-shell-command pattern. Policy: destructive_shell_command_blocked, from AGT's stock bundle (agt_stock). Matched at offset 0."}},{"key":"acs.reason_codes","value":{"arrayValue":{"values":[{"stringValue":"destructive_shell_command_blocked"}]}}},{"key":"acs.policy.ids","value":{"arrayValue":{"values":[{"stringValue":"agt_stock"}]}}},{"key":"acs.policy.versions","value":{"arrayValue":{"values":[{"stringValue":""}]}}},{"key":"acs.policy.rule_ids","value":{"arrayValue":{"values":[{"stringValue":"destructive_shell_command_blocked"}]}}}],"name":"acs.decision","timeUnixNano":"1792065947141419958","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"95c92494c004b23662900ae8c17ea04c","spanId":"02d46f81d8206a0f","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065947141000000","endTimeUnixNano":"1792065947141270000","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b04"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.failure.class","value":{"stringValue":"guardian_error"}},{"key":"acs.failure.posture","value":{"stringValue":"fail_open"}},{"key":"acs.failure.reported_by","value":{"stringValue":"guardian"}},{"key":"acs.failure.error_code","value":{"intValue":-32010}},{"key":"acs.failure.elapsed_ms","value":{"intValue":0}},{"key":"acs.failure.payload_bytes","value":{"intValue":340}}],"name":"acs.decision.failure","timeUnixNano":"1792065947141268125","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":2,"message":"The Guardian returned an error instead of a decision."},"links":[],"droppedLinksCount":0,"flags":257},{"traceId":"e5703a6e87d75324de28fd7cc555724b","spanId":"e813670f52f6a811","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065947142000000","endTimeUnixNano":"1792065949203024291","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b05"}},{"key":"gen_ai.tool.name","value":{"stringValue":"Bash"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.decision","value":{"stringValue":"allow"}}],"name":"acs.decision","timeUnixNano":"1792065949202988875","droppedAttributesCount":0},{"attributes":[{"key":"acs.failure.class","value":{"stringValue":"timeout"}},{"key":"acs.failure.posture","value":{"stringValue":"fail_open"}},{"key":"acs.failure.reported_by","value":{"stringValue":"guardian"}},{"key":"acs.failure.timeout_ms","value":{"intValue":1500}},{"key":"acs.failure.elapsed_ms","value":{"intValue":2061}},{"key":"acs.failure.payload_bytes","value":{"intValue":367}}],"name":"acs.decision.failure","timeUnixNano":"1792065949203022666","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":0},"links":[],"droppedLinksCount":0,"flags":257}]}]}]} +{"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"acs-reference-audit-converter"}}],"droppedAttributesCount":0},"scopeSpans":[{"scope":{"name":"acs-reference-audit-converter"},"spans":[{"traceId":"9b082079ac3695104b2da1df27340e16","spanId":"99e8d283763d16d3","name":"gen_ai.tool.call","kind":1,"startTimeUnixNano":"1792065949205000000","endTimeUnixNano":"1792065949205000000","attributes":[{"key":"acs.session.id","value":{"stringValue":"5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"}},{"key":"acs.request_id","value":{"stringValue":"1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b06"}}],"droppedAttributesCount":0,"events":[{"attributes":[{"key":"acs.failure.class","value":{"stringValue":"unreachable"}},{"key":"acs.failure.posture","value":{"stringValue":"fail_open"}},{"key":"acs.failure.reported_by","value":{"stringValue":"observed_agent"}},{"key":"acs.failure.detail","value":{"stringValue":"The host could not reach the Guardian."}}],"name":"acs.decision.failure","timeUnixNano":"1792065949205000000","droppedAttributesCount":0}],"droppedEventsCount":0,"status":{"code":2,"message":"The host could not reach the Guardian."},"links":[],"droppedLinksCount":0,"flags":257}]}]}]} diff --git a/reference-implementations/agt/scripts/make-sample-trace.ts b/reference-implementations/agt/scripts/make-sample-trace.ts new file mode 100644 index 00000000..de4d5e4a --- /dev/null +++ b/reference-implementations/agt/scripts/make-sample-trace.ts @@ -0,0 +1,190 @@ +/** + * Writes the committed sample trace (issue #266): what the reference + * Guardian's OTLP export actually looks like, from a fixed set of synthetic + * requests through the two live hook methods, plus one host-side failure run + * through the audit converter. + * + * bun run sample:trace + * + * Decisions come from the real AGT policy in policy/manifest.yaml. The one + * change is a delay on the request carrying SLOW_MARKER, so that decision + * finishes after the short timeout this script negotiates and the sample + * shows a `timeout` failure. Every id is fixed and every value synthetic: + * the sample is committed, and `acs.reasoning` and `acs.provenance.source_id` + * go out as written. + */ +import { appendFileSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { convertAuditLog } from "../packages/guardian/src/audit-to-otlp.ts"; +import { createDeploymentBridge } from "../packages/guardian/src/deployment-bridge.ts"; +import { buildServerHello } from "../packages/guardian/src/handshake.ts"; +import { startGuardian } from "../packages/guardian/src/index.ts"; +import { createOtelTraceExporter } from "../packages/guardian/src/otel-trace-exporter.ts"; + +export const SAMPLE_PATH = fileURLToPath(new URL("../samples/trace/acs-trace-sample.jsonl", import.meta.url)); +const MANIFEST_PATH = fileURLToPath(new URL("../policy/manifest.yaml", import.meta.url)); + +/** How long the host waits for a decision in this sample, and how long the + * marked request makes the Guardian take. Far enough apart that the stock + * policy's own evaluation time never crosses the timeout. */ +const TIMEOUT_MS = 1500; +const SLOW_MS = 2000; +const SLOW_MARKER = "sample-slow-marker"; + +const SESSION_ID = "5a3c1e7e-0b1d-4c6a-9e2f-7d4b8c0a1f01"; + +export type Scenario = { + name: string; + requestId: string; + /** The `capability` the request carried, if any. */ + capability?: string; + /** The `request_id_ref` a result carried, if any. */ + requestIdRef?: string; + /** The decision the Guardian answered with, or none when it answered an + * error or was never reached. */ + decision?: "allow" | "deny"; +}; + +/** Every step the sample holds, keyed by the request id its span carries. */ +export const SCENARIOS: readonly Scenario[] = [ + { name: "allowed", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01", capability: "process.exec", decision: "allow" }, + { name: "result", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b02", requestIdRef: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b01", decision: "allow" }, + { name: "denied", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b03", decision: "deny" }, + { name: "error", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b04" }, + { name: "late", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b05", decision: "allow" }, + { name: "unreachable", requestId: "1b6f0c2a-3d4e-4f5a-8b6c-7d8e9f0a1b06" }, +]; + +function requestIdOf(name: string): string { + return SCENARIOS.find((scenario) => scenario.name === name)!.requestId; +} + +function envelope(method: string, requestId: string, payload: Record, id: number | null): Record { + return { + jsonrpc: "2.0", + method, + ...(id === null ? {} : { id }), + params: { + acs_version: "0.1.0", + request_id: requestId, + timestamp: new Date().toISOString(), + metadata: { agent_id: "sample-agent", session_id: SESSION_ID }, + payload, + }, + }; +} + +function bash(command: string, extra: Record = {}): Record { + return { tool: { name: "Bash" }, arguments: { command: { value: command } }, ...extra }; +} + +/** The envelopes the Guardian receives, in order. */ +function requests(): Record[] { + return [ + envelope("steps/toolCallRequest", requestIdOf("allowed"), bash("ls -la", { capability: "process.exec" }), 1), + envelope( + "steps/toolCallResult", + requestIdOf("result"), + { + tool: { name: "Bash" }, + exit_status: "success", + outputs: [{ value: "README.md\nsrc" }], + request_id_ref: requestIdOf("allowed"), + }, + 2, + ), + // Two Provenance objects, one with a source_id and one without, so the + // source_id column shows its "" padding. + envelope( + "steps/toolCallRequest", + requestIdOf("denied"), + { + tool: { name: "Bash" }, + arguments: { + command: { + // Matches the stock policy's destructive-command rule without + // naming a real path. + value: "rm -rf .", + provenance: { provenance_id: "prov-sample-1", origin: "user_input", source_id: "sample-user-1" }, + }, + note: { value: "cleanup", provenance: { provenance_id: "prov-sample-2", origin: "agent_generated" } }, + }, + }, + 3, + ), + // No JSON-RPC id: the Guardian answers with an error, not a decision. + envelope("steps/toolCallRequest", requestIdOf("error"), bash("pwd"), null), + envelope("steps/toolCallRequest", requestIdOf("late"), bash(`echo ${SLOW_MARKER}`), 5), + ]; +} + +/** The host's record of a step that never reached the Guardian, in the + * audit-log format packages/host-adapter/src/audit-sink.ts writes. */ +function unreachableAuditLine(): string { + return `${JSON.stringify({ + seq: 1, + recorded_at: new Date().toISOString(), + session_id: SESSION_ID, + method: "steps/toolCallRequest", + rpc_id: requestIdOf("unreachable"), + outcome: "proceeded", + posture: "proceed", + posture_source: "negotiated", + failure: { kind: "transport", message: "connect ECONNREFUSED 127.0.0.1:8787" }, + })}\n`; +} + +export async function generateSampleTrace(outPath: string = SAMPLE_PATH): Promise { + const dir = mkdtempSync(join(tmpdir(), "acs-make-sample-trace-")); + try { + const guardianTrace = join(dir, "guardian.jsonl"); + const auditPath = join(dir, "audit.jsonl"); + const converterTrace = join(dir, "converter.jsonl"); + + // The bridge the Guardian itself builds, built-in annotator included: the + // stock manifest declares an egress annotator, and a bridge without it + // fails every step closed, which is not what a sample should show. + const real = createDeploymentBridge(MANIFEST_PATH); + const bridge = { + async evaluate(point: string, snapshot: Record) { + if (JSON.stringify(snapshot).includes(SLOW_MARKER)) { + await Bun.sleep(SLOW_MS); + } + return real.evaluate(point, snapshot); + }, + }; + const traceExporter = createOtelTraceExporter({ + filePath: guardianTrace, + methodsEvaluated: buildServerHello().methods_evaluated, + timeoutConfig: { default_ms: TIMEOUT_MS }, + posture: "fail_open", + }); + const guardian = await startGuardian({ port: 0, manifestPath: MANIFEST_PATH, bridge, traceExporter }); + try { + for (const request of requests()) { + await fetch(guardian.url, { method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(request) }); + } + } finally { + // Closing the Guardian shuts the exporter down, which flushes the file. + await guardian.close(); + } + + writeFileSync(auditPath, unreachableAuditLine()); + const summary = await convertAuditLog(auditPath, converterTrace); + if (summary.written !== 1) { + throw new Error(`the audit converter wrote ${summary.written} span(s), expected 1`); + } + + writeFileSync(outPath, readFileSync(guardianTrace, "utf8")); + appendFileSync(outPath, readFileSync(converterTrace, "utf8")); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +} + +if (import.meta.main) { + await generateSampleTrace(); + console.log(`wrote ${SAMPLE_PATH}`); +} diff --git a/reference-implementations/agt/test/audit-to-otlp.test.ts b/reference-implementations/agt/test/audit-to-otlp.test.ts index b898ede5..5d8896db 100644 --- a/reference-implementations/agt/test/audit-to-otlp.test.ts +++ b/reference-implementations/agt/test/audit-to-otlp.test.ts @@ -1,16 +1,17 @@ -import { describe, expect, it } from "bun:test"; -import { appendFileSync, existsSync, mkdtempSync, readFileSync, rmdirSync, unlinkSync } from "node:fs"; +import { afterEach, describe, expect, it } from "bun:test"; +import { appendFileSync, existsSync, linkSync, mkdtempSync, readFileSync, rmdirSync, rmSync, statSync, symlinkSync, unlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { auditUngovernedStep, createAuditSink, createGuardianClient, governStep, toSessionUuid, type Hookmap } from "host-adapter"; import { + CONVERTER_SERVICE_NAME, convertAuditLog, FAILURE_CLASS_BY_KIND, FAILURE_DETAIL_BY_KIND, failureClassOf, sessionUuid as converterSessionUuid, } from "../packages/guardian/src/audit-to-otlp.ts"; -import { loadOtelMapping } from "../packages/guardian/src/otel-trace-exporter.ts"; +import { loadOtelMapping, OTEL_SERVICE_NAME, SPAN_LIMITS } from "../packages/guardian/src/otel-trace-exporter.ts"; /** * The converter's contract, measured with the Guardian down. The host @@ -331,3 +332,157 @@ describe("audit-to-otlp: review fixes", () => { } }); }); + +// Issue #276 item 4 and the issue #274 service name: the converter's output +// must not depend on OTEL_* variables in the shell that runs it, and its +// spans carry their own producer label. +describe("audit-to-otlp: output does not depend on the environment", () => { + const AUDIT_LINE = `${JSON.stringify({ + seq: 1, + recorded_at: "2026-10-15T00:00:00.000Z", + session_id: "sess-1", + method: "steps/toolCallRequest", + rpc_id: "3f6c1e2a-8b4d-4c1e-9a7f-2d5b6e8c9a01", + outcome: "proceeded", + posture: "proceed", + posture_source: "negotiated", + failure: { kind: "transport", message: "connect ECONNREFUSED 127.0.0.1:1" }, + })}\n`; + const HOSTILE: Record = { + OTEL_ATTRIBUTE_VALUE_LENGTH_LIMIT: "8", + OTEL_SPAN_EVENT_COUNT_LIMIT: "0", + OTEL_SPAN_ATTRIBUTE_PER_EVENT_COUNT_LIMIT: "1", + OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT: "1", + OTEL_TRACES_SAMPLER: "always_off", + OTEL_SERVICE_NAME: "zzz", + OTEL_RESOURCE_ATTRIBUTES: "service.name=evil", + }; + const saved: Record = {}; + afterEach(() => { + for (const [key, value] of Object.entries(saved)) { + if (value === undefined) delete process.env[key]; + else process.env[key] = value; + } + }); + + it("keeps full ids, the whole failure event, and the converter's own label", async () => { + for (const [key, value] of Object.entries(HOSTILE)) { + saved[key] = process.env[key]; + process.env[key] = value; + } + await withScratch(async (auditPath, otelPath) => { + appendFileSync(auditPath, AUDIT_LINE); + const summary = await convertAuditLog(auditPath, otelPath); + expect(summary.written).toBe(1); + const request = JSON.parse(readFileSync(otelPath, "utf8").split("\n")[0]!); + const resourceSpans = request.resourceSpans[0]; + expect(resourceSpans.resource.attributes).toEqual([{ key: "service.name", value: { stringValue: CONVERTER_SERVICE_NAME } }]); + expect(resourceSpans.scopeSpans[0].scope.name).toBe(CONVERTER_SERVICE_NAME); + const span = resourceSpans.scopeSpans[0].spans[0]; + expect(attribute(span, "acs.request_id")).toBe("3f6c1e2a-8b4d-4c1e-9a7f-2d5b6e8c9a01"); + expect(attribute(span, "acs.session.id")).toBe(converterSessionUuid("sess-1")); + expect(span.droppedAttributesCount ?? 0).toBe(0); + const event = span.events.find((e: OtlpEvent) => e.name === "acs.decision.failure"); + expect(event).toBeDefined(); + expect(event.droppedAttributesCount ?? 0).toBe(0); + expect(attribute(event, "acs.failure.class")).toBe("unreachable"); + expect(attribute(event, "acs.failure.posture")).toBe("fail_open"); + expect(attribute(event, "acs.failure.reported_by")).toBe("observed_agent"); + expect(attribute(event, "acs.failure.detail")).toBe(FAILURE_DETAIL_BY_KIND.transport); + }); + }, 10_000); + + it("labels its spans apart from the Guardian's", () => { + expect(CONVERTER_SERVICE_NAME).toBe("acs-reference-audit-converter"); + expect(CONVERTER_SERVICE_NAME).not.toBe(OTEL_SERVICE_NAME); + }); + + it("shares one frozen set of span limits with the Guardian", () => { + expect(Object.isFrozen(SPAN_LIMITS)).toBe(true); + expect(SPAN_LIMITS).toEqual({ + attributeValueLengthLimit: 4096, + attributeCountLimit: 128, + eventCountLimit: 128, + attributePerEventCountLimit: 128, + linkCountLimit: 128, + attributePerLinkCountLimit: 128, + }); + }); +}); + +// Converting onto an existing file appended OTLP lines to it. Onto the audit +// log itself (or a hardlink of it) that corrupts the evidence and shifts the +// next audit seq. Twice over the same output it doubles every span. Onto the +// Guardian's trace file it merges the two streams. One rule covers all of them. +describe("audit-to-otlp: refuses an output that already exists", () => { + const LINE = `${JSON.stringify({ + seq: 1, + recorded_at: "2026-10-15T00:00:00.000Z", + session_id: "sess-1", + method: "steps/toolCallRequest", + rpc_id: "3f6c1e2a-8b4d-4c1e-9a7f-2d5b6e8c9a01", + outcome: "proceeded", + posture: "proceed", + posture_source: "negotiated", + failure: { kind: "transport", message: "connect ECONNREFUSED 127.0.0.1:1" }, + })}\n`; + + /** A fresh directory with the audit log in it, removed whatever the test does. */ + async function inScratch(run: (dir: string, audit: string) => Promise): Promise { + const dir = mkdtempSync(join(tmpdir(), "acs-audit-exclusive-")); + const audit = join(dir, "audit.jsonl"); + writeFileSync(audit, LINE); + try { + await run(dir, audit); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + } + + it("refuses to convert a log onto itself and leaves it byte-identical", () => + inScratch(async (_dir, audit) => { + await expect(convertAuditLog(audit, audit)).rejects.toThrow(/^refusing to write/); + expect(readFileSync(audit, "utf8")).toBe(LINE); + })); + + it("refuses a hardlink to the log", () => + inScratch(async (dir, audit) => { + const link = join(dir, "otel.jsonl"); + linkSync(audit, link); + await expect(convertAuditLog(audit, link)).rejects.toThrow(/^refusing to write/); + expect(readFileSync(audit, "utf8")).toBe(LINE); + })); + + it("refuses an existing unrelated output, such as the Guardian's trace file", () => + inScratch(async (dir, audit) => { + const guardianTrace = join(dir, "otel-traces.jsonl"); + writeFileSync(guardianTrace, "guardian line\n"); + await expect(convertAuditLog(audit, guardianTrace)).rejects.toThrow(/^refusing to write/); + expect(readFileSync(guardianTrace, "utf8")).toBe("guardian line\n"); + })); + + it("refuses a dangling symlink and creates nothing at its target", () => + inScratch(async (dir, audit) => { + const target = join(dir, "elsewhere.jsonl"); + const link = join(dir, "otel.jsonl"); + symlinkSync(target, link); + await expect(convertAuditLog(audit, link)).rejects.toThrow(/^refusing to write/); + expect(existsSync(target)).toBe(false); + })); + + it("leaves no output behind when the audit log cannot be read, so the corrected rerun is not refused", () => + inScratch(async (dir) => { + const out = join(dir, "otel.jsonl"); + await expect(convertAuditLog(join(dir, "missing.jsonl"), out)).rejects.toThrow(); + expect(existsSync(out)).toBe(false); + })); + + it("writes a new output owner-only, creating its directory owner-only", () => + inScratch(async (dir, audit) => { + const out = join(dir, "new-dir", "otel.jsonl"); + const summary = await convertAuditLog(audit, out); + expect(summary.written).toBe(1); + expect(statSync(out).mode & 0o777).toBe(0o600); + expect(statSync(join(dir, "new-dir")).mode & 0o777).toBe(0o700); + })); +}); diff --git a/reference-implementations/agt/test/collector-config.test.ts b/reference-implementations/agt/test/collector-config.test.ts new file mode 100644 index 00000000..85b57b12 --- /dev/null +++ b/reference-implementations/agt/test/collector-config.test.ts @@ -0,0 +1,59 @@ +import { describe, expect, it } from "bun:test"; +import { readFileSync } from "node:fs"; +import { fileURLToPath } from "node:url"; + +/** + * otel-collector.yaml as committed (issue #276). Structure only: the proof + * that otelcol-contrib 0.162.0 accepts it and behaves is the end-to-end run + * in the PR. What this pins is what a later edit could quietly undo. + */ +const CONFIG_PATH = fileURLToPath(new URL("../otel-collector.yaml", import.meta.url)); +const IMAGE = "otel/opentelemetry-collector-contrib:0.162.0@sha256:39923a8e431bd1f57be82411999d389fcfe40857492e4365456d97a4c1f74be6"; +const text = readFileSync(CONFIG_PATH, "utf8"); +const config = Bun.YAML.parse(text) as { + receivers: { otlp: { protocols: { http: { endpoint: string } } } }; + exporters: { file: { path: string; append?: unknown; create_directory?: unknown; directory_permissions?: unknown } }; + service: { telemetry?: { metrics?: { level?: string } } }; +}; + +describe("otel-collector.yaml", () => { + it("listens on loopback only, as a literal: an empty env var would bind every interface", () => { + expect(config.receivers.otlp.protocols.http.endpoint).toBe("127.0.0.1:4318"); + expect(text).not.toMatch(/\$\{/); + }); + + it("publishes the container's port on host loopback only", () => { + expect(text).toContain("-p 127.0.0.1:4318:4318"); + expect(text).not.toMatch(/-p 4318:4318/); + }); + + it("appends across restarts and creates its directory owner-only", () => { + expect(config.exporters.file.append).toBe(true); + expect(config.exporters.file.create_directory).toBe(true); + expect(config.exporters.file.directory_permissions).toBe("0700"); + }); + + it("turns off its own metrics endpoint, so port 8888 cannot clash", () => { + expect(config.service.telemetry?.metrics?.level).toBe("none"); + }); + + it("names the digest-pinned image", () => { + expect(text).toContain(IMAGE); + }); +}); + +const README_PATH = fileURLToPath(new URL("../README.md", import.meta.url)); +const readme = readFileSync(README_PATH, "utf8"); + +describe("the walkthrough's collector step", () => { + it("names the same digest-pinned image and loopback-only port as the config", () => { + expect(readme).toContain(IMAGE); + expect(readme).toContain("-p 127.0.0.1:4318:4318"); + }); + + it("states what was verified, with no unfilled placeholder", () => { + expect(readme).not.toContain("[run on"); + expect(readme).not.toContain("| not run yet]"); + expect(readme).toContain("with the Docker command above on Docker Desktop for Mac"); + }); +}); diff --git a/reference-implementations/agt/test/sample-trace.test.ts b/reference-implementations/agt/test/sample-trace.test.ts new file mode 100644 index 00000000..dbc86ab5 --- /dev/null +++ b/reference-implementations/agt/test/sample-trace.test.ts @@ -0,0 +1,215 @@ +import { describe, expect, it } from "bun:test"; +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { CONVERTER_SERVICE_NAME } from "../packages/guardian/src/audit-to-otlp.ts"; +import { OTEL_SERVICE_NAME } from "../packages/guardian/src/otel-trace-exporter.ts"; +import { generateSampleTrace, SAMPLE_PATH, SCENARIOS } from "../scripts/make-sample-trace.ts"; + +/** + * The committed sample trace (issue #266), held to the mapping it claims to + * follow, and to the exporter that wrote it. If the exporter stops emitting + * an attribute the mapping requires or makes conditional, every unit test can + * still pass; a test over real output cannot. + */ + +const MAPPING_PATH = new URL("../../../specification/v0.1.0/trace/otel-mapping.json", import.meta.url); + +type OtlpValue = { stringValue?: string; intValue?: string | number; arrayValue?: { values?: OtlpValue[] } }; +type OtlpAttribute = { key: string; value: OtlpValue }; +type OtlpEvent = { name: string; timeUnixNano?: string; attributes?: OtlpAttribute[] }; +type OtlpSpan = { + name: string; + traceId?: string; + spanId?: string; + startTimeUnixNano?: string; + endTimeUnixNano?: string; + attributes?: OtlpAttribute[]; + events?: OtlpEvent[]; + status?: { code?: number; message?: string }; +}; +type OtlpRequest = { + resourceSpans: { + resource?: { attributes?: OtlpAttribute[] }; + scopeSpans: { scope?: { name?: string }; spans: OtlpSpan[] }[]; + }[]; +}; +type LabeledSpan = OtlpSpan & { service?: string; scope?: string }; + +function spansIn(path: string | URL): LabeledSpan[] { + return readFileSync(path, "utf8") + .split("\n") + .filter((line) => line.trim() !== "") + .flatMap((line) => + (JSON.parse(line) as OtlpRequest).resourceSpans.flatMap((rs) => + rs.scopeSpans.flatMap((ss) => + ss.spans.map((span) => ({ + ...span, + service: rs.resource?.attributes?.find((a) => a.key === "service.name")?.value.stringValue, + scope: ss.scope?.name, + })), + ), + ), + ); +} + +function plain(value: OtlpValue): unknown { + if (value.arrayValue !== undefined) return (value.arrayValue.values ?? []).map(plain); + if (value.intValue !== undefined) return Number(value.intValue); + return value.stringValue; +} + +function attributes(carrier: { attributes?: OtlpAttribute[] } | undefined): Record { + return Object.fromEntries((carrier?.attributes ?? []).map((a) => [a.key, plain(a.value)])); +} + +/** What a span says once the values that differ between runs are taken out: + * ids, timestamps, elapsed time, and which export batch it landed in. */ +function normalize(spans: LabeledSpan[]): unknown[] { + return spans + .map((span) => ({ + service: span.service, + scope: span.scope, + name: span.name, + attributes: attributes(span), + status: span.status ?? {}, + events: (span.events ?? []).map((event) => { + const { "acs.failure.elapsed_ms": _elapsed, ...rest } = attributes(event); + return { name: event.name, attributes: rest }; + }), + })) + .sort((a, b) => JSON.stringify(a).localeCompare(JSON.stringify(b))); +} + +type MappingSchema = { + properties: { + step_to_span: { default: Record }; + step_span_attributes: { properties: { required_attributes: { default: string[] } } }; + decision_failure_event: { properties: { attribute_values: { default: Record } } }; + }; +}; +const MAPPING = JSON.parse(readFileSync(MAPPING_PATH, "utf8")) as MappingSchema; +const STEP_BY_SPAN_NAME = Object.fromEntries( + Object.entries(MAPPING.properties.step_to_span.default).map(([method, entry]) => [entry.span_name, entry]), +); + +/** Required attributes a span reported by the host audit converter cannot + * carry, and why. Anything not named here is required of every span. */ +const CONVERTER_EXEMPT: Readonly> = { + "gen_ai.tool.name": "the host audit log records no tool name (issue #274)", +}; + +const SPANS = spansIn(SAMPLE_PATH); + +function spanFor(requestId: string): OtlpSpan | undefined { + return SPANS.find((span) => attributes(span)["acs.request_id"] === requestId); +} + +function failureOf(span: OtlpSpan | undefined): Record | undefined { + const event = span?.events?.find((e) => e.name === "acs.decision.failure"); + return event === undefined ? undefined : attributes(event); +} + +describe("the committed sample trace matches the mapping", () => { + it("names every span with a step_to_span value", () => { + expect(SPANS.length).toBeGreaterThan(0); + for (const span of SPANS) { + expect({ name: span.name, mapped: span.name in STEP_BY_SPAN_NAME }).toEqual({ name: span.name, mapped: true }); + } + }); + + it("labels converter spans as the converter and every other span as the Guardian", () => { + for (const span of SPANS) { + const fromConverter = failureOf(span)?.["acs.failure.reported_by"] === "observed_agent"; + const expected = fromConverter ? CONVERTER_SERVICE_NAME : OTEL_SERVICE_NAME; + expect({ span: span.name, service: span.service, scope: span.scope }).toEqual({ span: span.name, service: expected, scope: expected }); + } + }); + + it("carries every required attribute, apart from the named converter exemptions", () => { + const always = MAPPING.properties.step_span_attributes.properties.required_attributes.default; + for (const span of SPANS) { + const present = attributes(span); + const fromConverter = failureOf(span)?.["acs.failure.reported_by"] === "observed_agent"; + for (const attribute of [...always, ...STEP_BY_SPAN_NAME[span.name]!.required_attributes]) { + if (fromConverter && attribute in CONVERTER_EXEMPT) continue; + expect({ span: span.name, attribute, present: attribute in present }).toEqual({ span: span.name, attribute, present: true }); + } + } + }); + + it("carries a conditional attribute exactly when the request carried its source", () => { + for (const scenario of SCENARIOS) { + const present = attributes(spanFor(scenario.requestId)); + expect({ scenario: scenario.name, capability: "acs.capability" in present }).toEqual({ + scenario: scenario.name, + capability: scenario.capability !== undefined, + }); + expect({ scenario: scenario.name, ref: "acs.request_id_ref" in present }).toEqual({ + scenario: scenario.name, + ref: scenario.requestIdRef !== undefined, + }); + } + }); + + // A sample whose decisions are all policy failures would pass every check + // above and still show a reader nothing true, so each one is pinned. + it("records the decision each scenario was built to produce, and none from a policy failure", () => { + for (const scenario of SCENARIOS) { + const decision = attributes(spanFor(scenario.requestId)?.events?.find((e) => e.name === "acs.decision")); + expect({ scenario: scenario.name, decision: decision["acs.decision"] }).toEqual({ scenario: scenario.name, decision: scenario.decision }); + expect(String(decision["acs.reason_codes"] ?? "")).not.toContain("runtime_error"); + } + }); + + it("links the result to its request through acs.request_id_ref", () => { + const result = SCENARIOS.find((scenario) => scenario.requestIdRef !== undefined)!; + expect(attributes(spanFor(result.requestId))["acs.request_id_ref"]).toBe(result.requestIdRef); + expect(spanFor(result.requestIdRef!)).toBeDefined(); + }); + + // This Guardian always names a rule and never sends policy_version, so the + // padding it exercises is the version column: one "" per reference. + it("pads the policy and provenance columns to a common length", () => { + const denied = attributes(spanFor(SCENARIOS.find((s) => s.name === "denied")!.requestId)); + const decision = attributes(spanFor(SCENARIOS.find((s) => s.name === "denied")!.requestId)?.events?.find((e) => e.name === "acs.decision")); + const ids = decision["acs.policy.ids"] as string[]; + expect(ids.length).toBeGreaterThan(0); + expect((decision["acs.policy.versions"] as string[]).length).toBe(ids.length); + expect((decision["acs.policy.rule_ids"] as string[]).length).toBe(ids.length); + expect(decision["acs.policy.versions"]).toEqual(ids.map(() => "")); + expect((denied["acs.provenance.source_id"] as string[]).length).toBe((denied["acs.provenance.origin"] as string[]).length); + expect(denied["acs.provenance.source_id"]).toContain(""); + }); + + it("uses only failure classes, postures and reporters the mapping lists, and covers the four failures", () => { + const allowed = MAPPING.properties.decision_failure_event.properties.attribute_values.default; + const seen = new Set(); + for (const span of SPANS) { + const failure = failureOf(span); + if (failure === undefined) continue; + for (const key of ["acs.failure.class", "acs.failure.posture", "acs.failure.reported_by"]) { + expect({ key, value: failure[key], listed: allowed[key]!.includes(String(failure[key])) }).toEqual({ + key, + value: failure[key], + listed: true, + }); + } + seen.add(`${failure["acs.failure.class"]}/${failure["acs.failure.reported_by"]}`); + } + expect([...seen].sort()).toEqual(["guardian_error/guardian", "timeout/guardian", "unreachable/observed_agent"]); + }); +}); + +describe("the committed sample trace is current", () => { + it("is what the generator writes today, once ids and timestamps are set aside", async () => { + const dir = mkdtempSync(join(tmpdir(), "acs-sample-trace-")); + try { + const fresh = join(dir, "sample.jsonl"); + await generateSampleTrace(fresh); + expect(normalize(spansIn(fresh))).toEqual(normalize(SPANS)); + } finally { + rmSync(dir, { recursive: true, force: true }); + } + }, 30_000); +}); diff --git a/tests/test_trace_docs_example.py b/tests/test_trace_docs_example.py new file mode 100644 index 00000000..fe7e075b --- /dev/null +++ b/tests/test_trace_docs_example.py @@ -0,0 +1,56 @@ +"""The example span on the Trace with OpenTelemetry page is real exporter output. + +The page shows one span from the committed sample trace +(reference-implementations/agt/samples/trace/acs-trace-sample.jsonl), so a +reader sees what the reference Guardian actually emits. An example copied by +hand drifts the first time the exporter or the mapping changes, so this test +requires it to match the sample's span with the same acs.request_id, once +the values that differ between runs (ids and timestamps) are set aside. When +the sample is regenerated, the example has to be updated with it. +""" + +import json +import re +from pathlib import Path + +ROOT = Path(__file__).resolve().parent.parent +PAGE = ROOT / "docs" / "spec" / "trace" / "extend_opentelemetry.md" +SAMPLE = ROOT / "reference-implementations" / "agt" / "samples" / "trace" / "acs-trace-sample.jsonl" +HEADING = "## Example: a denied tool call" +VOLATILE = {"traceId", "spanId", "parentSpanId", "startTimeUnixNano", "endTimeUnixNano", "timeUnixNano"} + + +def _strip(value): + if isinstance(value, dict): + return {k: _strip(v) for k, v in value.items() if k not in VOLATILE} + if isinstance(value, list): + return [_strip(v) for v in value] + return value + + +def _request_id(span): + for attribute in span.get("attributes", []): + if attribute["key"] == "acs.request_id": + return attribute["value"]["stringValue"] + return None + + +def _sample_spans(): + for line in SAMPLE.read_text(encoding="utf-8").splitlines(): + if line.strip(): + for resource in json.loads(line)["resourceSpans"]: + for scope in resource["scopeSpans"]: + yield from scope["spans"] + + +def test_docs_example_span_matches_the_committed_sample(): + text = PAGE.read_text(encoding="utf-8") + assert HEADING in text, f"{PAGE.name} has no '{HEADING}' section" + block = re.search(r"```json\n(.*?)```", text[text.index(HEADING):], re.S) + assert block, f"the '{HEADING}' section has no JSON example" + example = json.loads(block.group(1)) + + request_id = _request_id(example) + matches = [span for span in _sample_spans() if _request_id(span) == request_id] + assert len(matches) == 1, f"the sample has no single span with acs.request_id {request_id}" + assert _strip(example) == _strip(matches[0]), "the docs example no longer matches the sample; copy the span from the regenerated sample"