[Bug] §6.4's mandatory fail-open audit event has no conformant shape, so the decision-failure rate is unmeasurable

@astrogilda

Found while checking §6.4 against infrastructure measurements from an agent evaluation I ran; the numbers are below.

The gap

§6.4 requires: "Every step that proceeds without a decision MUST be recorded as an audit event, so the bypass is visible rather than silent." I went looking for where that event can conformantly live, and there is no answer:

The SessionContext chain can't hold it. context-entry.json defines step_type as "the method name from the originating request envelope," and a fail-open proceed is not a method. Structurally worse: the chain is written by the Guardian (§8), and the defining feature of a decision failure is that the Guardian produced nothing for this step.
The Trace vocabulary can't hold it. The acs.decision span event enumerates exactly the five dispositions, and the OCSF severity mapping covers only those five. "Proceeded without a decision" is not representable.
sessionEnd has no field for it. Its summary object carries outcome, step_count, errors[]. No failure classes, no counts.

So the spec contains a MUST that no deployment can satisfy in a way another party could consume. It has also already solved this exact problem once, at the handshake: §4.1 says a session started unguarded "MUST be recorded in the deployment's own audit log — there is no Guardian to receive the event — and SHOULD be surfaced on Trace events when the deployment claims ACS-Trace." That pattern stops at the handshake and never reaches per-step decision failures, which is where the volume is.

This bears on #32. The proposal there to flip the default rests on "the audit event already records every fail-open proceed, so the choice remains observable." As of v0.1.1 that event has no defined shape, so the premise is asserted rather than specified. The default question and this issue are separable: either answer to #32 needs the event to exist first.

Why the rate matters: measured data

§6.4 prices the trade qualitatively ("an adversary who can disrupt the channel converts control into audit"), and the handshake schema prices timeout latency per millisecond. Nothing anywhere lets a deployment or an auditor learn what fraction of steps resolved by posture instead of decision. From what I measured, that fraction is not an edge case on realistic infrastructure.

Over ten weeks I evaluated an LLM agent on a self-hosted serving stack (~120B open-weight model, 4-bit quantized, behind a reverse proxy; data could not leave the network). A production Guardian under §12.2 has the same shape: an LLM evaluator receiving full hook payloads, and agentResponse, toolCallResult, and knowledgeRetrieval all carry content. What I measured:

Same 86 KB payload, 10 sends, 2 s apart: 4 of 10 returned 504 at the proxy's ~60 s upstream timeout; 6 of 10 returned 200 in 25–47 s. The stack was healthy at ~46 KB bodies; the failing calls were 83–86 KB. Payload size, not load, was the controlling variable.
Effective per-step failure rate: 16% with a single retry, 2.56% after retry-with-backoff (3 attempts, 10/20/40 s).
In one evaluation run, 3 "final answers" were verbatim proxy error pages (504 Gateway Time-out ... nginx), recorded as if the model had answered. Until traced, the substrate's failure was indistinguishable from a decision.

Small sample, single environment, stated as such. Even the mitigated rate is ~1 in 40 governed actions resolving by posture — under the default, proceeding ungoverned — with no conformant way to see the aggregate. Unmitigated it is 1 in 6. And because the rate is payload-size-dependent, it is attacker-improvable without compromising anything: inflate the payload → cross the proxy's tolerance → timeout → posture proceeds. Same bypass shape as the -32600 oversize path #32 documents, reached through the timeout half of §6.4 instead of the refusal half.

Proposed direction (wire-minimal, per SPEC_REVIEW_PRINCIPLES.md, Principle 3)

Define the decision-failure audit event. A named Trace event class alongside acs.decision — e.g. acs.decision.failure — carrying: failure class (timeout / transport / error_without_decision), the §17.1 error code when one exists (adopting @astrogilda's point in [Bug] §6.4 conflates Guardian refusal with decision failure; refusals are attacker-reachable and should fail closed #32 that the class must travel as a code, not prose), negotiated timeout, elapsed ms, payload-size bucket, and the posture applied. Emitted by the Observed Agent, since the Guardian by definition cannot.
Extend §4.1's language to per-step failures: recorded in the deployment's own audit log; SHOULD be surfaced on Trace when ACS-Trace is claimed.
Add a session-cumulative count to sessionEnd.summary (decision_failures by class), so the per-session rate is computable from artifacts the spec already defines.

Payload-size bucketing is the one field I'd argue for from data rather than symmetry: it is the dimension along which the failure rate actually moved, and the one an anomaly detector needs to tell organic failure from induced failure.

Out of scope here: whether proceed is the right default. That is #32's live question; this issue is the observability floor under either answer.

References

docs/spec/instrument/specification.md §6.4, §4.1, §8, §13 (the ping caveat — "a successful ping does not prove the enforcement path is healthy" — already concedes this path is otherwise invisible)
docs/spec/trace/events.md (five-disposition acs.decision vocabulary)
specification/v0.1.0/context-entry.json, specification/v0.1.0/hooks/session-end.json
Related: [Bug] §6.4 conflates Guardian refusal with decision failure; refusals are attacker-reachable and should fail closed #32 (refusal vs failure taxonomy; default-flip proposal), Trace pillar overclaims verifiability: OTel/OCSF emission is observability, not evidence #36 (trace self-reporting)

Prem Methuku

This is the right kind of finding, and the measured data is what makes the case. I have hit the same failure class from the offensive-evaluation side, which corroborates the "indistinguishable from a decision" point from a different direction.

Running LLM agents against targets through a harness, I repeatedly saw transport and delivery failures (malformed or rejected deliveries, truncated exchanges) recorded by the harness as if they were the model's output, silently understating capability until I separated the delivery layer from the model's actual answer. That is the same shape as your three "final answers" that were verbatim nginx 504 pages: the substrate failed, and nothing downstream could tell that apart from a real decision.

Your payload-size mechanism sharpens it in a direction I had not measured but which follows directly: an adversary who inflates a payload to cross the proxy's tolerance produces a timeout indistinguishable from organic failure. That is exactly why the acs.decision.failure event and the payload-size bucket are worth specifying rather than leaving implicit, since without them the fail-open event cannot separate induced from organic failure, which is the one distinction an anomaly detector needs. +1 on all three proposed changes; the session-cumulative count in sessionEnd.summary makes the per-session rate computable from artifacts the spec already defines.

@astrogilda

Thanks, corroboration from the attack side is the cross-check this needed. We hit the same failure class from opposite directions and both saw the substrate answering for the model.

Your induced-vs-organic point firms up something I left soft. I proposed the payload-size bucket as a field worth having, but if this event is meant to feed anomaly detection, it can't be optional, a stream that can't tell induced from organic failure answers the wrong question. So: payload-size bucket and the class code (per @astrogilda #32) both at MUST, everything else can stay optional.

The three-place search settles where the event can live. It leaves one constraint open, on the emitter, and that constraint decides whether the count is measurable.

Your event is emitted by the Observed Agent, because the Guardian by definition cannot. So the record that a governance decision failed is written by the party the governance was watching, using a code it selects for itself. That holds up under organic failure and breaks under induced failure, which is the case payload-size bucketing exists to detect. An agent that benefits from proceeding unguarded can emit timeout for a step where nothing timed out, omit the event, or emit nothing at all, and none of those is distinguishable downstream from a healthy deployment that had no failures. A zero in decision_failures and an unemitted event look identical.

This is the same shape as the #36 trace argument, one layer down. There the question was whether emission is evidence; here the emitter is the party under scrutiny and the subject is its own non-compliance.

A fourth change makes the count checkable. Without it, the MUST records a failure that only its perpetrator can vouch for.

Give the session a monotonic step counter that the Observed Agent cannot skip, and have the Guardian sign its own decision receipts with a sequence number. Then the count is derivable by subtraction: total steps, minus the steps that carry a signed Guardian receipt. Nobody has to be trusted to report it. An agent that suppresses failure events now has to suppress steps, which changes the step count, which the counter records. The self-reported class code then answers what kind of failure, and the counter answers how many, so no number depends on the agent's honesty about its own count.

This makes two records checkable. First, sessionEnd.summary.decision_failures becomes a claim a consumer can check against the receipts. If they disagree, that disagreement is itself the finding, and it is the one an anomaly detector should fire on first.

Second, the section 4.1 extension needs the same counter. An unguarded session recorded only in the deployment's own audit log has the deployment attesting to its own bypass, and a sequence gap is visible to a party who was not there where a missing log line leaves no trace of itself.

On the ping caveat in section 13, that a successful ping does not prove the enforcement path is healthy. A signed receipt per decision is the thing that would, since it is produced by the enforcement path doing its actual work, where a ping is answered by a side channel that stays up when the path is dead.

Codes for the failure classes, and one more distinction inside the enumeration: whether the Guardian was reached and did not answer, or was never reached. Those have the same effect on the step and different security meanings, so a deployment carrying only one code for both reads a slow Guardian and an absent one the same way.

+1 to deriving the count rather than trusting it. Moving from "the emitter reports its failures" to "the failures are whatever is left after the signed receipts" is the same self-report-to-verifiable move as #36, and it is the right one.

Two conditions decide whether the subtraction actually holds, and both point back at #36:

The total has to come from an authenticated source, not the agent. Count-by-subtraction is total steps minus signed-receipt steps, but if the Observed Agent supplies the total it just truncates the tail: drop the last N steps and their receipts together, and the failures in that tail never show up in the subtraction. That is the same truncation case your run-end seal defends against in Trace pillar overclaims verifiability: OTel/OCSF emission is observability, not evidence #36. The session needs an authenticated maximum sequence number (a sealed step count at session end) for the subtraction to be sound. Un-skippable is not the same as un-truncatable.
The receipt signing scope has to bind the sequence number, canonically. "Signed with a sequence number" is only sound if the index is inside the signature and over a canonical form; otherwise a receipt can be replayed onto a different step, and two verifiers can disagree on which steps are covered. That is the defined-scope and canonical-serialization point I raised on the corpus in Trace pillar overclaims verifiability: OTel/OCSF emission is observability, not evidence #36: the receipt is evidence of "step k was decided" only if k is inside the signature, byte-exact.

With those two, monotonic-counter-plus-signed-receipts gives a count nobody has to be trusted to report, which is what the MUST needs.

Both conditions are right and the first one is a hole in what I proposed, not a refinement of it.

I made the counter un-skippable and left the total in the agent's hands, and those are not the same property. Drop the last N steps with their receipts and the subtraction returns zero for a tail that never gets counted, which is worse than a self-reported number because it looks derived. Un-skippable is not un-truncatable, and I had the first and claimed the second.

So the total has to be sealed, and the run-end seal from #36 already does it, so this needs no new mechanism. A sealed maximum sequence number at session end makes the tail non-droppable, and forgery cost is what does it, not ignorance. The seal is written at session end, so the producer does know by then which steps it would like to lose; what it cannot do is present a smaller count without forging a signature over the sealed one. The arming record and the end seal bracket the run, and the counter is what makes the bracket say something about each step inside it.

Your second condition is the sharper of the two for anyone implementing it. The index has to be inside the signature and over a canonical form, or a receipt is a receipt for nothing in particular: replay it onto a different step and two verifiers reach different covered sets from the same bytes. That is the signing-scope pin I put on #18, and it is the same defect underneath, so it wants one fix and not three.

Which makes the three issues one mechanism, and the count is the part that proves it works. #36 gives the bracket, #18 gives the scope the signature covers, #37 gives the quantity you can derive once the other two hold. Any one of them alone leaves a number somebody has to be trusted for.

Two things to state normatively, not leave to implementers.

The seal has to cover the count, not sit beside it, or a producer emits an honest seal and then reports a smaller total beside it.

And a session whose seal is absent has to read as unmeasurable, never as clean. If a missing seal reads as zero failures, truncating the entire session becomes the cheapest attack in the set, and it would sit inside the same fail-closed default #36 already argues for.

On the last normative point, a missing seal reading as unmeasurable rather than clean, my data backs it from the organic side: silent non-emission is the default state of instrumentation, not just an attack. In the same evaluation, my own harness under-emitted its recovery events 3 of 4 times, recovery happened in a code path the instrumentation didn't cover. An earlier audit found only 2 of 13 claimed runs had left logs at all, which forced a rebuild so every run writes evidence before it counts. Honest deployments will under-emit like this constantly. So the derived count isn't only defeating a lying agent, it catches the ordinary bugs that produce the same silence. That's why absent-reads-as-unmeasurable belongs in spec text: it'll fire far more often on bugs than on adversaries.

Close the governance gaps the reference implementation opened (#69)

Correct the one documentation claim that outlived its facts (#102)

@bar-capsule

Status from the reference side: #244 (the OpenTelemetry exporter) is taking on both halves of this event for the reference Guardian, using the names in Bar's decision above.

On the Guardian side, the event is emitted for guardian_error (a JSON-RPC error instead of a decision) and for timeout (a decision finished after the negotiated timeout, so the host had already applied its posture). The exporter also uses other with a detail string for one more case: a step still open when the Guardian shuts down. @bar-capsule, that last one goes a little past the two Guardian cases you listed. Tell us if you'd rather it be a span status only.

On the Observed Agent side, a converter turns the host adapter's audit log (.acs/audit.jsonl) into the same OTLP file, with acs.failure.reported_by = observed_agent. Here is the class mapping we proposed. It's worth a second look, especially the refused row:

Host audit `failure.kind`	`acs.failure.class`
`timeout`	`timeout`
`transport`	`unreachable`
`error_without_decision`	`guardian_error`
`refused`	`guardian_error`
`host_configuration`	`observed_agent_error`
`decision_unrenderable`	`observed_agent_error`
`unknown`, or anything else	`other`

The mapping entry for the event lands in the #228 PR. #244 references this issue but does not close it, because the §6.4 and §4.1 spec text and the sessionEnd.summary count are still open.

@bar-capsule

@bar-capsule, two gaps in the class list came up while I wrote the mapping entry for this event. It's on the #228 branch, using your names exactly. I didn't add anything to your list, because the list is your decision.

A response whose signature fails verification has no class of its own. §10 tells a verifier to reject it. Today it would land in malformed_response or other, so an attacker who tampers with responses on the channel blends into organic noise under fail-open. Should there be a signature_invalid class?
A session that starts unguarded has no event. §4.1 says it SHOULD be surfaced on Trace when the deployment claims ACS-Trace. acs.decision.failure is a span event on a step span, so a handshake that fails, followed by a whole session running unguarded, never shows up in the failure count. Blocking the handshake is the cheapest induced failure there is. Should that case get a class (say handshake_failure) recorded on the acs.session span, or a separate event?

The #228 text adds three clarifications, listed there as my proposals for you to confirm:

A response the spec already resolves is not a decision failure. A §6.3 violation gets DENY. A MODIFY the client can't apply gets DENY, or at postCompact the original summary (§6.5).
A count built from the Observed Agent's events is a lower bound. That follows from your point that its records can miss events.
acs.failure.detail must be text the emitter composes itself, not bytes copied from the request or response.

Edited: corrected the §6.5 case, which keeps the original summary at postCompact.

Carry the decision's replay facts and decision failures on Trace

Say what a decision event proves, and what it does not

Carry the decision's replay facts and decision failures on Trace

Say what a decision event proves, and what it does not

@astrogilda

@astrogilda, short answer: the count and the seal land in the same place. The reason is the key rather than who runs the check.

What decides it is whose account a consumer is reading. That is the definition now on the wire in your vocabulary entry.

Under the HMAC baseline a consumer holding the artifacts has two accounts available: the Guardian's and the Observed Agent's. Both hold the session key. Nothing in the head separates a head the Observed Agent retained from one the Guardian could have written. The Observed Agent could as easily have written any head the Guardian could. The second account is therefore not a different account. That is the fact your acs-f-10 member already rests on. It is SELF and unmeasurable, because the only accounts of who signed are the two key-holders' own.

acs-f-6 uses the same fact to reach PEER. The README glosses the column as who can check the member without the enforcement point's cooperation. By that gloss the Observed Agent qualifies. By the definition the field itself points at it does not, because what it can check it can equally produce.

So the disagreement is one sentence wide: does the scope name who can compute the check or whose account a consumer can rely on. I read it as the second. The baseline seal pair then matches the baseline signer member at SELF. If the intent is the first, the two members want reconciling before v0.2 lands the count, since they are one key-sharing property read two ways.

The consequence of shipping the Core half as PEER is the one we raised on the count: the corpus would assert a second-party witness where the profile provides two holders of one key. The CI job would then be scoring a claim rather than a property.

The ACS-Crypto half is EXTERNAL and I agree with it. If the first reading is what you want, the term needs one sentence about key custody to carry it. That sentence belongs where the witness scope wording already lives rather than in the corpus.

On the begun session, the order you set is the order I would keep: text first, then re-pin, then mint. A member written against ACS-R-005 would cite a sentence that does not say what it tests. Your awaitingText note is the honest place for it to wait.

Record decision failures the Guardian can see as acs.decision.failure

Convert the host audit log into the same OTLP file

Tidy the shutdown path after review

@rocklambros

@rocklambros yes to both!

signature_invalid - yes. A tampered response should stand out from normal failures, so it gets its own class. I also think it should fail closed no matter the posture, otherwise breaking the signature is a free bypass under fail-open. Add adapters/ with Claude Code, Cursor, and NAT reference implementations #22 already does both; the spec side belongs in [Bug] §6.4 conflates Guardian refusal with decision failure; refusals are attacker-reachable and should fail closed #32.
handshake_failure - yes. Without it, a blocked handshake means a whole unguarded session with zero failures, which looks clean. Let's record it as the same acs.decision.failure event, on the acs.session span. On the [Bug]: acs.decision trace event cannot carry reason_codes, policy_references, cited_provenance_ids, or chain_hash, so replay from the trace alone is not possible #228 branch the event is defined on the step span, so it needs "or on the session span for handshake_failure". Counting still works - handshake/hello has a request_id and session_id like any request.

Also yes to the three clarifications.

The rest of #37 is in #273 (draft until #228 lands). It has a test that checks the failure classes match the mapping, so it'll go green once both classes are in #228. If it's easier, I can add them in #273 after #228 merges.

Add the signature_invalid and handshake_failure classes Bar decided o…

Answer the 12 October review of the Go Guardian

Status after 0.1.4: the first half of Bar's decision has shipped. The acs.decision.failure event is in the Trace mappings with all eight failure classes (PR #275), and the reference Guardian and its audit converter emit it (PR #244). The verifiable failure count stays open here for v0.2, so this issue stays open.

Navigation Menu

Repository navigation

[Bug] §6.4's mandatory fail-open audit event has no conformant shape, so the decision-failure rate is unmeasurable #37

The gap

Why the rate matters: measured data

Proposed direction (wire-minimal, per SPEC_REVIEW_PRINCIPLES.md, Principle 3)

References

8 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug] §6.4's mandatory fail-open audit event has no conformant shape, so the decision-failure rate is unmeasurable #37

Description

The gap

Why the rate matters: measured data

Proposed direction (wire-minimal, per SPEC_REVIEW_PRINCIPLES.md, Principle 3)

References

Activity

ossumpossum commented on Aug 29, 2026

Premmethuku75 commented on Aug 29, 2026

astrogilda commented on Aug 29, 2026

ossumpossum commented on Aug 29, 2026

astrogilda commented on Aug 30, 2026

Premmethuku75 commented on Aug 30, 2026

8 remaining items

rocklambros commented on Oct 12, 2026

rocklambros commented on Oct 12, 2026

narko4u commented on Oct 12, 2026

bar-capsule commented on Oct 13, 2026

rocklambros commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions