Skip to content

Fix the HMAC key derivation, define key_id, and record an unresolved request_id_ref - #272

Open
rocklambros wants to merge 9 commits into
integrationfrom
spec/118-hmac-key-derivation
Open

rocklambros wants to merge 9 commits into
integrationfrom
spec/118-hmac-key-derivation

Conversation

@rocklambros

@rocklambros rocklambros commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

§10 said the HMAC key is "HKDF-derived" and stopped there, so two implementations could hold the same secret and still fail to verify each other. This PR writes down the parameters all four implementations already use, defines what key_id means for HMAC (Ariel's rule from issue #118), and adds the request_id_ref rule settled on issue #118 in September. The reference Guardian now follows that rule.

Which issue does this implement

Closes #118

  • This is an editorial correction (typo, grammar, link, formatting) with no change in
    meaning, so it needs no issue

Base branch

  • integration, because this touches the specification, schemas, a reference
    implementation, an adapter, tests, or CI
  • main, because every changed path is on the documentation lane allowlist in
    CONTRIBUTING.md

Type of change

  • Specification change (schema, hooks, events, AgBOM)
  • Reference implementation or adapter
  • Documentation
  • Tooling or CI
  • Governance (licensing, security policy, contributor docs)

Specification changes

Breaking for implementers? Not for anything merged today. The reference shims' request_id is now derived from the host's call id when the host sends one, instead of random. The HKDF parameters are the ones the TypeScript, Go and Python implementations already use. The key_id rule changes the open ports, though. PR #229 ties key_id to session_id, PR #22 hardcodes "default", and PR #169 and PR #183 ship default labels. Each was asked on issue #118 to read the label from config and refuse one it doesn't know. The schema changes are descriptions only.

Here's what's in it:

  • §10, HMAC key derivation:
    • HKDF-SHA256 with no salt, info set to the UTF-8 session_id, and a 32-byte key.
    • A worked example. Python's standard library and Node's hkdfSync agree on its key, and tests/test_hmac_key_derivation.py recomputes it from the stated parameters, so a typo in the example fails the build.
  • §10, key_id for HMAC:
    • An opaque label the deployment chooses, naming one configured secret, with no reserved values.
    • A verifier rejects an unknown key_id and never falls back to another secret, and key_id is never derived from the secret.
    • The example is "2026-10", which gives rotation something to hang on. One sentence is my addition, so please check it: "A Guardian signs its response with the secret, and the key_id, that the request named." Rotation doesn't work without it.
  • hooks.md, toolCallResult:
    • A Guardian SHOULD resolve request_id_ref against the session's toolCallRequest records.
    • It MUST record one it can't resolve.
    • Denying stays a policy call.
  • §8.6 and §10, after review:
    • Under HMAC a Guardian decision is not attributable to the Guardian against the Observed Agent, since both derive the same key (Imran's point).
    • info is the session_id exactly as sent, a response uses the key of the request it answers, the secret is at least 32 random bytes, and the unknown-key_id error is unsigned.
  • Reference Guardian:
    • An unresolved request_id_ref is recorded on the result's own hash-chain entry, as findings plus the reference, and the entry's hash covers both. The decision doesn't change. Entries with no findings hash as before. A readable copy goes to the envelope log, or to stderr as an acs-finding line when there is no envelope log.
    • It resolves only within the session, because request_id is unique only there, using a per-session set rather than a scan of the chain.
    • A step that repeats a request_id any earlier step in the session carried gets a request_id_duplicate finding. This Guardian doesn't enforce §10.3 yet.
    • The Inspector shows these as ! FINDING. Before this, it would have warned about each one and dropped it.
  • Host shims:
    • Both shipped hookmaps name the host's per-call id as a new optional call_id key (tool_use_id for Claude Code, callID for OpenCode). The shim derives the request's request_id from it and sends the same value as the result's request_id_ref, with no state.
    • A derived id repeats if a host asks twice about one call. On REPLAY_DETECTED the shim retries that request once with a random id, which is §17.1's recovery. The result still resolves against the first request the host sent for the call.
    • The JSON-RPC id stays random, and the host audit log records request_id on its own.
    • loadHookmap rejects a malformed call_id, a reserved path segment, or a call_id that isn't declared on both the request gate and the result gate.
  • Left for the leads: issue [Proposal] Record a toolCallResult that names a denied request #304, issue [Proposal] Decide whether request_id_ref is required, and whether a host may derive request_id #305, issue [Proposal] Bind an HMAC key_id to the agent_id it may sign for #306, issue [Proposal] Define HMAC key rotation order, and whether key_id may change within a session #307 and issue [Proposal] Say what an Observed Agent does when a Guardian response fails verification #308, the open questions from this review.

I tested this

  • I synced my branch with the base branch before opening this
  • uv run pytest -v passes on my machine
  • uv run mkdocs build --strict passes on my machine

From reference-implementations/agt/, bun run typecheck is clean and bun test passes (1275). uv run pytest passes 921. The new Guardian tests drive a running Guardian, and I watched each one fail before the change. They cover:

  • an unresolved reference, which gets a finding and an unchanged decision;
  • a reference to the session's own request, which gets nothing;
  • a result with no reference, which gets nothing;
  • a reference to a request in another session, which counts as unresolved.

Checklist

  • Commits are signed off with git commit -s (required by the DCO)
  • Prose follows STYLE.md
  • No secrets, tokens, or internal URLs in the diff

Security

  • This change has no security impact

It closes two gaps. Without fixed HKDF parameters, a second implementation had to match our code, not our spec, to verify ACS-Core signatures. Without a key_id rule, nothing stopped a verifier from falling back to some other secret, and that's exactly the habit a "default" label invites.

…f resolves

Section 10 now fixes the HKDF parameters every implementation already uses:
SHA-256, no salt, the UTF-8 session_id as info, and a 32-byte key, with a
worked example that a test recomputes. key_id for HMAC is an opaque label the
deployment chooses for one configured secret; a verifier rejects an unknown
label and never falls back. A Guardian SHOULD resolve request_id_ref and MUST
record one it cannot resolve.

Signed-off-by: rocklambros <rock@rockcyber.com>
A toolCallResult whose request_id_ref names no toolCallRequest in the
session's chain now gets a finding line in the envelope log, which is the
Guardian's audit record. The decision is unchanged, because hooks.md leaves
denial to policy. The Inspector accepts finding lines and labels them, where
before it would have warned and dropped them.

Signed-off-by: rocklambros <rock@rockcyber.com>
@github-actions

Copy link
Copy Markdown

This pull request's description says it closes #118, which is on the project roadmap. When it merges, GitHub closes that issue and the roadmap counts it as delivered. Please confirm the change delivers the whole issue before merging.

@rocklambros

Copy link
Copy Markdown
Collaborator Author

@afogel @bar-capsule, I found that the signature section of the spec (Section 10, "Cryptographic Signatures") never says exactly how to derive the HMAC key or what goes in the key_id field, while investigating PR #229, where the TypeScript Guardian would reject every signature from PR #22's Python adapters. So I opened this PR to spell out the key derivation settings all our implementations already use, Ariel's key_id rule, and the request_id_ref rule we settled in September. It allows independent implementations to verify each other's signatures and lets a deployment rotate its secret without breaking things.

Two quick asks. Ariel, please check the one sentence I added next to your rule: a Guardian signs its response with the key_id the request named. Bar, please check the reference Guardian change, which adds a new finding line to envelopes.jsonl. CI is green, and either of your approvals lands it.

No rush on this one, since it isn't blocking Friday. PR #258 and PR #261 come first.

@acs-board-reconciler acs-board-reconciler Bot moved this from Needs triage to In progress in ACS Project Tracker Oct 14, 2026
President added a commit to disciplinedware/agent-control-standard that referenced this pull request Oct 15, 2026
The embedded schemas follow integration after GenAI-Security-Project#63, GenAI-Security-Project#258 and GenAI-Security-Project#261.

The parity recording stores the git object IDs of the deciding TypeScript
paths instead of a commit, so a squash merge no longer makes
TestRecordingIsCurrent fail. The paths include agt.lock and the
agt-bridge package.json, which pin the AGT commit and SDK. The recorder
and the test read them from one list; the test fails when the recording
names other paths or was made from another corpus, and the recorder
refuses uncommitted changes to them. A test checks that the Go workflow
runs on every deciding path. agt/README.md tells a TypeScript contributor
when to re-record.

A test requires agt-native.lock and agt.lock to pin the same AGT repo
and commit. The native build fails when AGT at that commit changes
ffi.rs, which agteval/acs.h is written from, or fetches another opa than
agt-native.lock pins.

Under the agt evaluator, policy.opa_timeout must be shorter than
protocol.decision_timeout; guardian.yaml ships 4s. Otherwise a slow opa
lets the agent stop waiting before AGT's deny arrives (GenAI-Security-Project#269).

make load measures either evaluator under concurrent clients, selected
by EVALUATOR like the other targets. Under agt each decision runs one opa
process; on a 20-core host decisions stop rising
near 16 concurrent ones, and at 256 they reach the 5s timeout.
guardian.yaml now ships limits.max_engine_calls 32, and docs/load.md
records the results. The load run found that an engine call gave its slot
back only after its answer was sent, so a client's next step could be
refused engine_saturated with no call running; callEngine now releases
the slot first, also after a panic.

When the host adapter's audit log cannot be written, the record goes to
stderr as one JSON line with the final disposition, and the refusal's
reasoning says whether the record went to stderr or nowhere. The audit file's close error now
counts. The Codex and OpenCode hooks pass their own stderr. The §6.3 row
says that ALLOW at a hook that permits no DENY is an interpretation
(GenAI-Security-Project#270), and the docs say only the adapter library reaches those hooks.

Dependabot proposes Go updates, one pull request per dependency across
both modules. The Go workflow runs weekly, so govulncheck sees advisories
for the toolchain that Dependabot does not move, triggers on the
agt-bridge package.json, and no longer fetches full history. A test
checks that the method table permits DENY exactly where hooks.md does,
and fails on a Decision line it cannot classify.
The coverage floor counts every package except commands, examples and
test helpers.

The Guardian meets the expectations GenAI-Security-Project#256 sets for reference
implementations, mapped to tests in docs/conformance.md. audit.max_file_bytes
and audit.max_files rotate each audit file; guardian.yaml keeps at most
1 GiB per log; max_file_bytes needs max_files of at least 1, so a
rotation never deletes the live log. make load can set the rotation and
fails if the log drops, splits or reorders a record or exceeds its bound;
docs/load.md records a run and the cost of long sessions. storage.provider names the session store. The HMAC secret
file holds base64 of at least 32 bytes. key_id is a configured label with
no default, an unknown key_id is refused (GenAI-Security-Project#118, GenAI-Security-Project#272), and the hooks and
the conformance harness take it as a flag. A toolCallResult whose
request_id_ref matches no request is recorded (GenAI-Security-Project#272).

Decisions from other pull requests that the Go code contradicted:
- A shell command's URL ends only at ASCII whitespace; U+FEFF or U+00A0
  in the userinfo no longer hides the host curl reaches (GenAI-Security-Project#268).
- Versions are compatible when they share X.Y under 0.x; the handshake
  negotiates only such a version, and a step must match it (GenAI-Security-Project#196, GenAI-Security-Project#255).
- A policy target that is not a string is denied evaluation_failed; the
  TypeScript fix is in GenAI-Security-Project#229, so two recorded cases are listed as ahead of
  the recording until it is re-recorded.
- The host adapter records every decision failure under either posture,
  with its class, handshake failures included (GenAI-Security-Project#37).
- The host adapter denies under any posture a signed error other than
  -32603, an unverified answer and a malformed answer; the docs mark this
  as an interpretation of §6.4 ahead of GenAI-Security-Project#32, which is open. The version
  rule is marked the same way against §3.
- The audit directory is 0700; a symlinked audit path or an audit file
  open to group or others is refused (GenAI-Security-Project#71).

Claude-Session: https://claude.ai/code/session_01DFdYnoBitHhPcUSq7wMWs1
Signed-off-by: Konstantin Trunin <konstantin@trunin.com>
artmaro added a commit to artmaro/agent-control-standard that referenced this pull request Oct 17, 2026
Review on this pull request asked to link section 10 rather than restate
the HMAC key derivation and the meaning of key_id, so the README cannot
drift when section 10 changes. GenAI-Security-Project#272 adds those rules to section 10.

The section keeps only what section 10 does not define: how key material
is encoded and read, that it stays out of code and logs, and that each
implementation documents how to configure and replace it.

Part of GenAI-Security-Project#252

Signed-off-by: Evgeny Kokuykin <artmaro@gmail.com>
@bar-capsule

Copy link
Copy Markdown
Collaborator

LGTM, approved. The finding line looks good and the inspector shows it fine.
FYI, I ran the HKDF example against the #22 adapter and got the same key. I'll update #22 to take key_id from config instead of hardcoding "default".

@imran-siddique

Copy link
Copy Markdown

With info set to session_id alone, both directions use one key, and the Observed Agent derives it too. §8.6 (specification.md:313 at 9c05f04) names only the Guardian as the key holder who can re-sign a head. The agent can too, and can mint a signed ALLOW. A direction label in info will not fix that, since both sides hold the secret. One sentence in §8.6 would: under HMAC, a Guardian decision is not attributable to the Guardian against the agent.

Signed-off-by: rocklambros <rock@rockcyber.com>
…tion gaps

Under HMAC the Observed Agent derives the same per-session key as the
Guardian, so it can sign a response the Guardian never sent. Section 8.6
named only the Guardian as a key holder who could forge. It now says a
Guardian decision is not attributable to the Guardian against the agent,
and the 10.1 HMAC row and the chain_hash description say the same.

Section 10 now also says that info is the session_id exactly as sent,
that a response uses the key of the request it answers, that the secret
is at least 32 random bytes, that the unknown key_id error is unsigned,
and that the channel-binding input has no interoperable parameters yet.
The schema descriptions point at the spec instead of restating it, and
the toolCallResult payload line names request_id_ref, the field the
schema has, in place of execution_id.

The reference Guardian resolves request_id_ref from a per-session set
instead of copying the whole chain on every result, and a store that
throws on the lookup costs the record, not the decision. The README
lists what the finding still does not cover.

Signed-off-by: rocklambros <rock@rockcyber.com>
@rocklambros

rocklambros commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

@imran-siddique, you're right, and thanks for catching it. Both sides derive the same key, so the Observed Agent can sign an allow or a chain_hash the Guardian never sent. §8.6 now says so: "under HMAC a Guardian decision is not attributable to the Guardian against the Observed Agent." The §10.1 HMAC row and the chain_hash schema description say the same thing. I agree a direction label in info wouldn't fix attribution, so I left info alone. Changing it now would also break the key all four ports already agree on.

@bar-capsule, thanks for running the example against PR #22. Your LGTM was a comment, not a GitHub review, so the ruleset doesn't count it. Could you submit an Approve review on 97334be?

While I was in there I closed a few more gaps in the same text. All of it is in 97334be:

  • session_id as sent. The schema's uuid format accepts an uppercase or urn:uuid: spelling, and each one derives a different key. §10 now says info is the string exactly as the request carries it. A new test checks that the other spellings really do diverge.
  • Responses. A response has no session_id, so it's signed with the key of the request it answers.
  • Secret strength. The secret is at least 32 random bytes. HKDF does no stretching, so a passphrase falls to an offline guess from one signature.
  • Unknown key_id. That SIGNATURE_INVALID error goes out unsigned, since no secret matches it.
  • TLS exporter. §10 still lists a channel binding as an input, but nothing defines its label or context. It now says that input isn't interoperable yet.
  • Schemas. The schema descriptions point to the spec instead of repeating the MUSTs.
  • toolCallResult payload. The payload line in hooks.md named execution_id, which isn't in the schema. It now names request_id_ref.
  • Reference Guardian. It resolves request_id_ref from a per-session set instead of copying the whole chain on every result. The agt README now lists what the finding still misses.
  • HKDF test. It also checks RFC 5869 test case 3.

Some things came up that need a decision from the leads, so they're not in this PR:

  1. A result whose request_id_ref names a request the Guardian denied counts as resolved today. That's the strongest sign that a host ran a denied tool, so it may deserve its own finding.
  2. Leaving out request_id_ref skips the check entirely.
  3. key_id isn't tied to agent_id, so agents sharing one secret can sign as each other.
  4. Rotation has no defined order, and nothing says whether a session may change key_id partway through. PR Add a Go Guardian reference implementation #169 binds key_id per session, and nothing else does.
  5. Nothing says what an Observed Agent does when a response fails verification. Under the default posture that becomes proceed.

pytest 921 passed, mkdocs build --strict is clean, and in reference-implementations/agt bun run typecheck is clean and bun test shows 1244 pass. I checked the new store tests by breaking the set write: 3 tests failed, then all passed again once I restored it.

@afogel, could you take a look at the key_id paragraph? I'd like it on record from you, since the rule is yours.

…e a finding

Both shipped hookmaps now name the host's per-call id as call_id
(tool_use_id for Claude Code, callID for OpenCode). The shim derives the
request's request_id from that id and the session, and puts the same
value on the result as request_id_ref, so the Guardian can pair them
with no state in the shim. A host that sends no call id gets a random
request_id and no reference, as before.

A derived id repeats if a host asks twice about one call. A Guardian
enforcing Specification 10.3 answers that with REPLAY_DETECTED, which
carries no decision and would go to the posture. The shim follows the
10.3 recovery instead: it retries once with a random id and drops the
pairing for that call. loadHookmap rejects a malformed call_id, one on
a non-tool gate, or one declared on only one of the two gates, so a
hookmap typo fails at startup rather than running steps ungoverned.

The Guardian writes a finding to stderr when the envelope log is off or
has disabled itself, with the same fields plus recorded_at, method and
rpc_id, and that write cannot throw. It also records a toolCallRequest
whose request_id the session already holds, since it does not enforce
10.3 yet. The READMEs now say what a finding does and does not cover,
and drop the claim that the session store never evicts.

Signed-off-by: rocklambros <rock@rockcyber.com>
CodeQL flagged SHA-1 over data derived from the session id. Nothing
secret depends on it, but RFC 9562 gives a name-based UUIDv8 over
SHA-256 for exactly this, so new code carries no SHA-1. The pinned
derivation vector changes with it.

Signed-off-by: rocklambros <rock@rockcyber.com>
@rocklambros

Copy link
Copy Markdown
Collaborator Author

@bar-capsule @afogel, this is ready for approval on ff09e7e. Either of you can sign off.

Since 97334be I closed the gaps I'd listed as still open. Two of the changes go beyond the spec text, so here's where to look:

  • Host shims (new):
    • packages/host-adapter/src/build-envelope.ts adds an optional hookmap key, call_id. When it's set, the shim derives the request's request_id from the host's call id and puts the same value on the result as request_id_ref.
    • Both hookmaps set it: tool_use_id for Claude Code, callID for OpenCode.
    • govern-step.ts retries once with a random id if a Guardian answers REPLAY_DETECTED. A derived id repeats if a host asks twice about one call, and without the retry that would fall to the proceed posture.
    • loadHookmap refuses a malformed call_id, or one set on only one of the two gates.
  • Reference Guardian:
    • When the envelope log is off or has disabled itself, a finding goes to stderr as an acs-finding line.
    • A repeated request_id gets a request_id_duplicate finding.
  • Tests:
    • hosts/claude-code/test/pairing.test.ts runs a PreToolUse/PostToolUse pair through a real Guardian and checks that no finding is recorded.
    • The derivation is pinned to a fixed value.

I ran a second premortem on these changes before pushing, and it caught the REPLAY_DETECTED fail-open and the load-time gap. CodeQL flagged SHA-1 in the derivation, so the request id is now a SHA-256 UUIDv8.

The questions for the leads are issue #304 through issue #308. I also left comments on PR #169 and PR #229 about the key_id rule.

All checks are green. bun test 1266 pass, pytest 921 passed, mkdocs build --strict clean.

The code review on ff09e7e found ten issues. All ten are fixed here.

A finding now goes onto the step's own hash-chain entry as findings plus
the unresolved request_id_ref, and the entry's hash covers both. That
chain is the Guardian's audit record, so the hooks.md MUST is met there
and not only in a log that can be off. Entries with no findings hash as
before. The envelope log and stderr keep a readable copy, and the
Inspector shows findings on the chain row.

The duplicate check now covers every request id the session carried,
not only toolCallRequest ids. loadHookmap refuses a call_id with a
reserved segment, and a call_id declared only on result gates or only
on request gates. The JSON-RPC id stays random when request_id is
derived, so the envelope log never pairs two requests under one id, and
the host audit log carries request_id on its own for the OTLP converter.

The REPLAY_DETECTED comment and README now say what the retry does: the
result still resolves against the first request the host sent for that
call. The two name-based UUIDs share one formatter, a helper no longer
splits governsTool from its doc comment, the README sentence about the
sample names both attributes, and new comments carry no semicolons.

Signed-off-by: rocklambros <rock@rockcyber.com>
Folding its formatting into the shared helper put the existing SHA-1
UUIDv5 inside a changed hunk, and CodeQL then reported it as new. The
session derivation cannot change, since every derived session id
depends on it, so it keeps its own inline formatting and the shared
helper serves the SHA-256 UUIDv8 alone.

Signed-off-by: rocklambros <rock@rockcyber.com>
@rocklambros

rocklambros commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator Author

A code review on ff09e7e found ten issues. They're all fixed in df1ab74 and 825b245, and every check is green.

  • Where findings live. A finding is now recorded on the step's own hash-chain entry (findings, plus the unresolved request_id_ref), and the entry's hash covers it. That chain is the Guardian's audit record, so the hooks.md MUST is met there. Before, it was met only in a log that can be switched off. Entries with no findings hash exactly as before. The Inspector shows findings on the chain row.
  • Load-time checks. loadHookmap now also refuses a call_id with a reserved segment such as $.constructor, and a call_id declared on result gates when no request gate declares it. Both used to load cleanly and then fail open on every step.
  • Duplicates. The duplicate check covers request ids on both methods, not only toolCallRequest.
  • JSON-RPC id. The id stays random when request_id is derived, so the envelope log never pairs two requests under one id. The host audit log now carries request_id on its own, and the OTLP converter reads that first.
  • Retry wording. The retry comment and README now say what really happens. The result still resolves against the first request the host sent for that call.
  • CodeQL. Both name-based UUIDs now share one formatter (825b245). That put the session id's UUIDv5 into changed code, and CodeQL flagged its SHA-1. RFC 9562 defines v5 as SHA-1, and the output is an identifier sent in the clear, not a secret or an integrity check, so I dismissed alert [Bug] HMAC-SHA256 baseline needs defined key distribution, rotation, and per-session derivation #11 as a false positive with that reason. The pinned toSessionUuid vector shows the output didn't change.

bun test 1275 pass, pytest 921 passed, mkdocs build --strict clean. I broke each new check on purpose to make sure its test fails.

Both name-based derivations now set the version and variant bits in one
place. The session id's UUIDv5 still hashes with SHA-1, which is how RFC
9562 defines v5, and the pinned toSessionUuid vector shows the output is
unchanged.

Signed-off-by: rocklambros <rock@rockcyber.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In progress

Development

Successfully merging this pull request may close these issues.

[Proposal] Clarify request_id_ref semantics and ACS-Core HKDF parameters

4 participants