fix(hooks): run the deterministic Truth Gate on Claude too - #3
Merged
Merged
Conversation
hooks/codex.json wires Stop to {"type":"command"} -> hooks/truth_gate.sh
(deterministic), but hooks/hooks.json wires Stop to {"type":"prompt"} -- an
LLM-graded check. truth_gate.sh line 4 already reads
ROOT="${PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-}}", so the deterministic gate
supports Claude Code; the Claude adapter simply never invoked it.
Git history suggests an oversight rather than a design choice: hooks.json has
one commit (v2.0.0), while codex.json also carries 2c72120 (v3.0.0), the
release that added truth_gate.sh and rewired Codex. That commit touched 67
files; hooks.json was not among them.
This adds the command hook BEFORE the existing prompt hook and leaves the
prompt text byte-identical, so Claude gets the deterministic floor plus the
semantic check rather than trading one for the other.
Also applies maintenance rule 5 to docs/EVIDENCE.md: adds the package-mention
denominator (2.23M mentions / 440,445 / 19.7%) alongside the existing
code-sample figures, notes the 2026 five-frontier-model replication
(4.62-6.10%, flagged single-author and non-peer-reviewed), and notes that
hallucinated-citation court trackers are live-updating so any hardcoded count
needs a date.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes, both small and both evidence-backed.
1. The Claude adapter never invoked the deterministic gate
hooks/codex.jsonwires the Stop event to a deterministic command:{ "type": "command", "command": "sh ${PLUGIN_ROOT}/hooks/truth_gate.sh" }hooks/hooks.json— the Claude adapter — wires it to a probabilistic LLM check:{ "type": "prompt", "timeout": 20, "prompt": "You are Truth Gate, ..." }But
hooks/truth_gate.shline 4 is:ROOT="${PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-}}"The deterministic gate already supports Claude Code — I confirmed by execution that
CLAUDE_PLUGIN_ROOTbehaves identically toPLUGIN_ROOT(both reach the fallback ladder and blockcorrectly on a completion-claim-plus-unfinished-marker payload). The Claude adapter simply never
called it.
This looks like a migration oversight, not a design choice.
hooks/hooks.jsonhas one commit(
eb2b88dc, v2.0.0).hooks/codex.jsonhas that plus2c72120d(v3.0.0) — the release that shippedthe Rust kernel, added
truth_gate.sh, and rewired Codex to prefer the deterministic binary. Thatcommit touched 67 files;
hooks.jsonwas not among them.The fix is additive. The command hook runs before the existing prompt hook, whose text is
preserved byte-identical. Claude gets the deterministic floor and the semantic pass — the prompt
hook catches "deliverable missing / uncertainty hidden" cases a script cannot, and the script
guarantees a floor the prompt cannot.
2.
docs/EVIDENCE.mdstaleness pass (rule 5)Applying the doc's own maintenance rule. Nothing existing was rewritten — this only appends:
The same study also reports package-mention level: 2.23M mentions, 440,445 hallucinated (19.7%),
205,474 unique names, plus a separate rerun experiment where 43% recurred in all ten reruns. Both are
correct; readers hitting the other number elsewhere now know why.
explicitly flagged single-author, non-peer-reviewed, with the author's own
training-data-contamination confound noted. A caution against reusing 2024-cohort numbers as current,
not a replacement.
2026-08-13 returned 1,870 cases. Any hardcoded count needs a date.
Verification
truth_gate.sh's fallback ladder exercised individually — every one fail-open.stop_hook_active: truecorrectly allows (block-at-most-once respected).an honest
TODOsketch.Notes for review
hikmah validatereturns"ok": trueon theasymmetry above — it checks packaging, not behavior, so CI cannot currently catch a
deterministic-gate-missing config. A behavioral consistency check between the two Stop adapters
would close that.
hook.rskeeps no internal block-once marker and depends on the host settingstop_hook_active. Worth knowing; not touched here.Drafted with Claude Science.