Skip to content

fix(hooks): run the deterministic Truth Gate on Claude too - #3

Merged
CodeWithJuber merged 1 commit into
mainfrom
fix/deterministic-truth-gate-on-claude
Aug 14, 2026
Merged

fix(hooks): run the deterministic Truth Gate on Claude too#3
CodeWithJuber merged 1 commit into
mainfrom
fix/deterministic-truth-gate-on-claude

Conversation

@CodeWithJuber

Copy link
Copy Markdown
Owner

Summary

Two changes, both small and both evidence-backed.

1. The Claude adapter never invoked the deterministic gate

hooks/codex.json wires the Stop event to a deterministic command:

{ "type": "command", "command": "sh ${PLUGIN_ROOT}/hooks/truth_gate.sh" }

hooks/hooks.json — the Claude adapter — wires it to a probabilistic LLM check:

{ "type": "prompt", "timeout": 20, "prompt": "You are Truth Gate, ..." }

But hooks/truth_gate.sh line 4 is:

ROOT="${PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-}}"

The deterministic gate already supports Claude Code — I confirmed by execution that
CLAUDE_PLUGIN_ROOT behaves identically to PLUGIN_ROOT (both reach the fallback ladder and block
correctly on a completion-claim-plus-unfinished-marker payload). The Claude adapter simply never
called it.

This looks like a migration oversight, not a design choice. hooks/hooks.json has one commit
(eb2b88dc, v2.0.0). hooks/codex.json has that plus 2c72120d (v3.0.0) — the release that shipped
the Rust kernel, added truth_gate.sh, and rewired Codex to prefer the deterministic binary. That
commit touched 67 files; hooks.json was not among them.

The fix is additive. The command hook runs before the existing prompt hook, whose text is
preserved byte-identical. Claude gets the deterministic floor and the semantic pass — the prompt
hook catches "deliverable missing / uncertainty hidden" cases a script cannot, and the script
guarantees a floor the prompt cannot.

2. docs/EVIDENCE.md staleness pass (rule 5)

Applying the doc's own maintenance rule. Nothing existing was rewritten — this only appends:

  • Denominator footnote. The existing 5.2% / 21.7% figures are code-sample level (576,000 samples).
    The same study also reports package-mention level: 2.23M mentions, 440,445 hallucinated (19.7%),
    205,474 unique names, plus a separate rerun experiment where 43% recurred in all ten reruns. Both are
    correct; readers hitting the other number elsewhere now know why.
  • 2026 replication. Five frontier models, rates compressed to 4.62%–6.10% (arXiv:2605.17062) —
    explicitly flagged single-author, non-peer-reviewed, with the author's own
    training-data-contamination confound noted. A caution against reusing 2024-cohort numbers as current,
    not a replacement.
  • Live-tracker caveat. Hallucinated-citation court trackers update continuously; a direct check on
    2026-08-13 returned 1,870 cases. Any hardcoded count needs a date.

Verification

  • All 5 branches of truth_gate.sh's fallback ladder exercised individually — every one fail-open.
  • stop_hook_active: true correctly allows (block-at-most-once respected).
  • Block fires only when a completion claim and an unfinished marker co-occur; no false positive on
    an honest TODO sketch.
  • Every statistic above traced to a primary source that was actually fetched.

Notes for review

  • No Rust or skill content changed. Two files, both additive.
  • One finding I did not fix, flagged for your call: hikmah validate returns "ok": true on the
    asymmetry above — it checks packaging, not behavior, so CI cannot currently catch a
    deterministic-gate-missing config. A behavioral consistency check between the two Stop adapters
    would close that.
  • Separately: the Rust hook.rs keeps no internal block-once marker and depends on the host setting
    stop_hook_active. Worth knowing; not touched here.
  • Opened for review, not for immediate merge.

Drafted with Claude Science.

hooks/codex.json wires Stop to {"type":"command"} -> hooks/truth_gate.sh
(deterministic), but hooks/hooks.json wires Stop to {"type":"prompt"} -- an
LLM-graded check. truth_gate.sh line 4 already reads
ROOT="${PLUGIN_ROOT:-${CLAUDE_PLUGIN_ROOT:-}}", so the deterministic gate
supports Claude Code; the Claude adapter simply never invoked it.

Git history suggests an oversight rather than a design choice: hooks.json has
one commit (v2.0.0), while codex.json also carries 2c72120 (v3.0.0), the
release that added truth_gate.sh and rewired Codex. That commit touched 67
files; hooks.json was not among them.

This adds the command hook BEFORE the existing prompt hook and leaves the
prompt text byte-identical, so Claude gets the deterministic floor plus the
semantic check rather than trading one for the other.

Also applies maintenance rule 5 to docs/EVIDENCE.md: adds the package-mention
denominator (2.23M mentions / 440,445 / 19.7%) alongside the existing
code-sample figures, notes the 2026 five-frontier-model replication
(4.62-6.10%, flagged single-author and non-peer-reviewed), and notes that
hallucinated-citation court trackers are live-updating so any hardcoded count
needs a date.
@CodeWithJuber
CodeWithJuber merged commit 89daa54 into main Aug 14, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant