Adopt the AHE harness findings; catch fabricated verification reports - #749
Merged
Conversation
added 12 commits
August 4, 2026 01:44
The verifier can latch green and then watch the agent throw the work away. Nothing intercepted a `git reset --hard`, a `git checkout -- .`, or an `rm` of a file in the verified diff, so a run could clean up after itself and report success on the cleanup's exit code. safe_state only notices after six weighted failures, which a confident happy-path destruction never produces. publish_guard arms at the fresh-green instant from the same worktree fingerprint safe_state already stamps — one git sample, one source of truth for what the run changed at green. It blocks only operations that DISCARD that work; ordinary editing of a protected file is untouched, since dirge sessions keep working after green and blocking rewrites would nag on every edit-test-edit cycle. off by default, advisory warns, blocking suppresses pre-dispatch. No override token. Separately, masks_failure handled `;` but not `\n`, which in bash means the same thing. So a validation block chained with newlines and ending in `echo passed` latched VerifiedGreen with the status belonging to the echo. The segment splitter feeding is_verification_command already documented the separators as "& | ; \n"; masks_failure was written against the same grammar and missed one. publish_guard's own tokenizer had the identical gap and could be bypassed with a newline — found in review, fixed alongside. Backslash-continued newlines are not separators in either place. Both come out of arXiv:2604.25850v4, which measured that its seed prompt already carried the "don't destroy verified state" advice and that only the execution-time interlock changed anything. Closes dirge-1elu.1, dirge-1elu.3
script_name_is_verification accepts any path-shaped command word whose basename carries a marker, so ./check.sh and /tmp/validate.sh counted as verification. A model could write its own validator, run it, watch it exit 0, and satisfy the gate without ever running the project's tests. The masked-command guard doesn't help — the script exits 0 honestly. Same asymmetry as the masked-command rule next to it: a self-authored script reporting success proves nothing (generator and validator share assumptions), so decline to record it and let the gate ask again; one reporting failure is still trustworthy, so the red stands. Only declines when recognition rests solely on the script-name branch — a check.sh that invokes cargo test is a wrapper, not a proxy, and still counts. Provenance comes from the modified registry plus mtime, since bash-created scripts never reach the registry. Refs dirge-1elu.2
open-issues and untracked-work emitted a bare SystemNotice, which is a UI/stream
event the model never sees. Both texts are imperatives aimed at the model
('close or defer them when done', 'add it with write_todo_list'), so both gates
fired, rendered to the user, and changed nothing about what the agent did next.
They now inject a tagged model-visible message; emit_harness_notices mirrors it,
so the human sees the same thing. Budgets are unchanged — this adds salience,
not frequency.
max_turns stays notice-only: the run is ending, there is no next turn to steer.
code_review's advisory arm already injected messages and needed no change.
Root cause was GateMode's doc comment describing Advisory as display-only while
code_review used it otherwise; rewritten to describe the policy (non-blocking,
one-shot, no react budget) rather than a delivery mechanism.
Also closes the cd-fold hole left in the previous commit: 'cd sub && ./check.sh'
resolved the script against the run cwd, so an agent-authored validator behind a
cd still latched green. Reuses fold_cd_dirs from the bash permission layer.
Within one finalization open-issues now preempts untracked-work rather than both
firing; each keeps its own one-shot budget, so both still fire across successive
finalizations.
Closes dirge-1elu.2, dirge-1elu.4
gate_tally counted gate and nudge firings as per-run totals, so two gates that always fire together at one boundary looked identical to two that never co-occur. That distinction is where arXiv:2604.25850v4 located its ceiling: stacked components pushing toward the same closure-style re-check gave back most of their individual gains, and on hard tasks made things worse. dirge has more gates than that harness did and this branch just added another. Boundaries are bracketed around the nudge poll and the finalization gate record, encoded onto the existing dirge::gates line. Observation only — a test runs the loop twice and asserts identical output, with a mechanism check so it can't pass vacuously. loop-ab.sh scrapes the new field and gained an N-arm mode. Four bugs in that awk turned up on the way, all silent — the report still printed, it just said "none" forever: - the arm list collected the model column instead of the tag column - the co-occurrence lookup key had its components reversed against the accumulation key - boundaries were read from a TSV column run_arm never wrote - an arm absent for one model divided by zero and aborted the report None of those could fail a Rust test, so loop-ab-selftest.sh now exercises the real reporting awk against a synthetic TSV with no models and no network. Its fixture is built so each assertion has a known-other-answer, and the treatment/allgates pair — the same two gates co-firing versus at separate boundaries — is the one that proves co-occurrence is measured at all. Closes dirge-1elu.6
Procedural memories were written by post-session review, injected into every later session at token cost, and never evaluated. effectiveness_bonus only moved when the agent volunteered a mark — discretionary, in-band self-report. A memory can now carry an optional expectation, settled deterministically in the post-session pass against the session digest. No LLM in the evaluation path. The expectation names a trigger and an expected command, not just one command, because the three cases have to stay distinguishable: the situation arose and the rule was followed (success), the situation arose and it wasn't (failure), the situation never arose (neither). Collapsing the last two is what makes an expectation unfalsifiable — it could only ever be confirmed. 'run cargo fmt before commit' is when=git commit, expect=cargo fmt; a session that committed without formatting is a measured failure of that memory, and a session that never committed says nothing about it. Inert by construction: existing rows migrate to a NULL column with no backfill, an unparseable or unknown wire value reads as no expectation, a memory outside the session-start snapshot is never evaluated, and nothing is ever deleted — only the existing bounded signed counters move. A VerificationGreen expectation was built and dropped before merge: the verifier gate is a run_loop local and final_verification was necessarily None at both post-session call sites, so it was settable but could never fire. Filed as dirge-1elu.7 rather than shipped inert. Follows arXiv:2604.25850v4 §3.3. Its §4.4.2 calibration is also why there is no regression-prediction field — those ran at ~2x random (43 issued, 5 landed, 40 unforeseen), so such a field would be worse than none. Closes dirge-1elu.5
Restores MemoryExpectation::VerificationGreen, dropped from dirge-1elu.5 because ExpectationSignals.final_verification was necessarily None at both post-session call sites and the variant would have been settable but unable to fire. The status is computed inside run_loop and consumed on the UI side off an AgentEvent, and nothing carried a value between those points. Three routes were tried before this one: returning it from run_agent_loop doesn't reach handle_done at all (that fires on the event, not the return); carrying it on AgentEnd -> AgentEvent::Done puts a verifier-internal type into the UI/stream event contract and touches ~15 construction sites across subagents, plan runtime, and background review; and persisting it on the session relocates the problem rather than solving it, since the session is owned by the UI and the value still has to travel first. So it goes through a process-global, like tools/modified and tools/snapshots, which exist for the same structural reason. Keyed by session id, because those are single-slot and a single slot would let a subagent or MCP-delegate run — sharing the process — hand its status to the wrong session. Reads take rather than copy, and recording None clears the slot, so a run that verified nothing cannot inherit an earlier run's green. Evaluation keeps the three-case discipline: green is success, observably red is failure, nothing-ran moves neither counter. The middle case is what makes the expectation falsifiable at all. Closes dirge-1elu.7
The h7_smoke scenarios call real providers on a plain cargo nextest run, so they went red for reasons that say nothing about dirge: DeepSeek retired the deepseek-chat model name, and the cerebras test asserted on the WORDING the model returned, failing whenever it paraphrased instead of quoting the tool result. Measured 4/5 on a clean tree before this change, 5/5 after. Both make a green suite look red locally, which trains people to ignore failures. Environmental conditions — auth, quota, rate limit, network, model-not-found — now skip rather than fail. The skip is keyed on the provider's own rejection wording, never on what the test asserts, so it stays lenient instead of becoming vacuous. The cerebras assertion is now structural: a final assistant turn exists and it follows the completed tool result, which is what the test was ever about. Also fixes cargo check --no-default-features, broken on three call sites (task.rs reaching for git_worktree, two more for semantic) rather than the one the bead recorded. Closes dirge-xets, dirge-8kag, dirge-1c77
headless_files_changed read the process-global modified-files registry, which is only emptied by clear_modified, so across a persistent MCP session it accumulated every file touched since the process started. A delegate call that changed nothing still reported the previous call's files — and that is the field a caller uses to answer 'did work actually happen'. Found while reviewing a delegation that reported results for work it had not done: the response listed six files from a task two calls earlier, so the evidence agreed with the claim. Only git diff --numstat disagreed. The tracker stays git-independent (issue #704: git is blind in a non-repo, off PATH, or when edits were committed mid-run); what changes is its scope. The registry now stores the version each path was marked at, so a caller can capture an epoch and ask what moved since. Deliberately an epoch rather than snapshot-and-diff: a re-touched path must appear in the later delta too, and a set difference would silently drop it. That false negative is the same bug pointed the other way, so there's a test that fails if the entry stops being re-versioned. Also adds an evidence object to the delegate response — the verification commands actually observed and their outcomes, plus turns and tool calls — so a claimed test result can be checked against observed state without re-running the suite. Closes dirge-d0e5.1
Fires when the final answer makes a specific claim the run's evidence doesn't
support: a test count or named-gate result ('4954 passed', 'clippy clean') when
the verifier saw no verification command at all, or an applied/fixed/changed
claim when zero files moved.
Both shapes are from a real delegation that reported passing gates it had never
run and fixes it had never applied.
Deterministic and free of judgement on purpose. A model asked to detect lying
can be talked out of it or can invent accusations; a pattern match conjoined
with 'the verifier observed zero verification commands' cannot. The conjunction
is also the over-detection control — a specific claim together with no observed
verification is unlikely to be innocent, and a gate that nags on honest work
gets switched off and then catches nothing. Quoted and attributed claims ('CI
reported ...') are carved out.
Model-visible tagged message, one-shot, GateMode tri-state, off by default.
Closes dirge-d0e5.2
run_unified_review already received the transcript, the run diff and the final verification status; nothing asked the critic to use them as a check on the assistant's account of its own actions. CRITIC_PREAMBLE only ever said to judge completeness and correctness. The prompt now carries what actually happened this run — files mutated, verification commands observed with their outcomes, tool-call count — and the preamble asks for claims the evidence contradicts, with the concrete mismatch. It's told not to invent a mismatch the evidence doesn't show, since a critic that hunts for lying finds it everywhere. Covers what the deterministic gate can't: 'I applied the two awk fixes' when the diff shows no edit to that file. Sharpens the existing critic pass rather than adding a second one. The files figure reuses the per-run epoch added for the delegate response, so there's one notion of what this run changed rather than two. Closes dirge-d0e5.3, dirge-d0e5
Fails on Linux CI, passes on macOS, and the fold logic is platform-neutral. Temporary — reverted once the CI output identifies the divergence.
Linux sets filesystem timestamps from a clock the kernel caches at timer-tick granularity, so a file written microseconds after SystemTime::now() can carry an mtime marginally earlier than it. macOS's finer clock hides this entirely, which is why the agent-authored-script check passed locally and failed on every Linux CI job that runs tests. The consequence was a real hole in the feature, not just a red test: a proxy validator the agent had just written read as pre-existing and latched the green the check exists to decline. The marker is now backdated by a second — far beyond any tick granularity, far below anything that would sweep in real repo scripts. Erring toward 'authored this run' declines a green and asks again, which is the direction this gate already prefers. Reverts the temporary diagnostic from the previous commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adopts the empirical findings from Agentic Harness Engineering (arXiv:2604.25850v4),
plus the anti-fabrication work that fell out of using dirge to build it.
Not adopting the evolution loop itself — that needs binary-verified benchmark tasks, E2B
sandboxes, ~32h per campaign, and an LLM debugger over 10M-token traces. What transfers
is the measured findings. The headline one endorses a bet this repo already made: in the
paper's component ablation, evolved memory alone was +5.6pp, tools alone +3.3pp,
middleware alone +2.2pp, and the evolved system prompt alone was −2.3pp. Prose loses
to mechanism, so every change here is a mechanism.
Harness findings
git reset --hard,git checkout -- .,or
rmof a file in the verified diff. Three of the paper's four winning iterationswere this one failure family, and its seed prompt already carried the advice that
didn't work. Arms at fresh-green off the fingerprint safe-state already stamps; blocks
discarding, not editing.
./check.shwritten this runcan't satisfy the verifier; one that invokes
cargo teststill can.\ntreated as the command separator it is.masks_failurehandled;but notits bash synonym, so a newline-chained block ending in
echo passedlatched green.SystemNotice, which the modelnever sees.
digest, no LLM in the path.
gate_tally, plus a self-test forloop-ab.sh.Fabrication detection
A delegate call reported passing gates it had never run and fixes it had never applied.
files_changedagreed with it, because it unioned a session-cumulative set — so the fielda caller uses to check "did work happen" said yes.
files_changedis now scoped to the delegation via a registry epoch, plus anevidenceobject on the response.
evidence. Off by default.
Pre-existing fixes
h7_smoke coupled the default suite to live providers (retired DeepSeek model name; the
cerebras test asserted on model phrasing — measured 4/5 on a clean tree, 5/5 after).
cargo check --no-default-featureswas broken on three call sites, not the one the beadrecorded.
Verification
4902 → 4987 tests.
fmt,clippy -D warnings(default andwindows-default),--no-default-featuresall clean. Every new mechanism was mutation-tested — the guard wasbroken deliberately to confirm its test goes red.
Closes dirge-1elu, dirge-d0e5, dirge-xets, dirge-8kag, dirge-1c77