Skip to content

fix(audit): restore p7-quiet's warn for a failed --help and dash-less confirm_flags entries - #182

Open
brettdavies wants to merge 4 commits into
fix/help-flags-usage-wraps-and-examplesfrom
fix/help-flags-quiet-probe-and-dashless-config
Open

brettdavies wants to merge 4 commits into
fix/help-flags-usage-wraps-and-examplesfrom
fix/help-flags-quiet-probe-and-dashless-config

Conversation

@brettdavies

@brettdavies brettdavies commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Summary

From the code review of the stack. Moving two audits onto the definition query changed more than how a flag is matched. Both differences are restored here.

p7-quiet and a --help that does not exit on its own. The audit used to run --help itself and warn could not run --help to detect quiet flag for any outcome but a normal exit. In #171 it moved to the shared help probe, which keeps whatever a --help printed before it crashed or timed out. A tool whose --help printed -q, --quiet and then died on a signal passed. run() reads the --help call's status again, from the runner's cache, and grades the help only when the call exited on its own, as p2-json-output does. The cache is keyed by arguments, so this is the same run the probe read, not a second spawn.

confirm_flags entries without dashes. Flag::matches added the dashes to a name that had none, so a [p5] confirm_flags entry such as "noconfirm" or "y" matched --noconfirm or -y. The definition query compares spellings. Since #164 those entries matched nothing: the subcommand failed p5-must-force-yes with the entry still listed under the accepted flags. force_yes.rs adds the dashes before it asks the query, two for a word and one for a single character. Only a bare word gets them: an entry that leads with -, + or / is a name as written. Evidence quotes the entry as the file spells it.

Changelog

Fixed

  • Fix p7-must-quiet passing on the partial output of a --help that crashed or timed out. The row warns that --help could not be run.
  • Fix .anc.toml [p5] confirm_flags entries written without dashes ("noconfirm", "y") no longer matching a flag.

Documentation

  • The README's confirm_flags section says an entry without dashes is read as a flag.

Type of Change

  • fix: Bug fix (non-breaking change which fixes an issue)

Related Issues/Stories

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing completed
  • All tests passing

Test Summary:

  • Unit tests: a stand-in that prints -q, --quiet and then kills itself warns that --help could not be run. The same gate is tested for a timeout through status_after, which takes the run status and the help, so the test does not wait out the runner's five seconds. A declared noconfirm confirms a subcommand whose help declares --noconfirm, with evidence destroy accepts noconfirm via .anc.toml [p5].confirm_flags, a declared k confirms on -k, and a declared +n confirms on +n.
  • 1325 passing across the suite. Self-audit: cargo test --test dogfood passes.
  • Snapshots: none change.

Negatives observed failing. The two new tests against the base (#181):

---- audits::behavioral::force_yes::tests::a_declared_flag_written_without_dashes_is_a_flag stdout ----
assertion `left == right` failed
  left: Fail("destructive subcommand(s) whose --help lists no confirmation flag: destroy. Accepted flags: --force, --yes, -y, -f, --auto-approve, --assume-yes, --confirm, noconfirm via .anc.toml [p5].confirm_flags. Irreversible operations must require explicit confirmation so they can't be invoked accidentally.")
 right: Pass
---- audits::behavioral::quiet::tests::a_help_that_crashes_is_not_searched stdout ----
assertion `left == right` failed
  left: Pass
 right: Warn("could not run --help to detect quiet flag")
test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 1054 filtered out; finished in 1.08s

The plus-led entry against the commit that added the dashes, and the timeout case against a gate that lets a timeout through:

---- audits::behavioral::force_yes::tests::a_declared_flag_that_leads_with_a_plus_is_matched_as_written stdout ----
assertion `left == right` failed
  left: Fail("destructive subcommand(s) whose --help lists no confirmation flag: destroy. Accepted flags: --force, --yes, -y, -f, --auto-approve, --assume-yes, --confirm, +n via .anc.toml [p5].confirm_flags. Irreversible operations must require explicit confirmation so they can't be invoked accidentally.")
 right: Pass
---- audits::behavioral::quiet::tests::a_help_that_timed_out_is_not_searched stdout ----
assertion `left == right` failed
  left: Pass
 right: Warn("could not run --help to detect quiet flag")
test result: FAILED. 30 passed; 2 failed; 0 ignored; 0 measured; 1028 filtered out; finished in 1.08s

Expected moves. Written before the corpus run.

None. No scored tool's --help crashes or times out: across the last corpus run every p7-must-quiet row is a pass or the searched-and-not-found warn. No corpus target has a confirm_flags declaration. Replaying the full registry capture set through the base and head builds moves no row.

Corpus before/after. Run after the table above was written. Base afbfeffb80a4, head 7e3ca56dfddd. Image sha256:05db16cf226cd14a93a2682aea41b72d3e6b4d240cadba006f2881d3ffa996b3, network bridge. 98 tools selected, 95 scored under both builds, 1 rerun three times (mods).

Moved rows (0)

None.

Derived fields moved (0)

None.

No row and no derived field moved, as expected. mods p3-should-about-long-about, the one row on the A/A noise list, is reported as not moved: both builds returned the same result for it in at least one run. No row is unstable, no harness bug is flagged, and no tool failed to score under one build only. cursor, nvidia-smi and xai-grok-build are absent from the image and ran under neither build.

Files Modified

Modified:

  • src/audits/behavioral/quiet.rs
  • src/audits/behavioral/force_yes.rs
  • README.md

Created:

  • None.

Renamed:

  • None.

Deleted:

  • None.

Breaking Changes

  • No breaking changes

Deployment Notes

  • No special deployment steps required

Checklist

  • Code follows project conventions and style guidelines
  • Commit messages follow Conventional Commits
  • Self-review of code completed
  • Tests added/updated and passing
  • No new warnings or errors introduced
  • Changes are backward compatible (or breaking changes documented)

…ts own

Moving `p7-quiet` onto the shared help probe changed one more thing than how the flag is matched. The shared probe keeps whatever a `--help` printed before it crashed or timed out, so a tool whose `--help` printed `-q, --quiet` and then died on a signal passed, where the audit used to warn that it could not run `--help`.

`run()` reads the `--help` call's status again, from the runner's cache, and grades the help only when the call exited on its own.
A `[p5] confirm_flags` entry such as `"noconfirm"` or `"y"` matched `--noconfirm` or `-y` while flags were looked up through `Flag::matches`, which added the dashes. The definition query compares spellings, so those entries stopped matching: the subcommand failed `p5-must-force-yes` and the entry was still listed under the accepted flags.

`force_yes.rs` adds the dashes before it asks the query, two for a word and one for a single character, and takes an entry that already leads with a dash as written. Evidence still quotes the entry as the file spells it.
…tten

Adding dashes to a `[p5] confirm_flags` entry that had none treated every entry not starting with `-` as a bare word. An entry spelled `+n` became `--+n` and stopped matching a help that declares `+n`. Only an entry that starts with a letter or digit is a bare word; one that leads with `-`, `+` or `/` is a name as written.
The gate that keeps a `--help` which did not exit on its own from being searched had a test for a crash and none for a timeout. The match moves into `status_after`, which takes the run status and the help, so the timeout case is tested without waiting out the runner's five seconds. Letting `RunStatus::Timeout` through the gate fails the new test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant