Repository navigation
fix(audit): restore p7-quiet's warn for a failed --help and dash-less confirm_flags entries - #182
Open
brettdavies wants to merge 4 commits into
Conversation
This was referenced Oct 9, 2026
brettdavies
added this pull request to stack #166
October 9, 2026 15:40
…ts own Moving `p7-quiet` onto the shared help probe changed one more thing than how the flag is matched. The shared probe keeps whatever a `--help` printed before it crashed or timed out, so a tool whose `--help` printed `-q, --quiet` and then died on a signal passed, where the audit used to warn that it could not run `--help`. `run()` reads the `--help` call's status again, from the runner's cache, and grades the help only when the call exited on its own.
A `[p5] confirm_flags` entry such as `"noconfirm"` or `"y"` matched `--noconfirm` or `-y` while flags were looked up through `Flag::matches`, which added the dashes. The definition query compares spellings, so those entries stopped matching: the subcommand failed `p5-must-force-yes` and the entry was still listed under the accepted flags. `force_yes.rs` adds the dashes before it asks the query, two for a word and one for a single character, and takes an entry that already leads with a dash as written. Evidence still quotes the entry as the file spells it.
…tten Adding dashes to a `[p5] confirm_flags` entry that had none treated every entry not starting with `-` as a bare word. An entry spelled `+n` became `--+n` and stopped matching a help that declares `+n`. Only an entry that starts with a letter or digit is a bare word; one that leads with `-`, `+` or `/` is a name as written.
The gate that keeps a `--help` which did not exit on its own from being searched had a test for a crash and none for a timeout. The match moves into `status_after`, which takes the run status and the help, so the timeout case is tested without waiting out the runner's five seconds. Letting `RunStatus::Timeout` through the gate fails the new test.
brettdavies
force-pushed
the
fix/help-flags-quiet-probe-and-dashless-config
branch
from
October 9, 2026 17:21
cc46238 to
7e3ca56
Compare
12 of 13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
From the code review of the stack. Moving two audits onto the definition query changed more than how a flag is matched. Both differences are restored here.
p7-quietand a--helpthat does not exit on its own. The audit used to run--helpitself and warncould not run --help to detect quiet flagfor any outcome but a normal exit. In #171 it moved to the shared help probe, which keeps whatever a--helpprinted before it crashed or timed out. A tool whose--helpprinted-q, --quietand then died on a signal passed.run()reads the--helpcall's status again, from the runner's cache, and grades the help only when the call exited on its own, asp2-json-outputdoes. The cache is keyed by arguments, so this is the same run the probe read, not a second spawn.confirm_flagsentries without dashes.Flag::matchesadded the dashes to a name that had none, so a[p5] confirm_flagsentry such as"noconfirm"or"y"matched--noconfirmor-y. The definition query compares spellings. Since #164 those entries matched nothing: the subcommand failedp5-must-force-yeswith the entry still listed under the accepted flags.force_yes.rsadds the dashes before it asks the query, two for a word and one for a single character. Only a bare word gets them: an entry that leads with-,+or/is a name as written. Evidence quotes the entry as the file spells it.Changelog
Fixed
p7-must-quietpassing on the partial output of a--helpthat crashed or timed out. The row warns that--helpcould not be run..anc.toml[p5] confirm_flagsentries written without dashes ("noconfirm","y") no longer matching a flag.Documentation
confirm_flagssection says an entry without dashes is read as a flag.Type of Change
fix: Bug fix (non-breaking change which fixes an issue)Related Issues/Stories
docs/plans/2026-10-06-0034-fix-help-flag-names-across-frameworks-plan.md(review follow-up)p7-quiet), refactor(audit): look flags up through one model and one query #164 and fix(audit): match a declared confirm flag against the definitions the help declares #170 (declared flags onto the query)Testing
Test Summary:
-q, --quietand then kills itself warns that--helpcould not be run. The same gate is tested for a timeout throughstatus_after, which takes the run status and the help, so the test does not wait out the runner's five seconds. A declarednoconfirmconfirms a subcommand whose help declares--noconfirm, with evidencedestroy accepts noconfirm via .anc.toml [p5].confirm_flags, a declaredkconfirms on-k, and a declared+nconfirms on+n.cargo test --test dogfoodpasses.Negatives observed failing. The two new tests against the base (#181):
The plus-led entry against the commit that added the dashes, and the timeout case against a gate that lets a timeout through:
Expected moves. Written before the corpus run.
None. No scored tool's
--helpcrashes or times out: across the last corpus run everyp7-must-quietrow is a pass or the searched-and-not-found warn. No corpus target has aconfirm_flagsdeclaration. Replaying the full registry capture set through the base and head builds moves no row.Corpus before/after. Run after the table above was written. Base
afbfeffb80a4, head7e3ca56dfddd. Imagesha256:05db16cf226cd14a93a2682aea41b72d3e6b4d240cadba006f2881d3ffa996b3, networkbridge. 98 tools selected, 95 scored under both builds, 1 rerun three times (mods).Moved rows (0)
None.
Derived fields moved (0)
None.
No row and no derived field moved, as expected.
modsp3-should-about-long-about, the one row on the A/A noise list, is reported as not moved: both builds returned the same result for it in at least one run. No row is unstable, no harness bug is flagged, and no tool failed to score under one build only.cursor,nvidia-smiandxai-grok-buildare absent from the image and ran under neither build.Files Modified
Modified:
src/audits/behavioral/quiet.rssrc/audits/behavioral/force_yes.rsREADME.mdCreated:
Renamed:
Deleted:
Breaking Changes
Deployment Notes
Checklist