diff --git a/CLAUDE.md b/CLAUDE.md index 0bf05eb..702db6b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -222,22 +222,26 @@ Existing field semantics: appends a post-summary hint via `BadgeInfo::text_hint()` when `eligible`; the same `tool.name` is used for the slug so the JSON `embed_markdown` and the printed hint can never disagree. -`0.8` addition (`MitigationInfo` carrier on `AuditResult`): - -- `MitigationInfo { using_domain_verbs, domain_match_count, domain_match_examples, builtin_match_count, subcommand_total - }` is attached to an `AuditResult` when the audit's verdict was assisted by a documented per-CLI opt-in. Today's only - producer is `src/audits/behavioral/standard_names.rs`: when `p6-standard-names` Passes because one or more subcommands - were recognized via `.anc.toml [p6] domain_verbs` (rather than the built-in `STANDARD_VERBS` list), the audit fills - `MitigationInfo` with the bifurcated match counts and the first `DOMAIN_MATCH_EXAMPLES_LIMIT` (5) matched domain-verb - names in encounter order. -- `AuditResultView` surfaces two top-level fields derived from the carrier: `using_domain_verbs: Option` and - `domain_match_count: Option`. Both use `skip_serializing_if = "Option::is_none"` so they are absent from rows - that did not consult `domain_verbs`. The Pass row's `evidence` field is populated (rather than `null`) via - `format_pass_evidence(&mitigation)`; rows without mitigation keep the historical `evidence: null` on Pass. -- The carrier shape is deliberately not audit-specific. Future audits that admit per-CLI mitigation (suppression profile - assistance, conditional-applicability config) can populate `MitigationInfo` with the same fields rather than growing - parallel typed carriers. The semantic contract is "this verdict depended on a self-declared opt-in; here is what - assisted." +`0.8` addition (`Mitigation` carrier on `AuditResult`): + +- `AuditResult.mitigation: Option` is set when a Pass depended on a `.anc.toml` setting. The semantic + contract is "this verdict depended on a self-declared opt-in; here is what assisted." `Mitigation` has two variants: + - `DomainVerbs(MitigationInfo)`: `MitigationInfo { using_domain_verbs, domain_match_count, domain_match_examples, + builtin_match_count, subcommand_total }`, filled by `src/audits/behavioral/standard_names.rs` when + `p6-standard-names` Passes because one or more subcommands were recognized via `.anc.toml [p6] domain_verbs` + (rather than the built-in `STANDARD_VERBS` list), with the bifurcated match counts and the first + `DOMAIN_MATCH_EXAMPLES_LIMIT` (5) matched domain-verb names in encounter order. + - `Config(String)`: every other setting. The prose names the setting, what it contributed, and the file that + supplied it, cited through `anc_toml::Sourced::cite` (`destroy accepts -auto-approve via .anc.toml + [p5].confirm_flags`). An audit that credits a setting returns a `types::Verdict { status, mitigation }` from its + core helper. +- `AuditResultView` surfaces two top-level fields from the `DomainVerbs` variant: `using_domain_verbs: Option` + and `domain_match_count: Option`. Both use `skip_serializing_if = "Option::is_none"` so they are absent from + every other row. `scorecard::pass_row_evidence` composes a Pass row's `evidence`: `AuditResult.pass_evidence` (what + the audit observed, such as the subcommands and flags `p7-limit` matched) first, then the mitigation prose + (`format_pass_evidence` for `DomainVerbs`, the prose itself for `Config`), joined by a semicolon. A Pass with + neither keeps `evidence: null`. A setting that shapes a non-Pass verdict says so in that status's own evidence + string. `0.9` addition (`ConfigHint` carrier on `AuditResult`): @@ -324,7 +328,10 @@ agentnative. Three rules guard the probe: 1. **Bare invocation prints help** (`cli.rs`): `arg_required_else_help = true` means children spawned with no args get instant help output instead of running `audit .`. This is also correct CLI behavior (P1 principle). 2. **Safe probing only** (`json_output.rs`): Subcommands are probed with `--help`/`--version` suffixes only, never bare. - Bare `subcmd --output json` is unsafe for any CLI with side-effecting subcommands. + Bare `subcmd --output json` is unsafe for any CLI with side-effecting subcommands. The one call outside those + suffixes is a `.anc.toml [p2] json_probe`: `run_declared_probe` runs the declared arguments exactly as written, + through the same `BinaryRunner` (no shell, timeout, closed stdin), because the tool's repository or the operator + chose them. An empty declaration declares nothing, so the bare invocation never runs. 3. **Binary discovery picks the newer of release/debug by mtime** (`src/project/bins.rs::rust_artifact`): when both `release/` and `debug/` exist in a target directory, the function returns the one with the more recent mtime. The target directory is cargo's (`src/project/cargo_target.rs`): `CARGO_TARGET_DIR`, else what `cargo @@ -336,7 +343,7 @@ agentnative. Three rules guard the probe: **Rules for new behavioral audits:** -- NEVER probe subcommands without `--help`/`--version` suffixes +- NEVER probe subcommands without `--help`/`--version` suffixes, except through a declared `[p2] json_probe` - NEVER remove `arg_required_else_help` from `Cli`; it prevents recursive self-invocation - NEVER revert binary discovery to the always-prefer-release shape (rule 3); that pattern silently masked `p2-must-schema-print` regressions during the v0.4.0 spec sync diff --git a/README.md b/README.md index bb848ac..b3dcd94 100644 --- a/README.md +++ b/README.md @@ -98,14 +98,74 @@ anc . -q ## Configuration (`.anc.toml`) -`.anc.toml` declares a CLI's own vocabulary so audits stop counting it against the CLI. It carries one setting today, -`[p6] domain_verbs`, which adds verbs to the standard list that `p6-may-standard-names` checks subcommand names against: +`.anc.toml` declares a CLI's own vocabulary so audits stop counting it against the CLI: ```toml +[p2] +json_probe = ["version", "--client", "-o", "json"] +schema_command = ["explain"] + +[p5] +confirm_flags = ["-auto-approve"] +not_destructive = ["clean"] + [p6] domain_verbs = ["post", "like", "repost", "timeline"] ``` +| Setting | Audit | What it declares | +| ---------------------- | ----------------------- | -------------------------------------------------------------------------- | +| `[p2] json_probe` | `p2-must-output-flag` | A read-only call that prints JSON, for anc to run and check | +| `[p2] schema_command` | `p2-must-schema-print` | The subcommand that prints the output schema, when it is not `schema` | +| `[p5] confirm_flags` | `p5-must-force-yes` | Flags that confirm a destructive subcommand, beside the built-in names | +| `[p5] not_destructive` | `p5-must-force-yes` | Subcommands whose names read as destructive but are not | +| `[p6] domain_verbs` | `p6-may-standard-names` | Verbs added to the standard list that subcommand names are checked against | + +### The settings + +`json_probe`: when a tool's help shows an `--output` or `--format` flag, `p2-must-output-flag` checks it by passing +`json` to the flag beside `--help` and `--version`, the only calls `anc` makes unprompted. Most tools answer those in +text, and then the row is `skip`: `anc` could not check, so the row is not scored. A declared probe is the call `anc` +runs instead, exactly as written: no shell, with the timeout, closed stdin, and `NO_COLOR=1` of every other probe. The +row passes when the call exits 0 and its stdout parses as JSON, and fails otherwise, with evidence naming the call and +the file: `` `kubectl version --client -o json` printed JSON; probe declared via .anc.toml [p2].json_probe ``. +Declare a call that only reads and exits on its own; `kubectl version -o json` contacts a cluster and exits 1 without +one, while `version --client` stays local. A probe that prints JSON also shows `p2-must-schema-print` that the tool +emits structured output when its help does not say so. The probe never stands in for the flag: with no `--output` or +`--format` in the help, the row stays `opt_out`. The nearest file that declares `json_probe` supplies it, and an empty +list declares nothing. + +`schema_command`: `p2-must-schema-print` looks for a `schema` subcommand or a `--schema` flag, at the top level and one +level down. A tool whose schema surface has another name, such as kubectl's `explain`, declares the subcommand path, one +token per level (`["emit", "schema"]` for ` emit schema`). The declared path counts when each token is listed in +its parent's `--help`; `anc` reads a parent's help with `--help` and never runs the command itself. The row then +passes, and its evidence names the command and the file: `` `kubectl explain` is the schema command declared via +.anc.toml [p2].schema_command ``. A declared path the help does not list leaves the row failing, and the evidence says +so. A built-in `schema` surface takes priority. Entries are lowercase. The nearest file that declares `schema_command` +supplies it, and an empty list declares nothing. + +`confirm_flags`: `p5-must-force-yes` requires each destructive subcommand's own `--help` to list a confirmation flag. +The built-in names are `--force`, `--yes`, `-y`, `-f`, `--auto-approve`, `--assume-yes`, and `--confirm`. A declared +flag counts beside them, and only where the subcommand's `--help` lists it, so a declaration names the flag and cannot +stand in for one. A single-dash name such as terraform's `-auto-approve` matches as a whole word. A pass that needed a +declared flag says so in the row's evidence, naming the subcommand, the flag, and the file: `destroy accepts +-auto-approve via .anc.toml [p5].confirm_flags`. + +`not_destructive`: `p5-must-force-yes` treats a subcommand as destructive by its name (`delete`, `rm`, `purge`, +`clean`, and names built on them). A tool whose `clean` clears regenerable caches, for one, declares it here, and the +audit leaves it out of the destructive set. Entries are lowercase; they are compared with the lowercased subcommand +name. The row's evidence names each subcommand left out and the file that declared it: `declared not destructive: clean +via .anc.toml [p5].not_destructive`. When every destructive subcommand is declared, the row is `skip`, as for a tool +with none. A declared subcommand still counts as a write for `p5-must-read-write-distinction`, because clearing a +cache changes state. + +`domain_verbs`: `p6-may-standard-names` passes when most subcommand names are standard verbs. A declared verb counts +beside the built-in list. Entries are lowercase; they are compared with the lowercased subcommand name. A pass that +needed a declared verb carries `using_domain_verbs` and `domain_match_count` on the row. + +A key `anc` does not know is ignored. A known key with a value of the wrong type, such as `confirm_flags = +"-auto-approve"`, is a parse error. + ### Where `anc` looks `anc` finds `.anc.toml` from the audit target's location, never from the directory you run it in. The files that apply, @@ -123,9 +183,11 @@ A binary counts the same whether you pass its path or `--command` resolves it on a dev build linked onto `PATH` keeps its repository's config. The repository root is the nearest directory holding a `.git` entry, so a linked `git worktree` checkout or a submodule is its own root. -The files merge: each one's `domain_verbs` adds to the ones above it, and a verb listed twice keeps its first position. -If any file in the chain cannot be read or parsed, no config applies, and the `p6-may-standard-names` warning names the -failing file, such as `could not parse .anc.toml at crates/cli/.anc.toml`. +The files merge. A list setting gathers every file's entries, each file's after the ones above it, and an entry listed +twice keeps its first position; evidence credits it to the nearer file, so a flag both your `~/.anc.toml` and the tool's +repository declare reads as the repository's. If any file in the chain cannot be read or parsed, no config applies: the +`p6-may-standard-names` warning names the failing file, such as `could not parse .anc.toml at crates/cli/.anc.toml`, and +a row another setting could have shaped ends its evidence with `No .anc.toml setting applied:` and the same message. ### `~/.anc.toml` @@ -133,7 +195,8 @@ The file in your home directory applies under every audit, inside repositories t vocabulary there. Two machines with different home files can score the same tool differently; CI runners have none. `AGENTNATIVE_HOME_CONFIG` relocates the file. Evidence and the hint then name it `$AGENTNATIVE_HOME_CONFIG`, never the path it holds, and when it names a file that does not exist, `anc` prints a `warning:` line on stderr and applies no -user-level file. +user-level file. A `json_probe` or `schema_command` here applies to every tool you audit, so point +`AGENTNATIVE_HOME_CONFIG` at a one-off file to declare either for a single run. ### A repository you fetched: `--repo` @@ -565,6 +628,10 @@ and how. Each scorecard conforms to the JSON Schema emitted by `anc emit schema` a slug exists, even below the floor, so the site renders an SVG for every scored tool (a regression below the floor shifts color rather than 404s). `convention_url` always points at `https://anc.dev/badge`. Schema `0.5` addition. +- `evidence` on a `pass` row: what the audit matched, for an audit that names it (`p7-limit` names each list command + and its limit flag), then, when an `.anc.toml` setting decided the pass, the setting, what it contributed, and the + file that supplied it, such as `destroy accepts -auto-approve via .anc.toml [p5].confirm_flags`. The two parts are + joined by a semicolon. `null` when the pass carries neither. See [Configuration](#configuration-anctoml). - `config_hint`: present only on a `p6-may-standard-names` warning when no `.anc.toml` declared `domain_verbs`. `files` lists where the setting can go, each as `{file, scope}`: `file` is `.anc.toml`, `~/.anc.toml`, or `$AGENTNATIVE_HOME_CONFIG` (never an absolute path), and `scope` is `repository` (the root of the repository this diff --git a/schema/scorecard.schema.json b/schema/scorecard.schema.json index b940f08..ea52ea7 100644 --- a/schema/scorecard.schema.json +++ b/schema/scorecard.schema.json @@ -89,7 +89,7 @@ }, "evidence": { "type": ["string", "null"], - "description": "Structured evidence for non-Pass statuses: the suppression-table marker, the unrecognized-flag list, the missing-file path, the antecedent propagation reason. `null` for clean Pass." + "description": "Structured evidence for non-Pass statuses: the suppression-table marker, the unrecognized-flag list, the missing-file path, the antecedent propagation reason. On a Pass: what the audit matched, for an audit that names it (`list (--max)`), then, when a `.anc.toml` setting decided the Pass, the setting, what it contributed, and the file that supplied it, e.g. `destroy accepts -auto-approve via .anc.toml [p5].confirm_flags`, the two joined by a semicolon. `null` for a Pass that carries neither." }, "confidence": { "type": "string", @@ -107,7 +107,7 @@ }, "using_domain_verbs": { "type": "boolean", - "description": "Schema 0.8+. `true` when the verdict depended on a per-CLI opt-in: today, `p6-standard-names` passing because `.anc.toml [p6] domain_verbs` recognized at least one subcommand. Absent from every row that consulted no opt-in." + "description": "Schema 0.8+. `true` when `p6-standard-names` passed because `.anc.toml [p6] domain_verbs` recognized at least one subcommand. Absent from every other row; a Pass another `.anc.toml` setting decided names that setting in `evidence`." }, "domain_match_count": { "type": "integer", diff --git a/src/anc_toml/mod.rs b/src/anc_toml/mod.rs index a10ab87..1f0c751 100644 --- a/src/anc_toml/mod.rs +++ b/src/anc_toml/mod.rs @@ -1,18 +1,31 @@ //! `.anc.toml` loader — per-CLI configuration found by the audit target's //! location. //! -//! Today the schema carries one section: -//! //! ```toml +//! [p2] +//! json_probe = ["version", "--client", "-o", "json"] +//! schema_command = ["explain"] +//! +//! [p5] +//! confirm_flags = ["-auto-approve"] +//! not_destructive = ["clean"] +//! //! [p6] //! domain_verbs = ["mentions", "timeline", "whoami"] //! ``` //! -//! `domain_verbs` extends the built-in standard-verb list consulted by the -//! `p6-may-standard-names` audit. Built-ins stay conservative across all -//! CLIs; CLIs whose platform vocabulary diverges from the global verb set -//! (e.g. an X CLI shipping `post` / `like` / `repost`) declare those verbs -//! here instead of being penalized for using their native terminology. +//! `json_probe` names a read-only call that prints JSON, which +//! `p2-must-output-flag` runs when it cannot validate JSON on its own probes; +//! `schema_command` names the subcommand `p2-must-schema-print` accepts as +//! the schema surface. +//! `confirm_flags` names flags that confirm a destructive subcommand, beside +//! the built-in names `p5-must-force-yes` accepts; `not_destructive` names +//! subcommands that audit leaves out of its destructive set. `domain_verbs` extends the +//! built-in standard-verb list consulted by the `p6-may-standard-names` +//! audit. Built-ins stay conservative across all CLIs; a CLI whose +//! vocabulary diverges from them (an X CLI shipping `post` / `like` / +//! `repost`, terraform's `-auto-approve`) declares it here instead of being +//! penalized for its native terminology. [`settings`] holds the shape. //! //! Loader contract ([`load_for_target`]): //! @@ -29,13 +42,15 @@ use std::ffi::OsString; use std::fs; use std::path::{Path, PathBuf}; -use serde::Deserialize; - use crate::types::{ConfigFile, ConfigScope}; mod chain; +mod settings; use chain::Chain; +pub use settings::{ + AncConfig, CONFIRM_FLAGS_KEY, JSON_PROBE_KEY, NOT_DESTRUCTIVE_KEY, SCHEMA_COMMAND_KEY, Sourced, +}; /// Filename probed in each directory of the chain. pub const ANC_TOML_FILENAME: &str = ".anc.toml"; @@ -46,23 +61,6 @@ pub const HOME_CONFIG_ENV: &str = "AGENTNATIVE_HOME_CONFIG"; /// The README section that explains where anc looks for `.anc.toml`. pub const DOCS_URL: &str = "https://github.com/brettdavies/agentnative-cli#configuration-anctoml"; -/// Root document for `.anc.toml`. New sections land here as the schema grows. -#[derive(Debug, Default, Deserialize, PartialEq, Eq)] -pub struct AncConfig { - #[serde(default)] - pub p6: P6Config, -} - -/// `[p6]` section — per-principle config bag for P6 (Predictable Surface). -#[derive(Debug, Default, Deserialize, PartialEq, Eq)] -pub struct P6Config { - /// Per-CLI domain vocabulary that augments the global standard-verb list. - /// Treated additively: a verb is recognized if it appears in the built-in - /// list OR this slice. - #[serde(default)] - pub domain_verbs: Vec, -} - /// Outcome of probing a target directory for `.anc.toml`. `Absent` is the /// happy path for the overwhelming majority of CLIs; `Loaded` carries the /// parsed config; `Invalid` carries a human-readable parse error suitable @@ -125,6 +123,22 @@ pub struct ResolvedConfig { pub settings_files: Vec, } +impl ResolvedConfig { + /// The merged settings; `None` when no file exists or the chain is void. + pub fn config(&self) -> Option<&AncConfig> { + self.load.as_config() + } + + /// The note a row carries when a void chain kept every setting from + /// applying, so a declaration that did nothing says why. + pub fn void_note(&self) -> Option { + match &self.load { + AncConfigLoad::Invalid(msg) => Some(format!("No .anc.toml setting applied: {msg}.")), + _ => None, + } + } +} + /// The user-level layer: [`HOME_CONFIG_ENV`] when set, otherwise /// `.anc.toml` in the home directory, and `None` without either. pub fn home_layer() -> Option { @@ -192,32 +206,25 @@ fn settings_files(chain: &Chain, home_label: &str) -> Vec { fn load_chain(chain: &Chain, home_label: &str) -> AncConfigLoad { let mut merged: Option = None; for file in chain.candidates() { + let shown = display_path(chain, file, home_label); let raw = match fs::read_to_string(file) { Ok(s) => s, Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue, Err(e) => { - let shown = display_path(chain, file, home_label); return AncConfigLoad::Invalid(format!("could not read .anc.toml at {shown}: {e}")); } }; - let cfg = match toml::from_str::(&raw) { - Ok(cfg) => cfg, + let settings = match toml::from_str::(&raw) { + Ok(settings) => settings, Err(e) => { - let shown = display_path(chain, file, home_label); return AncConfigLoad::Invalid(format!( "could not parse .anc.toml at {shown}: {e}" )); } }; - let verbs = &mut merged + merged .get_or_insert_with(AncConfig::default) - .p6 - .domain_verbs; - for verb in cfg.p6.domain_verbs { - if !verbs.contains(&verb) { - verbs.push(verb); - } - } + .absorb(settings, &shown); } merged.map_or(AncConfigLoad::Absent, AncConfigLoad::Loaded) } @@ -549,6 +556,256 @@ mod tests { } } + fn confirm_flags(load: &AncConfigLoad) -> Vec<(&str, &str)> { + match load { + AncConfigLoad::Loaded(cfg) => cfg + .p5 + .confirm_flags + .iter() + .map(|flag| (flag.value.as_str(), flag.file.as_str())) + .collect(), + other => panic!("expected Loaded, got {other:?}"), + } + } + + #[test] + fn loaded_confirm_flags_name_the_file_that_declared_them() { + let root = repo("confirm-parse"); + let cli = root.join("crates/cli"); + write(&root, "[p5]\nconfirm_flags = [\"-auto-approve\"]\n"); + write(&cli, "[p5]\nconfirm_flags = [\"--nuke\"]\n"); + + assert_eq!( + confirm_flags(&load_for_target(&cli, None, None).load), + [ + ("-auto-approve", ".anc.toml"), + ("--nuke", "crates/cli/.anc.toml") + ] + ); + } + + #[test] + fn repo_file_is_credited_over_the_home_file_for_a_shared_confirm_flag() { + let home = unique_tmp("confirm-home"); + let root = repo("confirm-precedence"); + write( + &home, + "[p5]\nconfirm_flags = [\"-auto-approve\", \"--nuke\"]\n", + ); + write(&root, "[p5]\nconfirm_flags = [\"-auto-approve\"]\n"); + let home_file = home.join(ANC_TOML_FILENAME); + + let load = load_chain( + &chain::resolve(&root, Some(&home_file), None), + DEFAULT_HOME_LABEL, + ); + + assert_eq!( + confirm_flags(&load), + [("-auto-approve", ".anc.toml"), ("--nuke", "~/.anc.toml")] + ); + } + + #[test] + fn confirm_flags_of_the_wrong_type_void_the_chain() { + let home = unique_tmp("confirm-void-home"); + let root = repo("confirm-void"); + write(&home, "[p5]\nconfirm_flags = [\"--nuke\"]\n"); + write(&root, "[p5]\nconfirm_flags = \"-auto-approve\"\n"); + let home_file = home.join(ANC_TOML_FILENAME); + + let msg = invalid(load_chain( + &chain::resolve(&root, Some(&home_file), None), + DEFAULT_HOME_LABEL, + )); + + assert!( + msg.starts_with("could not parse .anc.toml at .anc.toml:"), + "got: {msg}" + ); + } + + fn not_destructive(load: &AncConfigLoad) -> Vec<(&str, &str)> { + match load { + AncConfigLoad::Loaded(cfg) => cfg + .p5 + .not_destructive + .iter() + .map(|entry| (entry.value.as_str(), entry.file.as_str())) + .collect(), + other => panic!("expected Loaded, got {other:?}"), + } + } + + #[test] + fn loaded_not_destructive_names_the_file_that_declared_it() { + let root = repo("not-destructive-parse"); + write(&root, "[p5]\nnot_destructive = [\"clean\"]\n"); + + assert_eq!( + not_destructive(&load_for_target(&root, None, None).load), + [("clean", ".anc.toml")] + ); + } + + #[test] + fn repo_file_is_credited_over_the_home_file_for_a_shared_not_destructive_entry() { + let home = unique_tmp("not-destructive-home"); + let root = repo("not-destructive-precedence"); + write(&home, "[p5]\nnot_destructive = [\"rmdir\", \"clean\"]\n"); + write(&root, "[p5]\nnot_destructive = [\"clean\"]\n"); + let home_file = home.join(ANC_TOML_FILENAME); + + let load = load_chain( + &chain::resolve(&root, Some(&home_file), None), + DEFAULT_HOME_LABEL, + ); + + assert_eq!( + not_destructive(&load), + [("rmdir", "~/.anc.toml"), ("clean", ".anc.toml")] + ); + } + + #[test] + fn not_destructive_of_the_wrong_type_voids_the_chain() { + let root = repo("not-destructive-void"); + write(&root, "[p5]\nnot_destructive = \"clean\"\n"); + + let msg = invalid(load_for_target(&root, None, None).load); + + assert!( + msg.starts_with("could not parse .anc.toml at .anc.toml:"), + "got: {msg}" + ); + } + + fn json_probe(load: &AncConfigLoad) -> Option<(Vec<&str>, &str)> { + match load { + AncConfigLoad::Loaded(cfg) => cfg.p2.json_probe.as_ref().map(|probe| { + ( + probe.value.iter().map(String::as_str).collect(), + probe.file.as_str(), + ) + }), + other => panic!("expected Loaded, got {other:?}"), + } + } + + #[test] + fn loaded_json_probe_names_the_file_that_declared_it() { + let root = repo("json-probe-parse"); + write( + &root, + "[p2]\njson_probe = [\"version\", \"-o\", \"json\"]\n", + ); + + assert_eq!( + json_probe(&load_for_target(&root, None, None).load), + Some((vec!["version", "-o", "json"], ".anc.toml")) + ); + } + + #[test] + fn repo_json_probe_replaces_the_home_one() { + let home = unique_tmp("json-probe-home"); + let root = repo("json-probe-precedence"); + write( + &home, + "[p2]\njson_probe = [\"version\", \"-o\", \"json\"]\n", + ); + write( + &root, + "[p2]\njson_probe = [\"repo\", \"list\", \"-o\", \"json\"]\n", + ); + let home_file = home.join(ANC_TOML_FILENAME); + + let load = load_chain( + &chain::resolve(&root, Some(&home_file), None), + DEFAULT_HOME_LABEL, + ); + + assert_eq!( + json_probe(&load), + Some((vec!["repo", "list", "-o", "json"], ".anc.toml")) + ); + } + + #[test] + fn json_probe_of_the_wrong_type_voids_the_chain() { + let home = unique_tmp("json-probe-void-home"); + let root = repo("json-probe-void"); + write( + &home, + "[p2]\njson_probe = [\"version\", \"-o\", \"json\"]\n", + ); + write(&root, "[p2]\njson_probe = \"version -o json\"\n"); + let home_file = home.join(ANC_TOML_FILENAME); + + let msg = invalid(load_chain( + &chain::resolve(&root, Some(&home_file), None), + DEFAULT_HOME_LABEL, + )); + + assert!( + msg.starts_with("could not parse .anc.toml at .anc.toml:"), + "got: {msg}" + ); + } + + fn schema_command(load: &AncConfigLoad) -> Option<(Vec<&str>, &str)> { + match load { + AncConfigLoad::Loaded(cfg) => cfg.p2.schema_command.as_ref().map(|path| { + ( + path.value.iter().map(String::as_str).collect(), + path.file.as_str(), + ) + }), + other => panic!("expected Loaded, got {other:?}"), + } + } + + #[test] + fn loaded_schema_command_names_the_file_that_declared_it() { + let root = repo("schema-command-parse"); + let cli = root.join("cli"); + write(&cli, "[p2]\nschema_command = [\"explain\"]\n"); + + assert_eq!( + schema_command(&load_for_target(&cli, None, None).load), + Some((vec!["explain"], "cli/.anc.toml")) + ); + } + + #[test] + fn repo_schema_command_replaces_the_home_one() { + let home = unique_tmp("schema-command-home"); + let root = repo("schema-command-precedence"); + write(&home, "[p2]\nschema_command = [\"describe\"]\n"); + write(&root, "[p2]\nschema_command = [\"explain\"]\n"); + let home_file = home.join(ANC_TOML_FILENAME); + + let load = load_chain( + &chain::resolve(&root, Some(&home_file), None), + DEFAULT_HOME_LABEL, + ); + + assert_eq!(schema_command(&load), Some((vec!["explain"], ".anc.toml"))); + } + + #[test] + fn schema_command_of_the_wrong_type_voids_the_chain() { + let root = repo("schema-command-void"); + write(&root, "[p2]\nschema_command = \"explain\"\n"); + + let msg = invalid(load_for_target(&root, None, None).load); + + assert!( + msg.starts_with("could not parse .anc.toml at .anc.toml:"), + "got: {msg}" + ); + } + #[test] fn file_target_reads_the_config_beside_it() { let dir = unique_tmp("file-target"); @@ -685,9 +942,10 @@ mod tests { #[test] fn as_config_returns_inner_for_loaded() { let cfg = AncConfig { - p6: P6Config { + p6: settings::P6Config { domain_verbs: vec!["mentions".into()], }, + ..AncConfig::default() }; let load = AncConfigLoad::Loaded(cfg); let got = load.as_config().expect("as_config returns inner"); diff --git a/src/anc_toml/settings.rs b/src/anc_toml/settings.rs new file mode 100644 index 0000000..970e536 --- /dev/null +++ b/src/anc_toml/settings.rs @@ -0,0 +1,282 @@ +//! The settings `.anc.toml` carries: as one file writes them, and as the +//! chain merges them into [`AncConfig`]. + +use serde::Deserialize; + +/// How evidence cites `[p2] json_probe`. +pub const JSON_PROBE_KEY: &str = "[p2].json_probe"; + +/// How evidence cites `[p2] schema_command`. +pub const SCHEMA_COMMAND_KEY: &str = "[p2].schema_command"; + +/// How evidence cites `[p5] confirm_flags`. +pub const CONFIRM_FLAGS_KEY: &str = "[p5].confirm_flags"; + +/// How evidence cites `[p5] not_destructive`. +pub const NOT_DESTRUCTIVE_KEY: &str = "[p5].not_destructive"; + +/// A merged setting and the file that supplied it. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Sourced { + pub value: T, + /// The file, named the way evidence names files: repo-relative inside a + /// repository, `~/.anc.toml` or `$AGENTNATIVE_HOME_CONFIG` for the + /// user-level file, never an absolute path. + pub file: String, +} + +impl Sourced { + /// The file and the setting, the way evidence cites them: + /// `.anc.toml [p5].confirm_flags`. + pub fn cite(&self, key: &str) -> String { + format!("{} {key}", self.file) + } +} + +/// The settings of every file in the chain, merged. +#[derive(Debug, Default, PartialEq, Eq)] +pub struct AncConfig { + pub p2: P2Config, + pub p5: P5Config, + pub p6: P6Config, +} + +/// `[p2]`: P2 (Structured, Parseable Output). +#[derive(Debug, Default, PartialEq, Eq)] +pub struct P2Config { + /// The arguments of a read-only call that prints JSON, run as written + /// when `p2-must-output-flag` cannot validate JSON on its own probes. + /// The nearest file that declares one supplies it. + pub json_probe: Option>>, + /// The subcommand path that prints the tool's output schema, for a + /// tool whose schema surface is not named `schema`. The nearest file + /// that declares one supplies it. + pub schema_command: Option>>, +} + +/// `[p5]`: P5 (Safe Retries and Explicit Mutation Boundaries). +#[derive(Debug, Default, PartialEq, Eq)] +pub struct P5Config { + /// Flags that confirm a destructive subcommand non-interactively, + /// accepted beside the built-in names. + pub confirm_flags: Vec>, + /// Subcommand names the tool declares are not destructive, compared + /// with the lowercased name. + pub not_destructive: Vec>, +} + +/// `[p6]`: P6 (Composable and Predictable Command Structure). +#[derive(Debug, Default, Deserialize, PartialEq, Eq)] +pub struct P6Config { + /// Per-CLI domain vocabulary that augments the global standard-verb list. + /// Treated additively: a verb is recognized if it appears in the built-in + /// list OR this slice. + #[serde(default)] + pub domain_verbs: Vec, +} + +/// One file's settings, as written. Unknown keys are ignored; a known key +/// of the wrong type fails the parse. +#[derive(Debug, Default, Deserialize)] +pub(super) struct FileConfig { + #[serde(default)] + p2: FileP2, + #[serde(default)] + p5: FileP5, + #[serde(default)] + p6: P6Config, +} + +#[derive(Debug, Default, Deserialize)] +struct FileP2 { + #[serde(default)] + json_probe: Option>, + #[serde(default)] + schema_command: Option>, +} + +#[derive(Debug, Default, Deserialize)] +struct FileP5 { + #[serde(default)] + confirm_flags: Vec, + #[serde(default)] + not_destructive: Vec, +} + +impl AncConfig { + /// Fold one file's settings onto the files of lower precedence already + /// merged. `file` is how evidence names that file. + pub(super) fn absorb(&mut self, settings: FileConfig, file: &str) { + for verb in settings.p6.domain_verbs { + if !self.p6.domain_verbs.contains(&verb) { + self.p6.domain_verbs.push(verb); + } + } + replace_command(&mut self.p2.json_probe, settings.p2.json_probe, file); + replace_command( + &mut self.p2.schema_command, + settings.p2.schema_command, + file, + ); + merge_list(&mut self.p5.confirm_flags, settings.p5.confirm_flags, file); + merge_list( + &mut self.p5.not_destructive, + settings.p5.not_destructive, + file, + ); + } +} + +/// A nearer file's command replaces the one merged so far. An empty list +/// declares nothing, so the command below it stands. +fn replace_command( + merged: &mut Option>>, + declared: Option>, + file: &str, +) { + if let Some(value) = declared.filter(|args| !args.is_empty()) { + *merged = Some(Sourced { + value, + file: file.to_string(), + }); + } +} + +/// Append each entry `merged` lacks. An entry already present keeps its +/// position and is credited to `file`, the nearer of the two, so evidence +/// names the repository's file over the user-level one. +fn merge_list(merged: &mut Vec>, entries: Vec, file: &str) { + for value in entries { + match merged.iter_mut().find(|entry| entry.value == value) { + Some(entry) => entry.file = file.to_string(), + None => merged.push(Sourced { + value, + file: file.to_string(), + }), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn parse(raw: &str) -> FileConfig { + toml::from_str(raw).expect("valid settings") + } + + fn flags(cfg: &AncConfig) -> Vec<(&str, &str)> { + cfg.p5 + .confirm_flags + .iter() + .map(|flag| (flag.value.as_str(), flag.file.as_str())) + .collect() + } + + #[test] + fn nearer_file_adds_flags_after_and_takes_credit_for_shared_ones() { + let mut cfg = AncConfig::default(); + cfg.absorb( + parse("[p5]\nconfirm_flags = [\"-auto-approve\", \"--nuke\"]\n"), + "~/.anc.toml", + ); + cfg.absorb( + parse("[p5]\nconfirm_flags = [\"--nuke\", \"--really\"]\n"), + ".anc.toml", + ); + + assert_eq!( + flags(&cfg), + [ + ("-auto-approve", "~/.anc.toml"), + ("--nuke", ".anc.toml"), + ("--really", ".anc.toml"), + ] + ); + } + + #[test] + fn not_destructive_entries_merge_like_confirm_flags() { + let mut cfg = AncConfig::default(); + cfg.absorb( + parse("[p5]\nnot_destructive = [\"clean\", \"rmdir\"]\n"), + "~/.anc.toml", + ); + cfg.absorb(parse("[p5]\nnot_destructive = [\"clean\"]\n"), ".anc.toml"); + + let entries: Vec<(&str, &str)> = cfg + .p5 + .not_destructive + .iter() + .map(|entry| (entry.value.as_str(), entry.file.as_str())) + .collect(); + assert_eq!(entries, [("clean", ".anc.toml"), ("rmdir", "~/.anc.toml")]); + } + + #[test] + fn the_nearer_json_probe_replaces_the_one_below_and_an_empty_one_declares_nothing() { + let mut cfg = AncConfig::default(); + cfg.absorb( + parse("[p2]\njson_probe = [\"version\", \"-o\", \"json\"]\n"), + "~/.anc.toml", + ); + cfg.absorb(parse("[p2]\njson_probe = []\n"), "crates/.anc.toml"); + assert_eq!( + cfg.p2.json_probe, + Some(Sourced { + value: vec!["version".into(), "-o".into(), "json".into()], + file: "~/.anc.toml".into(), + }) + ); + + cfg.absorb( + parse("[p2]\njson_probe = [\"version\", \"--client\", \"-o\", \"json\"]\n"), + ".anc.toml", + ); + assert_eq!( + cfg.p2.json_probe, + Some(Sourced { + value: vec![ + "version".into(), + "--client".into(), + "-o".into(), + "json".into() + ], + file: ".anc.toml".into(), + }) + ); + } + + #[test] + fn the_nearer_schema_command_replaces_the_one_below() { + let mut cfg = AncConfig::default(); + cfg.absorb( + parse("[p2]\nschema_command = [\"explain\"]\n"), + "~/.anc.toml", + ); + cfg.absorb( + parse("[p2]\nschema_command = [\"emit\", \"schema\"]\n"), + ".anc.toml", + ); + + assert_eq!( + cfg.p2.schema_command, + Some(Sourced { + value: vec!["emit".into(), "schema".into()], + file: ".anc.toml".into(), + }) + ); + } + + #[test] + fn cite_names_the_file_then_the_setting() { + let flag = Sourced { + value: "-auto-approve".to_string(), + file: "crates/cli/.anc.toml".to_string(), + }; + assert_eq!( + flag.cite(CONFIRM_FLAGS_KEY), + "crates/cli/.anc.toml [p5].confirm_flags" + ); + } +} diff --git a/src/audits/behavioral/force_yes.rs b/src/audits/behavioral/force_yes.rs index 6fc2865..e7180ae 100644 --- a/src/audits/behavioral/force_yes.rs +++ b/src/audits/behavioral/force_yes.rs @@ -6,18 +6,32 @@ //! intent auditable in process tables and shell history. //! //! Rubric: identify destructive subcommands via [`destructive_subcommands`], -//! probe each one's `--help`, and audit for the presence of `--force`, -//! `--yes`, `-y`, or `-f`. Fail when any destructive subcommand lacks both. -//! Vacuous Skip when the binary has no destructive subcommands. +//! less any the `.anc.toml` chain declares in `[p5] not_destructive`, probe +//! each one's `--help`, and audit for one of [`CONFIRM_FLAGS`] or a flag the +//! chain declares in `[p5] confirm_flags`. Fail when any destructive +//! subcommand lists none. Vacuous Skip when the binary has no destructive +//! subcommands. +use crate::anc_toml::{CONFIRM_FLAGS_KEY, NOT_DESTRUCTIVE_KEY, Sourced}; use crate::audit::Audit; use crate::audits::behavioral::destructive_ops::destructive_subcommands; use crate::audits::behavioral::subcommand_help::probe_subcommands; use crate::project::Project; use crate::runner::HelpOutput; -use crate::types::{AuditGroup, AuditLayer, AuditResult, AuditStatus, Confidence}; +use crate::types::{ + AuditGroup, AuditLayer, AuditResult, AuditStatus, Confidence, Mitigation, Verdict, +}; -const REQUIRED_FLAGS: &[&str] = &["--force", "--yes", "-y", "-f"]; +/// Flags that confirm a destructive operation non-interactively. +const CONFIRM_FLAGS: &[&str] = &[ + "--force", + "--yes", + "-y", + "-f", + "--auto-approve", + "--assume-yes", + "--confirm", +]; pub struct ForceYesAudit; @@ -47,26 +61,36 @@ impl Audit for ForceYesAudit { } fn run(&self, project: &Project) -> anyhow::Result { - let status = match project.help_output() { - None => AuditStatus::Skip("could not probe --help".into()), + let p5 = project.anc_config.config().map(|cfg| &cfg.p5); + let declared_flags = p5.map_or(&[][..], |p5| p5.confirm_flags.as_slice()); + let not_destructive = p5.map_or(&[][..], |p5| p5.not_destructive.as_slice()); + let verdict = match project.help_output() { + None => AuditStatus::Skip("could not probe --help".into()).into(), Some(top_help) => { - let destructive: Vec = destructive_subcommands(top_help) - .into_iter() - .cloned() - .collect(); - if destructive.is_empty() { - AuditStatus::Skip( - "no destructive subcommands detected; MUST applies conditionally to CLIs \ - with destructive operations." - .into(), - ) + let (destructive, excluded) = + without_declared(&destructive_subcommands(top_help), not_destructive); + let verdict = if destructive.is_empty() { + let found = if excluded.is_empty() { + "no destructive subcommands detected" + } else { + "no destructive subcommands remain" + }; + AuditStatus::Skip(format!( + "{found}; MUST applies conditionally to CLIs with destructive operations." + )) + .into() } else { let runner = project.runner_ref(); let subhelp = probe_subcommands(runner, top_help); - audit_force_yes(&destructive, &subhelp) - } + audit_force_yes(&destructive, &subhelp, declared_flags) + }; + with_exclusions(verdict, &excluded) } }; + let status = match project.anc_config.void_note() { + Some(note) => verdict.status.with_note(¬e), + None => verdict.status, + }; Ok(AuditResult { id: self.id().to_string(), label: self.label().into(), @@ -74,42 +98,102 @@ impl Audit for ForceYesAudit { layer: self.layer(), status, confidence: Confidence::High, - mitigation: None, + mitigation: verdict.mitigation, config_hint: None, pass_evidence: None, }) } } +/// Pass when every destructive subcommand's `--help` lists a built-in +/// confirmation flag or one of `declared_flags`. A built-in match takes +/// priority; a Pass that needed a declared flag names the subcommand, the +/// flag, and the file that declared it. pub(crate) fn audit_force_yes( destructive: &[String], subhelp: &[(String, HelpOutput)], -) -> AuditStatus { + declared_flags: &[Sourced], +) -> Verdict { let mut missing: Vec<&str> = Vec::new(); + let mut confirmed_by_declaration: Vec = Vec::new(); for verb in destructive { - let entry = subhelp.iter().find(|(name, _)| name == verb); - match entry { - Some((_, help)) if has_force_or_yes(help) => {} - Some(_) => missing.push(verb.as_str()), + let Some((_, help)) = subhelp.iter().find(|(name, _)| name == verb) else { + missing.push(verb.as_str()); + continue; + }; + if CONFIRM_FLAGS.iter().any(|flag| help.advertises_flag(flag)) { + continue; + } + match declared_flags + .iter() + .find(|flag| help.advertises_flag(&flag.value)) + { + Some(flag) => confirmed_by_declaration.push(format!( + "{verb} accepts {} via {}", + flag.value, + flag.cite(CONFIRM_FLAGS_KEY) + )), None => missing.push(verb.as_str()), } } if missing.is_empty() { - AuditStatus::Pass - } else { - AuditStatus::Fail(format!( - "destructive subcommand(s) without `--force` or `--yes`: {}. \ - Irreversible operations must require explicit confirmation so \ - they can't be invoked accidentally.", - missing.join(", ") - )) + return Verdict { + status: AuditStatus::Pass, + mitigation: (!confirmed_by_declaration.is_empty()) + .then(|| Mitigation::Config(confirmed_by_declaration.join("; "))), + }; + } + AuditStatus::Fail(format!( + "destructive subcommand(s) whose --help lists no confirmation flag: {}. \ + Accepted flags: {}. Irreversible operations must require explicit \ + confirmation so they can't be invoked accidentally.", + missing.join(", "), + accepted_flags(declared_flags), + )) + .into() +} + +/// Split `detected` into the subcommands that stay destructive and, cited +/// with the file that declared it, each one `not_destructive` removes. +pub(crate) fn without_declared( + detected: &[&String], + not_destructive: &[Sourced], +) -> (Vec, Vec) { + let mut destructive = Vec::new(); + let mut excluded = Vec::new(); + for name in detected { + let lower = name.to_lowercase(); + match not_destructive.iter().find(|entry| entry.value == lower) { + Some(entry) => excluded.push(format!("{name} via {}", entry.cite(NOT_DESTRUCTIVE_KEY))), + None => destructive.push((*name).clone()), + } + } + (destructive, excluded) +} + +/// Name the subcommands a declaration removed: in a Pass's evidence beside +/// any other setting it needed, or at the end of any other status. +fn with_exclusions(verdict: Verdict, excluded: &[String]) -> Verdict { + if excluded.is_empty() { + return verdict; } + let declared = format!("declared not destructive: {}", excluded.join(", ")); + let note = format!("Subcommands {declared}."); + verdict.crediting(declared, ¬e) } -fn has_force_or_yes(help: &HelpOutput) -> bool { - help.flags() +/// The built-in names, then each declared flag with the file it came from. +fn accepted_flags(declared_flags: &[Sourced]) -> String { + CONFIRM_FLAGS .iter() - .any(|f| REQUIRED_FLAGS.iter().any(|name| f.matches(name))) + .map(|flag| (*flag).to_string()) + .chain( + declared_flags + .iter() + .map(|flag| format!("{} via {}", flag.value, flag.cite(CONFIRM_FLAGS_KEY))), + ) + .collect::>() + .join(", ") } #[cfg(test)] @@ -120,6 +204,178 @@ mod tests { HelpOutput::from_raw(raw) } + fn declared(flag: &str, file: &str) -> Sourced { + Sourced { + value: flag.to_string(), + file: file.to_string(), + } + } + + const GO_STYLE_DESTROY_HELP: &str = "Usage: tool destroy [options]\n\nOptions:\n\n -auto-approve Skip interactive approval.\n\n -lock=false Don't hold a lock.\n"; + + #[test] + fn a_declared_flag_confirms_where_the_builtins_do_not() { + let subhelp = vec![("destroy".to_string(), hp(GO_STYLE_DESTROY_HELP))]; + let destructive = ["destroy".to_string()]; + + assert!(matches!( + audit_force_yes(&destructive, &subhelp, &[]).status, + AuditStatus::Fail(_) + )); + + let verdict = audit_force_yes( + &destructive, + &subhelp, + &[declared("-auto-approve", ".anc.toml")], + ); + assert_eq!(verdict.status, AuditStatus::Pass); + assert_eq!( + verdict.mitigation, + Some(Mitigation::Config( + "destroy accepts -auto-approve via .anc.toml [p5].confirm_flags".into() + )) + ); + } + + #[test] + fn a_builtin_flag_takes_priority_over_a_declared_one() { + let subhelp = vec![ + ( + "delete".to_string(), + hp("Options:\n -auto-approve Skip approval.\n --force Skip approval.\n"), + ), + ("destroy".to_string(), hp(GO_STYLE_DESTROY_HELP)), + ]; + let destructive = ["delete".to_string(), "destroy".to_string()]; + + let verdict = audit_force_yes( + &destructive, + &subhelp, + &[declared("-auto-approve", "~/.anc.toml")], + ); + + assert_eq!(verdict.status, AuditStatus::Pass); + assert_eq!( + verdict.mitigation, + Some(Mitigation::Config( + "destroy accepts -auto-approve via ~/.anc.toml [p5].confirm_flags".into() + )) + ); + } + + #[test] + fn a_declared_flag_missing_from_the_subcommand_help_still_fails_and_is_named() { + let subhelp = vec![( + "destroy".to_string(), + hp("Usage: tool destroy [options]\n\n Destroy everything.\n"), + )]; + + match audit_force_yes( + &["destroy".to_string()], + &subhelp, + &[declared("-auto-approve", ".anc.toml")], + ) + .status + { + AuditStatus::Fail(msg) => assert!( + msg.contains("--confirm, -auto-approve via .anc.toml [p5].confirm_flags."), + "{msg}" + ), + other => panic!("expected Fail, got {other:?}"), + } + } + + #[test] + fn a_declared_not_destructive_subcommand_leaves_the_destructive_set() { + let detected = ["clean".to_string(), "Purge".to_string()]; + let detected: Vec<&String> = detected.iter().collect(); + + let (destructive, excluded) = + without_declared(&detected, &[declared("clean", ".anc.toml")]); + + assert_eq!(destructive, ["Purge"]); + assert_eq!(excluded, ["clean via .anc.toml [p5].not_destructive"]); + } + + /// A tool with `clean` (no confirmation flag) and `delete --force`. + const CLEAN_AND_DELETE_CLI: &str = r#"case "$*" in + "clean --help") printf 'Usage: tool clean\n\nOptions:\n -h, --help Show help.\n' ;; + "delete --help") printf 'Usage: tool delete \n\nOptions:\n --force Skip the prompt.\n' ;; + *) printf 'Usage: tool \n\nCommands:\n clean Remove cached logs\n delete Delete an item\n' ;; +esac"#; + + fn run_with_not_destructive(script: &str, names: &[&str]) -> AuditResult { + let mut project = crate::audits::behavioral::tests::test_project_with_sh_script(script); + let mut cfg = crate::anc_toml::AncConfig::default(); + cfg.p5.not_destructive = names + .iter() + .map(|name| declared(name, ".anc.toml")) + .collect(); + project.anc_config.load = crate::anc_toml::AncConfigLoad::Loaded(cfg); + ForceYesAudit.run(&project).expect("audit runs") + } + + #[test] + fn declaring_the_unconfirmed_subcommand_not_destructive_passes_and_says_so() { + let without = run_with_not_destructive(CLEAN_AND_DELETE_CLI, &[]); + assert!( + matches!(without.status, AuditStatus::Fail(_)), + "{:?}", + without.status + ); + + let with = run_with_not_destructive(CLEAN_AND_DELETE_CLI, &["clean"]); + + assert_eq!(with.status, AuditStatus::Pass); + assert_eq!( + with.mitigation, + Some(Mitigation::Config( + "declared not destructive: clean via .anc.toml [p5].not_destructive".into() + )) + ); + } + + #[test] + fn declaring_every_destructive_subcommand_not_destructive_skips_and_names_them() { + let script = r#"case "$*" in + "clean --help") printf 'Usage: tool clean\n' ;; + *) printf 'Usage: tool \n\nCommands:\n clean Remove cached logs\n list List items\n' ;; +esac"#; + + match run_with_not_destructive(script, &["clean"]).status { + AuditStatus::Skip(msg) => assert_eq!( + msg, + "no destructive subcommands remain; MUST applies conditionally to CLIs with \ + destructive operations. Subcommands declared not destructive: clean via \ + .anc.toml [p5].not_destructive." + ), + other => panic!("expected Skip, got {other:?}"), + } + } + + #[test] + fn a_void_config_says_no_setting_applied() { + let mut project = crate::audits::behavioral::tests::test_project_with_sh_script( + r#"case "$*" in + "delete --help") printf 'Usage: tool delete \n\nOptions:\n -h, --help Show help.\n' ;; + *) printf 'Usage: tool \n\nCommands:\n delete Delete an item\n' ;; +esac"#, + ); + project.anc_config.load = crate::anc_toml::AncConfigLoad::Invalid( + "could not parse .anc.toml at .anc.toml: bad".into(), + ); + + match ForceYesAudit.run(&project).expect("audit runs").status { + AuditStatus::Fail(msg) => assert!( + msg.ends_with( + "No .anc.toml setting applied: could not parse .anc.toml at .anc.toml: bad." + ), + "{msg}" + ), + other => panic!("expected Fail, got {other:?}"), + } + } + #[test] fn pass_when_destructive_subcommand_has_force() { let subhelp = vec![( @@ -129,7 +385,7 @@ mod tests { ), )]; assert_eq!( - audit_force_yes(&["delete".to_string()], &subhelp), + audit_force_yes(&["delete".to_string()], &subhelp, &[]).status, AuditStatus::Pass ); } @@ -143,18 +399,50 @@ mod tests { ), )]; assert_eq!( - audit_force_yes(&["purge".to_string()], &subhelp), + audit_force_yes(&["purge".to_string()], &subhelp, &[]).status, AuditStatus::Pass ); } + #[test] + fn pass_on_widely_used_confirmation_flags() { + for flag in ["--auto-approve", "--assume-yes", "--confirm"] { + let subhelp = vec![( + "destroy".to_string(), + hp(&format!( + "Usage: tool destroy\n\nOptions:\n {flag} Skip the prompt.\n -h, --help Show help.\n" + )), + )]; + assert_eq!( + audit_force_yes(&["destroy".to_string()], &subhelp, &[]).status, + AuditStatus::Pass, + "{flag} confirms a destructive subcommand" + ); + } + } + + #[test] + fn fail_names_the_accepted_flags() { + let subhelp = vec![( + "delete".to_string(), + hp("Usage: tool delete \n\nOptions:\n -h, --help Show help.\n"), + )]; + match audit_force_yes(&["delete".to_string()], &subhelp, &[]).status { + AuditStatus::Fail(msg) => assert!( + msg.contains("Accepted flags: --force, --yes, -y, -f, --auto-approve, --assume-yes, --confirm."), + "{msg}" + ), + other => panic!("expected Fail, got {other:?}"), + } + } + #[test] fn fail_when_destructive_subcommand_missing_flags() { let subhelp = vec![( "delete".to_string(), hp("Usage: tool delete \n\nOptions:\n -h, --help Show help.\n"), )]; - match audit_force_yes(&["delete".to_string()], &subhelp) { + match audit_force_yes(&["delete".to_string()], &subhelp, &[]).status { AuditStatus::Fail(msg) => { assert!(msg.contains("delete")); assert!(msg.contains("--force")); @@ -169,7 +457,7 @@ mod tests { // (timeout, crash, refused --help). Treat as Fail; the operator must // surface a confirmation flag in the documented help text. let subhelp: Vec<(String, HelpOutput)> = Vec::new(); - match audit_force_yes(&["delete".to_string()], &subhelp) { + match audit_force_yes(&["delete".to_string()], &subhelp, &[]).status { AuditStatus::Fail(msg) => assert!(msg.contains("delete")), other => panic!("expected Fail, got {other:?}"), } @@ -188,6 +476,9 @@ mod tests { ), ]; let destructive = vec!["delete".to_string(), "purge".to_string()]; - assert_eq!(audit_force_yes(&destructive, &subhelp), AuditStatus::Pass); + assert_eq!( + audit_force_yes(&destructive, &subhelp, &[]).status, + AuditStatus::Pass + ); } } diff --git a/src/audits/behavioral/json_output.rs b/src/audits/behavioral/json_output.rs index 36d3b4d..77d208e 100644 --- a/src/audits/behavioral/json_output.rs +++ b/src/audits/behavioral/json_output.rs @@ -1,8 +1,17 @@ +use std::ffi::OsString; + +use crate::anc_toml::{JSON_PROBE_KEY, Sourced}; use crate::audit::Audit; use crate::audits::behavioral::subcommand_help::should_skip; use crate::project::Project; use crate::runner::{BinaryRunner, HelpOutput, RunStatus}; -use crate::types::{AuditGroup, AuditLayer, AuditResult, AuditStatus, Confidence}; +use crate::types::{ + AuditGroup, AuditLayer, AuditResult, AuditStatus, Confidence, Mitigation, Verdict, +}; + +/// The evidence when an output flag is detected but no safe probe printed +/// JSON. A `[p2] json_probe` declaration takes over from here. +const UNVERIFIABLE: &str = "--output/--format flag detected but could not validate JSON via safe probes (--help/--version override output flags in most CLIs)"; pub struct JsonOutputAudit; @@ -33,25 +42,26 @@ impl Audit for JsonOutputAudit { fn run(&self, project: &Project) -> anyhow::Result { let runner = project.runner_ref(); - let help_result = runner.run(&["--help"], &[]); - - let status = match help_result.status { - RunStatus::Ok => { - let output = format!("{}{}", help_result.stdout, help_result.stderr); - let lower = output.to_lowercase(); - let has_output_flag = lower.contains("--output"); - let has_format_flag = lower.contains("--format"); - - if has_output_flag || has_format_flag { - // Flag found in top-level help, validate directly - validate_json_output(runner, &[], has_output_flag, has_format_flag) - } else { - // Flag not in top-level help. Probe subcommands, since most CLIs - // (gh, kubectl, cargo) put --output on subcommands, not top-level. - probe_subcommands(runner, project.help_output()) - } + let declared = project + .anc_config + .config() + .and_then(|cfg| cfg.p2.json_probe.as_ref()); + let verdict = match (detect_json_output(runner, project), declared) { + (AuditStatus::Skip(reason), Some(probe)) if reason == UNVERIFIABLE => { + audit_declared_probe(runner, probe) } - _ => AuditStatus::Skip("could not run --help to detect output flags".into()), + (status @ AuditStatus::OptOut(_), Some(probe)) => status + .with_note(&format!( + "The probe declared via {} runs only for a tool whose help shows an \ + --output or --format flag.", + probe.cite(JSON_PROBE_KEY) + )) + .into(), + (status, _) => Verdict::from(status), + }; + let status = match project.anc_config.void_note() { + Some(note) => verdict.status.with_note(¬e), + None => verdict.status, }; Ok(AuditResult { @@ -61,13 +71,96 @@ impl Audit for JsonOutputAudit { layer: AuditLayer::Behavioral, status, confidence: Confidence::High, - mitigation: None, + mitigation: verdict.mitigation, config_hint: None, pass_evidence: None, }) } } +/// Find an output flag in the help and validate JSON through the safe +/// probes, without any declaration. +fn detect_json_output(runner: &BinaryRunner, project: &Project) -> AuditStatus { + let help_result = runner.run(&["--help"], &[]); + match help_result.status { + RunStatus::Ok => { + let output = format!("{}{}", help_result.stdout, help_result.stderr); + let lower = output.to_lowercase(); + let has_output_flag = lower.contains("--output"); + let has_format_flag = lower.contains("--format"); + + if has_output_flag || has_format_flag { + // Flag found in top-level help, validate directly + validate_json_output(runner, &[], has_output_flag, has_format_flag) + } else { + // Flag not in top-level help. Probe subcommands, since most CLIs + // (gh, kubectl, cargo) put --output on subcommands, not top-level. + probe_subcommands(runner, project.help_output()) + } + } + _ => AuditStatus::Skip("could not run --help to detect output flags".into()), + } +} + +/// Run the declared probe: Pass when it exits 0 with JSON on stdout, naming +/// the probe and the file that declared it; Fail otherwise, saying what it +/// did instead. +fn audit_declared_probe(runner: &BinaryRunner, probe: &Sourced>) -> Verdict { + let shown = probe_invocation(runner, &probe.value); + let cited = probe.cite(JSON_PROBE_KEY); + match run_declared_probe(runner, &probe.value) { + Ok(()) => Verdict { + status: AuditStatus::Pass, + mitigation: Some(Mitigation::Config(format!( + "`{shown}` printed JSON; probe declared via {cited}" + ))), + }, + Err(why) => AuditStatus::Fail(format!( + "`{shown}`, the probe declared via {cited}, {why}. The declared probe must exit 0 \ + and print JSON on stdout." + )) + .into(), + } +} + +/// Run `args` exactly as declared: no shell, with the runner's timeout, +/// closed stdin, and `NO_COLOR=1`. `Ok` when the call exits 0 and its stdout +/// parses as JSON; otherwise what it did instead. +pub(crate) fn run_declared_probe(runner: &BinaryRunner, args: &[String]) -> Result<(), String> { + let args: Vec<&str> = args.iter().map(String::as_str).collect(); + let result = runner.run(&args, &[]); + match result.status { + RunStatus::Ok => {} + RunStatus::Timeout => return Err("timed out".into()), + RunStatus::Crash { signal } => return Err(format!("was killed by signal {signal}")), + RunStatus::NotFound | RunStatus::PermissionDenied | RunStatus::Error(_) => { + return Err("could not be run".into()); + } + } + match result.exit_code { + Some(0) => {} + Some(code) => return Err(format!("exited {code}")), + None => return Err("exited without an exit code".into()), + } + let stdout = result.stdout.trim(); + if stdout.is_empty() || serde_json::from_str::(stdout).is_err() { + return Err("printed no JSON on stdout".into()); + } + Ok(()) +} + +/// The declared call as evidence shows it: the binary's name, then the +/// arguments, quoted where a shell would need it. +pub(crate) fn probe_invocation(runner: &BinaryRunner, args: &[String]) -> String { + let argv: Vec = runner + .binary_stem() + .into_iter() + .map(OsString::from) + .chain(args.iter().map(OsString::from)) + .collect(); + crate::argv::format_invocation(&argv) +} + /// Probe each top-level subcommand from the shared help parse for --output/--format. /// /// Most CLI frameworks (clap, cobra, argparse) list subcommands under a "Commands:" @@ -182,7 +275,10 @@ fn validate_json_output( } } - AuditStatus::Warn("--output/--format flag detected but could not validate JSON via safe probes (--help/--version override output flags in most CLIs)".into()) + // The safe probes reach only `--help` and `--version`, which most CLIs + // answer in text whatever the output flag says, so a miss is anc's + // limit, not the tool's: the row is not scored. + AuditStatus::Skip(UNVERIFIABLE.into()) } /// Run a single JSON probe and return Some(status) if valid JSON found. @@ -218,6 +314,95 @@ mod tests { use crate::audits::behavioral::tests::test_project_with_sh_script; use crate::types::AuditStatus; + /// Run the audit on `script` with `probe` declared in `.anc.toml`. + fn run_with_probe(script: &str, probe: &[&str]) -> AuditResult { + let mut project = test_project_with_sh_script(script); + let mut cfg = crate::anc_toml::AncConfig::default(); + cfg.p2.json_probe = Some(Sourced { + value: probe.iter().map(|arg| (*arg).to_string()).collect(), + file: ".anc.toml".into(), + }); + project.anc_config.load = crate::anc_toml::AncConfigLoad::Loaded(cfg); + JsonOutputAudit.run(&project).expect("audit should run") + } + + /// An `--output` flag the safe probes cannot validate, and read-only + /// calls that print JSON, print text, or print JSON and then fail. + const OUTPUT_FLAG_WITH_PROBES: &str = r#" +case "$*" in + "version -o json") echo '{"version":"1.0"}';; + "version --text") echo "version 1.0";; + "status -o json") echo '{"status":"down"}'; exit 1;; + *--help*) + printf 'Usage: test [OPTIONS]\n\nOptions:\n --output Output format: text or json\n';; + *) + echo "this is not json";; +esac +"#; + + #[test] + fn a_declared_probe_validates_what_the_safe_probes_cannot() { + let without = JsonOutputAudit + .run(&test_project_with_sh_script(OUTPUT_FLAG_WITH_PROBES)) + .expect("audit should run"); + assert!( + matches!(without.status, AuditStatus::Skip(_)), + "{:?}", + without.status + ); + + let with = run_with_probe(OUTPUT_FLAG_WITH_PROBES, &["version", "-o", "json"]); + + assert_eq!(with.status, AuditStatus::Pass); + assert_eq!( + with.mitigation, + Some(Mitigation::Config( + "`test version -o json` printed JSON; probe declared via .anc.toml [p2].json_probe" + .into() + )) + ); + } + + #[test] + fn a_declared_probe_that_prints_text_fails_and_says_so() { + match run_with_probe(OUTPUT_FLAG_WITH_PROBES, &["version", "--text"]).status { + AuditStatus::Fail(msg) => assert_eq!( + msg, + "`test version --text`, the probe declared via .anc.toml [p2].json_probe, printed \ + no JSON on stdout. The declared probe must exit 0 and print JSON on stdout." + ), + other => panic!("expected Fail, got {other:?}"), + } + } + + #[test] + fn a_declared_probe_that_exits_nonzero_fails_even_with_json_on_stdout() { + match run_with_probe(OUTPUT_FLAG_WITH_PROBES, &["status", "-o", "json"]).status { + AuditStatus::Fail(msg) => assert!(msg.contains(", exited 1."), "{msg}"), + other => panic!("expected Fail, got {other:?}"), + } + } + + #[test] + fn a_declared_probe_does_not_stand_in_for_a_missing_output_flag() { + let script = r#" +case "$*" in + "version -o json") echo '{"version":"1.0"}';; + *) echo 'just some help text';; +esac +"#; + match run_with_probe(script, &["version", "-o", "json"]).status { + AuditStatus::OptOut(msg) => assert!( + msg.ends_with( + "The probe declared via .anc.toml [p2].json_probe runs only for a tool whose \ + help shows an --output or --format flag." + ), + "{msg}" + ), + other => panic!("expected OptOut, got {other:?}"), + } + } + #[test] fn json_output_pass_with_valid_json() { let script = r#" @@ -254,25 +439,56 @@ esac assert_eq!(result.status, AuditStatus::Pass, "got {:?}", result.status); } - #[test] - fn json_output_fail_with_invalid_json() { - let script = r#" + /// Advertises `--output` but answers every safe probe in text. + const UNVERIFIABLE_OUTPUT_FLAG: &str = r#" case "$*" in *--help*) - echo "Usage: test [--output FORMAT]";; + printf 'Usage: test [OPTIONS]\n\nOptions:\n --output Output format: text or json\n';; *--output*) echo "this is not json";; *) echo "hello";; esac "#; - let project = test_project_with_sh_script(script); + + #[test] + fn json_output_skips_when_no_safe_probe_can_validate() { + let project = test_project_with_sh_script(UNVERIFIABLE_OUTPUT_FLAG); let result = JsonOutputAudit.run(&project).expect("audit should run"); match &result.status { - AuditStatus::Warn(msg) => { + AuditStatus::Skip(msg) => { assert!(msg.contains("could not validate JSON"), "got: {msg}") } - other => panic!("expected Warn, got {other:?}"), + other => panic!("expected Skip, got {other:?}"), + } + } + + #[test] + fn an_unverifiable_output_flag_leaves_the_schema_rows_unmeasured() { + use crate::audit::Audit; + use crate::audits::behavioral::schema_print::SchemaPrintAudit; + + let project = test_project_with_sh_script(UNVERIFIABLE_OUTPUT_FLAG); + let catalog: Vec> = + vec![Box::new(JsonOutputAudit), Box::new(SchemaPrintAudit)]; + let raw: Vec = catalog + .iter() + .map(|audit| audit.run(&project).expect("audit should run")) + .collect(); + + let rows = crate::scorecard::build_row_results(&raw, &catalog); + let schema_print = rows + .iter() + .find(|(row, _)| row.id == "p2-must-schema-print") + .map(|(row, _)| &row.status) + .expect("schema-print row"); + + match schema_print { + AuditStatus::Skip(msg) => assert!( + msg.starts_with("antecedent `p2-json-output` could not be measured:"), + "got: {msg}" + ), + other => panic!("expected Skip, got {other:?}"), } } diff --git a/src/audits/behavioral/read_write_distinction.rs b/src/audits/behavioral/read_write_distinction.rs index b931335..65e7a77 100644 --- a/src/audits/behavioral/read_write_distinction.rs +++ b/src/audits/behavioral/read_write_distinction.rs @@ -15,6 +15,9 @@ //! in which case the distinction is unobservable but not necessarily //! missing. //! - Skip when neither side appears (no recognizable verb subcommands). +//! +//! `.anc.toml [p5] not_destructive` does not apply here: a subcommand a tool +//! declares not destructive still changes state, so it stays a write. use crate::audit::Audit; use crate::audits::behavioral::destructive_ops::{is_read_verb, is_write_verb}; diff --git a/src/audits/behavioral/schema_print.rs b/src/audits/behavioral/schema_print.rs index a687658..ce3323d 100644 --- a/src/audits/behavioral/schema_print.rs +++ b/src/audits/behavioral/schema_print.rs @@ -7,15 +7,21 @@ //! //! Applicability: gates on a help-text probe — only fires when the help //! mentions any structured-output indicator (`--output`, `--format`, `--json`, -//! `--jsonl`, or the words "json"/"jsonl"). When the probe finds no such -//! indicator the audit Skips with evidence; when it does, the audit looks for -//! either a `schema` subcommand or `--schema` flag. +//! `--jsonl`, or the words "json"/"jsonl"), or when the `.anc.toml` chain's +//! `[p2] json_probe` prints JSON. When neither shows structured output the +//! audit Skips with evidence; otherwise it looks for either a `schema` +//! subcommand or `--schema` flag, then for the subcommand the chain names +//! in `[p2] schema_command`. +use crate::anc_toml::{JSON_PROBE_KEY, SCHEMA_COMMAND_KEY, Sourced}; use crate::audit::Audit; -use crate::audits::behavioral::subcommand_help::probe_subcommands; +use crate::audits::behavioral::json_output::{probe_invocation, run_declared_probe}; +use crate::audits::behavioral::subcommand_help::{probe_help, probe_subcommands}; use crate::project::Project; -use crate::runner::HelpOutput; -use crate::types::{AuditGroup, AuditLayer, AuditResult, AuditStatus, Confidence}; +use crate::runner::{BinaryRunner, HelpOutput}; +use crate::types::{ + AuditGroup, AuditLayer, AuditResult, AuditStatus, Confidence, Mitigation, Verdict, +}; const STRUCTURED_OUTPUT_FLAG_NAMES: &[&str] = &["--output", "--format", "--json", "--jsonl", "--ndjson"]; @@ -50,24 +56,43 @@ impl Audit for SchemaPrintAudit { } fn run(&self, project: &Project) -> anyhow::Result { - let status = match project.help_output() { - None => AuditStatus::Skip("could not probe --help".into()), + let verdict = match project.help_output() { + None => AuditStatus::Skip("could not probe --help".into()).into(), Some(help) => { + let runner = project.runner_ref(); + let p2 = project.anc_config.config().map(|cfg| &cfg.p2); + let shown_by_probe = p2 + .and_then(|p2| p2.json_probe.as_ref()) + .filter(|_| !has_structured_output_indicator(help)) + .filter(|probe| run_declared_probe(runner, &probe.value).is_ok()); // First try the top-level help only. If that's inconclusive, // walk one level into each top-level subcommand to find // `schema` exposed as a nested subcommand (e.g., // `anc emit schema`). One-level walk matches how an agent // would discover the surface via `--help` chaining. - match audit_schema_print(help) { + let status = match audit_schema_print(help, shown_by_probe.is_some()) { AuditStatus::Fail(_) => { - let runner = project.runner_ref(); let subhelp = probe_subcommands(runner, help); - audit_schema_print_with_subhelp(help, &subhelp) + audit_schema_print_with_subhelp(help, shown_by_probe.is_some(), &subhelp) } other => other, + }; + let verdict = match (status, p2.and_then(|p2| p2.schema_command.as_ref())) { + (status @ AuditStatus::Fail(_), Some(path)) => { + audit_declared_schema_command(status, runner, help, path) + } + (status, _) => Verdict::from(status), + }; + match shown_by_probe { + Some(probe) => credit_the_probe(verdict, runner, probe), + None => verdict, } } }; + let status = match project.anc_config.void_note() { + Some(note) => verdict.status.with_note(¬e), + None => verdict.status, + }; Ok(AuditResult { id: self.id().to_string(), @@ -76,28 +101,101 @@ impl Audit for SchemaPrintAudit { layer: self.layer(), status, confidence: Confidence::Medium, - mitigation: None, + mitigation: verdict.mitigation, config_hint: None, pass_evidence: None, }) } } -/// Core unit for tests. Returns Skip when no structured-output indicator is -/// present (vacuous applicability), Pass when a schema surface is advertised, -/// Fail when structured output is advertised without a schema surface. -pub(crate) fn audit_schema_print(help: &HelpOutput) -> AuditStatus { - let raw = help.raw(); - let raw_lower = raw.to_lowercase(); +/// Pass when the help lists the declared schema command, naming it and the +/// file that declared it; otherwise keep the built-in `fail` and say the +/// declared command was not found. +fn audit_declared_schema_command( + fail: AuditStatus, + runner: &BinaryRunner, + help: &HelpOutput, + path: &Sourced>, +) -> Verdict { + let shown = probe_invocation(runner, &path.value); + let cited = path.cite(SCHEMA_COMMAND_KEY); + if lists_command_path(runner, help, &path.value) { + return Verdict { + status: AuditStatus::Pass, + mitigation: Some(Mitigation::Config(format!( + "`{shown}` is the schema command declared via {cited}" + ))), + }; + } + fail.with_note(&format!( + "The schema command `{shown}`, declared via {cited}, is not listed in --help." + )) + .into() +} + +/// Whether each token of `path` is listed in its parent's `--help`: the +/// first in `top_help`, each later one in ` --help`. +fn lists_command_path(runner: &BinaryRunner, top_help: &HelpOutput, path: &[String]) -> bool { + let Some(first) = path.first() else { + return false; + }; + lists_subcommand(top_help, first) + && (1..path.len()).all(|depth| { + let parent: Vec<&str> = path[..depth].iter().map(String::as_str).collect(); + probe_help(runner, &parent).is_some_and(|help| lists_subcommand(&help, &path[depth])) + }) +} + +/// Whether `help` lists `name` as a subcommand: a parsed name, or an +/// indented line whose text before the description gap is `name`, for a +/// command block whose heading the parser does not read (kubectl's `Basic +/// Commands (Beginner):`). +fn lists_subcommand(help: &HelpOutput, name: &str) -> bool { + help.subcommands() + .iter() + .any(|parsed| parsed.to_lowercase() == name) + || help.raw().lines().any(|line| { + line.starts_with(char::is_whitespace) && line.trim().split(" ").next() == Some(name) + }) +} + +/// Name the declared probe that showed structured output when the help did +/// not: in a Pass's evidence beside any other setting it needed, or at the +/// end of any other status. +fn credit_the_probe( + verdict: Verdict, + runner: &BinaryRunner, + probe: &Sourced>, +) -> Verdict { + let shown = format!( + "structured output shown by `{}`, the probe declared via {}", + probe_invocation(runner, &probe.value), + probe.cite(JSON_PROBE_KEY) + ); + let note = format!("The CLI has {shown}."); + verdict.crediting(shown, ¬e) +} + +/// Whether the help names a structured-output flag or format. +fn has_structured_output_indicator(help: &HelpOutput) -> bool { + let raw_lower = help.raw().to_lowercase(); let has_structured_flag = help .flags() .iter() .any(|f| STRUCTURED_OUTPUT_FLAG_NAMES.iter().any(|n| f.matches(n))); - let has_structured_token = STRUCTURED_OUTPUT_TOKENS - .iter() - .any(|t| raw_lower.contains(&t.to_lowercase())); + has_structured_flag + || STRUCTURED_OUTPUT_TOKENS + .iter() + .any(|t| raw_lower.contains(&t.to_lowercase())) +} - if !has_structured_flag && !has_structured_token { +/// Core unit for tests. Returns Skip when neither the help nor a declared +/// probe (`json_shown`) shows structured output (vacuous applicability), +/// Pass when a schema surface is advertised, Fail when structured output is +/// shown without a schema surface. +pub(crate) fn audit_schema_print(help: &HelpOutput, json_shown: bool) -> AuditStatus { + let raw = help.raw(); + if !json_shown && !has_structured_output_indicator(help) { return AuditStatus::Skip( "no structured-output indicator (--output / --format / json / jsonl) in --help".into(), ); @@ -136,11 +234,12 @@ pub(crate) fn audit_schema_print(help: &HelpOutput) -> AuditStatus { /// bound for an agent that does not have prior knowledge of the CLI. pub(crate) fn audit_schema_print_with_subhelp( top_help: &HelpOutput, + json_shown: bool, subhelp: &[(String, HelpOutput)], ) -> AuditStatus { // Re-run the top-level audit first so the applicability gate and // top-level positives short-circuit before we inspect nested help. - match audit_schema_print(top_help) { + match audit_schema_print(top_help, json_shown) { AuditStatus::Fail(_) => {} other => return other, } @@ -213,22 +312,148 @@ Options: -h, --help Show help "#; + /// Help with no structured-output indicator, an `--output` flag on + /// `get`, and a read-only call that prints JSON. + const JSON_ONLY_ON_A_SUBCOMMAND: &str = r#"case "$*" in + "version -o json") echo '{"version":"1.0"}' ;; + "get --help") printf 'Usage: test get\n\nOptions:\n -o, --output One of: text, yaml\n' ;; + *) printf 'Usage: test \n\nCommands:\n get Show a resource\n version Print the version\n' ;; +esac"#; + + fn sourced(args: &[&str]) -> Sourced> { + Sourced { + value: args.iter().map(|arg| (*arg).to_string()).collect(), + file: ".anc.toml".into(), + } + } + + /// Run the audit on `script` with the given `[p2]` declarations. + fn run_declared( + script: &str, + json_probe: Option<&[&str]>, + schema: Option<&[&str]>, + ) -> AuditResult { + let mut project = crate::audits::behavioral::tests::test_project_with_sh_script(script); + let mut cfg = crate::anc_toml::AncConfig::default(); + cfg.p2.json_probe = json_probe.map(sourced); + cfg.p2.schema_command = schema.map(sourced); + project.anc_config.load = crate::anc_toml::AncConfigLoad::Loaded(cfg); + SchemaPrintAudit.run(&project).expect("audit runs") + } + + /// Structured output in the help, no `schema` surface, and an `explain` + /// command plus an `emit shape` command. + const EXPLAIN_CLI: &str = r#"case "$*" in + "emit --help") printf 'Usage: test emit \n\nCommands:\n shape Print the output shape\n' ;; + *--help*) printf 'Usage: test \n\nCommands:\n get Show a resource\n explain Describe a resource type\n emit Emit artifacts\n\nOptions:\n --output text or json\n' ;; +esac"#; + + #[test] + fn a_declared_schema_command_the_help_lists_passes_and_names_its_file() { + assert!(matches!( + run_declared(EXPLAIN_CLI, None, None).status, + AuditStatus::Fail(_) + )); + + let result = run_declared(EXPLAIN_CLI, None, Some(&["explain"])); + + assert_eq!(result.status, AuditStatus::Pass); + assert_eq!( + result.mitigation, + Some(Mitigation::Config( + "`test explain` is the schema command declared via .anc.toml [p2].schema_command" + .into() + )) + ); + } + + #[test] + fn a_nested_declared_schema_command_is_found_in_its_parent_help() { + let result = run_declared(EXPLAIN_CLI, None, Some(&["emit", "shape"])); + + assert_eq!(result.status, AuditStatus::Pass); + } + + #[test] + fn a_declared_schema_command_the_help_does_not_list_keeps_the_fail_and_says_so() { + match run_declared(EXPLAIN_CLI, None, Some(&["emit", "schema"])).status { + AuditStatus::Fail(msg) => assert!( + msg.ends_with( + "The schema command `test emit schema`, declared via .anc.toml \ + [p2].schema_command, is not listed in --help." + ), + "{msg}" + ), + other => panic!("expected Fail, got {other:?}"), + } + } + + #[test] + fn a_kubectl_shaped_cli_passes_on_its_declared_probe_and_schema_command() { + let script = r#"case "$*" in + "version --client -o json") echo '{"clientVersion":{}}' ;; + *--help*) printf 'tool controls things.\n\nBasic Commands (Intermediate):\n explain Get documentation for a resource\n get Display one or many resources\n' ;; +esac"#; + + let result = run_declared( + script, + Some(&["version", "--client", "-o", "json"]), + Some(&["explain"]), + ); + + assert_eq!(result.status, AuditStatus::Pass); + assert_eq!( + result.mitigation, + Some(Mitigation::Config( + "`test explain` is the schema command declared via .anc.toml [p2].schema_command; \ + structured output shown by `test version --client -o json`, the probe declared \ + via .anc.toml [p2].json_probe" + .into() + )) + ); + } + + #[test] + fn a_declared_probe_that_prints_json_shows_structured_output() { + assert!(matches!( + run_declared(JSON_ONLY_ON_A_SUBCOMMAND, None, None).status, + AuditStatus::Skip(_) + )); + + match run_declared( + JSON_ONLY_ON_A_SUBCOMMAND, + Some(&["version", "-o", "json"]), + None, + ) + .status + { + AuditStatus::Fail(msg) => assert!( + msg.ends_with( + "The CLI has structured output shown by `test version -o json`, the probe \ + declared via .anc.toml [p2].json_probe." + ), + "{msg}" + ), + other => panic!("expected Fail, got {other:?}"), + } + } + #[test] fn happy_path_schema_subcommand() { let help = HelpOutput::from_raw(HELP_WITH_SCHEMA_SUBCMD); - assert_eq!(audit_schema_print(&help), AuditStatus::Pass); + assert_eq!(audit_schema_print(&help, false), AuditStatus::Pass); } #[test] fn happy_path_schema_flag() { let help = HelpOutput::from_raw(HELP_WITH_SCHEMA_FLAG); - assert_eq!(audit_schema_print(&help), AuditStatus::Pass); + assert_eq!(audit_schema_print(&help, false), AuditStatus::Pass); } #[test] fn skip_no_structured_output_indicator() { let help = HelpOutput::from_raw(HELP_NO_STRUCTURED_OUTPUT); - match audit_schema_print(&help) { + match audit_schema_print(&help, false) { AuditStatus::Skip(msg) => assert!(msg.contains("structured-output")), other => panic!("expected Skip, got {other:?}"), } @@ -237,7 +462,7 @@ Options: #[test] fn fail_structured_output_no_schema() { let help = HelpOutput::from_raw(HELP_STRUCTURED_NO_SCHEMA); - match audit_schema_print(&help) { + match audit_schema_print(&help, false) { AuditStatus::Fail(msg) => assert!(msg.contains("schema")), other => panic!("expected Fail, got {other:?}"), } diff --git a/src/audits/behavioral/standard_names.rs b/src/audits/behavioral/standard_names.rs index ae07d65..e231d79 100644 --- a/src/audits/behavioral/standard_names.rs +++ b/src/audits/behavioral/standard_names.rs @@ -24,7 +24,7 @@ use crate::project::Project; use crate::runner::HelpOutput; use crate::types::{ AuditGroup, AuditLayer, AuditResult, AuditStatus, Confidence, ConfigFile, ConfigHint, - MitigationInfo, + Mitigation, MitigationInfo, }; /// Cap on the number of domain-verb matches listed in the Pass evidence @@ -205,7 +205,7 @@ impl Audit for StandardNamesAudit { layer: self.layer(), status: result.status, confidence: Confidence::Low, - mitigation: result.mitigation, + mitigation: result.mitigation.map(Mitigation::DomainVerbs), config_hint: result.config_hint, pass_evidence: None, }) diff --git a/src/audits/behavioral/subcommand_help.rs b/src/audits/behavioral/subcommand_help.rs index f716d59..8bc5b35 100644 --- a/src/audits/behavioral/subcommand_help.rs +++ b/src/audits/behavioral/subcommand_help.rs @@ -37,32 +37,34 @@ pub(crate) fn probe_subcommands( probe_named(runner, &names) } -/// Probe ` --help` for each of `names`, under the same skip and -/// drop rules as [`probe_subcommands`]. +/// Probe ` --help` through [`probe_help`] for each of `names` +/// that [`should_skip`] leaves in, keeping the names whose probe returned +/// help. pub(crate) fn probe_named(runner: &BinaryRunner, names: &[&str]) -> Vec<(String, HelpOutput)> { - let mut out = Vec::new(); - for &name in names { - if should_skip(name) { - continue; - } - let result = runner.run(&[name, "--help"], &[]); - // Capture partial output from timeouts/crashes the same way HelpOutput::probe does. - // Only NotFound / PermissionDenied / Error are dropped here — those mean we - // couldn't even spawn the child, not that the subcommand misbehaved. - match result.status { - RunStatus::Ok | RunStatus::Timeout | RunStatus::Crash { .. } => { - let mut raw = String::with_capacity(result.stdout.len() + result.stderr.len()); - raw.push_str(&result.stdout); - raw.push_str(&result.stderr); - if raw.trim().is_empty() { - continue; - } - out.push((name.to_string(), HelpOutput::from_raw(raw))); - } - _ => continue, + names + .iter() + .filter(|name| !should_skip(name)) + .filter_map(|&name| Some((name.to_string(), probe_help(runner, &[name])?))) + .collect() +} + +/// Probe ` --help`. `None` when the child could not be +/// spawned or printed nothing. +pub(crate) fn probe_help(runner: &BinaryRunner, path: &[&str]) -> Option { + let args: Vec<&str> = path.iter().copied().chain(["--help"]).collect(); + let result = runner.run(&args, &[]); + // Capture partial output from timeouts/crashes the same way HelpOutput::probe does. + // Only NotFound / PermissionDenied / Error are dropped here — those mean we + // couldn't even spawn the child, not that the subcommand misbehaved. + match result.status { + RunStatus::Ok | RunStatus::Timeout | RunStatus::Crash { .. } => { + let mut raw = String::with_capacity(result.stdout.len() + result.stderr.len()); + raw.push_str(&result.stdout); + raw.push_str(&result.stderr); + (!raw.trim().is_empty()).then(|| HelpOutput::from_raw(raw)) } + _ => None, } - out } /// Whether `name` is a built-in the subcommand probes leave alone. diff --git a/src/runner/help_probe/mod.rs b/src/runner/help_probe/mod.rs index b2ba906..c49f29f 100644 --- a/src/runner/help_probe/mod.rs +++ b/src/runner/help_probe/mod.rs @@ -216,6 +216,21 @@ impl HelpOutput { self.flags.get_or_init(|| parse_flags(&self.raw)) } + /// Whether the help lists `name` as a flag. A `--long` or `-s` name + /// matches a parsed flag. A single-dash name longer than one letter, the + /// Go `flag` package's `-auto-approve`, matches a flag line that names + /// it, because the flag parser reads such a line as the short flag `-a`. + pub fn advertises_flag(&self, name: &str) -> bool { + let single_dash_word = name.len() > 2 && name.starts_with('-') && !name.starts_with("--"); + if single_dash_word { + return self + .raw + .lines() + .any(|line| flag_line_names(line).any(|n| n == name)); + } + self.flags().iter().any(|flag| flag.matches(name)) + } + /// `[env: FOO]` hints parsed out of the help surface. Lazy + cached. pub fn env_hints(&self) -> &[EnvHint] { self.env_hints.get_or_init(|| parse_env_hints(&self.raw)) @@ -312,6 +327,25 @@ fn parse_flags(raw: &str) -> Vec { flags } +/// The flag names a flag line declares, as written (`-auto-approve` from +/// `-auto-approve Skip approval`, `-lock` from `-lock=false`). Empty for a +/// line that is not a flag line. +fn flag_line_names(line: &str) -> impl Iterator { + let trimmed = line.trim_start(); + let is_flag_line = line.starts_with(char::is_whitespace) + && trimmed.starts_with('-') + && !trimmed.starts_with("---"); + let header = if is_flag_line { + before_description_gap(trimmed) + } else { + "" + }; + header + .split(',') + .filter_map(|piece| piece.split_whitespace().next()) + .map(|token| token.split(['=', '[']).next().unwrap_or(token)) +} + /// The text before clap's two-space description gap: the flag header of a /// flag line, the invocation of a command entry. The whole line when there /// is no description on it. @@ -664,6 +698,17 @@ Options: assert_eq!(regexp.short.as_deref(), Some("-e")); } + #[test] + fn advertises_a_single_dash_long_flag_by_its_whole_name() { + let help = HelpOutput::from_raw( + "Usage: terraform [global options] apply [options]\n\nOptions:\n\n -auto-approve Skip interactive approval of plan before applying.\n\n -lock=false Don't hold a state lock during the operation.\n", + ); + assert!(help.advertises_flag("-auto-approve")); + assert!(help.advertises_flag("-lock")); + assert!(!help.advertises_flag("-auto")); + assert!(!help.advertises_flag("-approve")); + } + #[test] fn parse_flags_ignores_prose_dashes() { // A line starting with '---' (separator) must not become a flag. diff --git a/src/scorecard/mod.rs b/src/scorecard/mod.rs index 115e6be..d8f8e33 100644 --- a/src/scorecard/mod.rs +++ b/src/scorecard/mod.rs @@ -7,7 +7,9 @@ use serde::Serialize; use crate::audit::Audit; use crate::principles::registry::{Level, REQUIREMENTS, SPEC_VERSION}; -use crate::types::{AuditGroup, AuditLayer, AuditResult, AuditStatus, ConfigHint, ConfigScope}; +use crate::types::{ + AuditGroup, AuditLayer, AuditResult, AuditStatus, ConfigHint, ConfigScope, Mitigation, +}; /// Current scorecard JSON schema version. Consumers (site rendering, /// leaderboard pipeline) pin against this to detect shape changes. @@ -418,6 +420,26 @@ pub struct AuditResultView { pub config_hint: Option, } +// A Pass row's evidence reads what the audit observed before the +// `.anc.toml` setting the Pass depended on, so a reader sees what the tool +// showed and then what it declared. `None` keeps an unassisted Pass that +// names nothing at `evidence: null`. +fn pass_row_evidence(r: &AuditResult) -> Option { + let declared = r.mitigation.as_ref().map(|m| match m { + Mitigation::DomainVerbs(info) => { + crate::audits::behavioral::standard_names::format_pass_evidence(info) + } + Mitigation::Config(prose) => prose.clone(), + }); + match (r.pass_evidence.as_deref(), declared) { + (Some(observed), Some(declared)) => { + let observed = observed.strip_suffix('.').unwrap_or(observed); + Some(format!("{observed}; {declared}")) + } + (observed, declared) => observed.map(str::to_string).or(declared), + } +} + impl AuditResultView { /// Construct from a raw probe result (pre-fan-out callers and test /// fixtures). `audit_id` defaults to `r.id` and `tier` is looked up @@ -436,19 +458,7 @@ impl AuditResultView { /// requirement row id. pub fn from_row(r: &AuditResult, audit_id: &str) -> Self { let (status, evidence) = match &r.status { - AuditStatus::Pass => { - // When a Pass was assisted by `domain_verbs`, surface the - // formatted ratio + matched names in the row's `evidence` - // field so text-mode rendering and JSON-mode dispatch see - // the same prose. A Pass that names nothing it matched - // keeps `evidence: null`. - let pass_evidence = r.pass_evidence.clone().or_else(|| { - r.mitigation - .as_ref() - .map(crate::audits::behavioral::standard_names::format_pass_evidence) - }); - ("pass".to_string(), pass_evidence) - } + AuditStatus::Pass => ("pass".to_string(), pass_row_evidence(r)), AuditStatus::Warn(e) => ("warn".to_string(), Some(e.clone())), AuditStatus::Fail(e) => ("fail".to_string(), Some(e.clone())), AuditStatus::OptOut(e) => ("opt_out".to_string(), Some(e.clone())), @@ -457,8 +467,10 @@ impl AuditResultView { AuditStatus::Error(e) => ("error".to_string(), Some(e.clone())), }; let (using_domain_verbs, domain_match_count) = match &r.mitigation { - Some(m) => (Some(m.using_domain_verbs), Some(m.domain_match_count)), - None => (None, None), + Some(Mitigation::DomainVerbs(m)) => { + (Some(m.using_domain_verbs), Some(m.domain_match_count)) + } + _ => (None, None), }; // Serialize AuditGroup / AuditLayer / Confidence via serde_json so // the JSON mirrors the canonical enum spelling (snake_case). @@ -1296,6 +1308,22 @@ mod tests { assert_eq!(view.evidence.as_deref(), Some("list (--limit)")); } + #[test] + fn pass_row_evidence_names_what_matched_then_the_setting() { + let mut r = make_result("pass-id", AuditStatus::Pass, AuditGroup::P5); + r.pass_evidence = Some("delete (--force).".into()); + r.mitigation = Some(Mitigation::Config( + "destroy accepts -auto-approve via .anc.toml [p5].confirm_flags".into(), + )); + let view = AuditResultView::from_result(&r); + assert_eq!( + view.evidence.as_deref(), + Some( + "delete (--force); destroy accepts -auto-approve via .anc.toml [p5].confirm_flags" + ) + ); + } + #[test] fn format_json_emits_audience_when_all_signals_present() { use crate::scorecard::audience::{SIGNAL_AUDIT_IDS, classify}; diff --git a/src/types.rs b/src/types.rs index e74607e..e7967d1 100644 --- a/src/types.rs +++ b/src/types.rs @@ -31,6 +31,63 @@ pub enum AuditStatus { Error(String), } +impl AuditStatus { + /// Append `note` to the evidence. `Pass` carries no evidence and is + /// returned unchanged. + pub fn with_note(self, note: &str) -> Self { + let join = |evidence: String| format!("{evidence} {note}"); + match self { + AuditStatus::Pass => AuditStatus::Pass, + AuditStatus::Warn(e) => AuditStatus::Warn(join(e)), + AuditStatus::Fail(e) => AuditStatus::Fail(join(e)), + AuditStatus::OptOut(e) => AuditStatus::OptOut(join(e)), + AuditStatus::NotApplicable(e) => AuditStatus::NotApplicable(join(e)), + AuditStatus::Skip(e) => AuditStatus::Skip(join(e)), + AuditStatus::Error(e) => AuditStatus::Error(join(e)), + } + } +} + +/// An audit's status and the opt-in a Pass depended on, if any. +#[derive(Debug, PartialEq)] +pub struct Verdict { + pub status: AuditStatus, + pub mitigation: Option, +} + +impl Verdict { + /// Record a declaration that shaped this verdict: `prose` in a Pass's + /// evidence, after any prose already there, or `note` at the end of any + /// other status. + pub fn crediting(self, prose: String, note: &str) -> Verdict { + match self.status { + AuditStatus::Pass => { + let evidence = match self.mitigation { + Some(Mitigation::Config(earlier)) => format!("{earlier}; {prose}"), + _ => prose, + }; + Verdict { + status: AuditStatus::Pass, + mitigation: Some(Mitigation::Config(evidence)), + } + } + status => Verdict { + status: status.with_note(note), + mitigation: self.mitigation, + }, + } + } +} + +impl From for Verdict { + fn from(status: AuditStatus) -> Self { + Verdict { + status, + mitigation: None, + } + } +} + /// How confident an audit is in its verdict. Direct probes (flag parsers, /// exit-code observation) report `High`; heuristic text inference reports /// `Medium`; soft cross-signal inference reports `Low`. Consumers use this @@ -93,21 +150,20 @@ pub struct AuditResult { #[serde(default)] pub confidence: Confidence, /// Per-audit transparency carrier: when an audit's Pass depended on a - /// per-CLI mitigation (today: `.anc.toml [p6] domain_verbs` for - /// `p6-standard-names`), the audit populates this so the scorecard - /// distinguishes a self-declared Pass from an unassisted one. `None` for - /// every audit that has no mitigation to declare. Carrier-shaped rather - /// than audit-specific so future audits with similar transparency needs - /// reuse the slot instead of growing parallel fields. + /// per-CLI `.anc.toml` setting, the audit populates this so the + /// scorecard distinguishes a self-declared Pass from an unassisted one. + /// Surfaces in the row's `evidence` after `pass_evidence`. `None` for + /// every audit that has no mitigation to declare. #[serde(default, skip_serializing_if = "Option::is_none")] - pub mitigation: Option, + pub mitigation: Option, /// The `.anc.toml` setting that would clear this row's warning, attached /// when no config supplied it. `None` for every other row. #[serde(default, skip_serializing_if = "Option::is_none")] pub config_hint: Option, /// What a Pass matched, for an audit whose Pass names the subcommands or - /// flags it found. Surfaces as the row's `evidence`. `None` for every - /// other row; a non-Pass status carries its evidence in the status. + /// flags it found. Surfaces as the row's `evidence`, ahead of any + /// `mitigation` prose. `None` for every other row; a non-Pass status + /// carries its evidence in the status. #[serde(default, skip_serializing_if = "Option::is_none")] pub pass_evidence: Option, } @@ -155,17 +211,25 @@ pub enum ConfigScope { User, } -/// Transparency metadata attached to an `AuditResult` when its verdict -/// depended on a documented opt-in (config-driven recognition, suppression -/// profile, etc.). Distinct from `evidence`, which is prose; `MitigationInfo` -/// is the structured signal a downstream consumer (scorecard renderer, -/// leaderboard) can dispatch on without parsing the evidence string. -/// -/// Current uses: -/// - `p6-standard-names`: when one or more subcommands matched the audit -/// target's `.anc.toml [p6] domain_verbs` list (not the built-in -/// `STANDARD_VERBS`), the audit fills `domain_match_count` and -/// `domain_match_examples`. +/// The `.anc.toml` setting a Pass depended on. The scorecard renders either +/// variant as the Pass row's `evidence`. +#[derive(Debug, Clone, Serialize, PartialEq)] +pub enum Mitigation { + /// `p6-standard-names` recognized subcommands through `[p6] + /// domain_verbs`; the structured counts also surface as row fields. + DomainVerbs(MitigationInfo), + /// Any other setting: the prose names the setting, the values it + /// contributed, and the file that supplied them. + Config(String), +} + +/// Transparency metadata attached to an `AuditResult` when +/// `p6-standard-names` passed because one or more subcommands matched the +/// audit target's `.anc.toml [p6] domain_verbs` list (not the built-in +/// `STANDARD_VERBS`). Distinct from `evidence`, which is prose; +/// `MitigationInfo` is the structured signal a downstream consumer +/// (scorecard renderer, leaderboard) can dispatch on without parsing the +/// evidence string. #[derive(Debug, Clone, Serialize, PartialEq)] pub struct MitigationInfo { /// True iff the verdict was assisted by the named opt-in. Always present diff --git a/tests/anc_toml_settings_integration.rs b/tests/anc_toml_settings_integration.rs new file mode 100644 index 0000000..55368fd --- /dev/null +++ b/tests/anc_toml_settings_integration.rs @@ -0,0 +1,224 @@ +//! End-to-end coverage for the `.anc.toml` settings audits read beyond +//! `[p6] domain_verbs`: each test runs the real `anc` binary against a shell +//! fixture and reads one scorecard row, so the loader, the audit, and the +//! row's evidence are checked together. + +use std::fs; +use std::path::{Path, PathBuf}; + +use assert_cmd::Command; +use serde_json::Value; + +/// A fixture CLI in its own directory, with a separate directory for the +/// repository `.anc.toml` (`--repo`) and one for the user-level file. +struct Fixture { + _tmp: tempfile::TempDir, + bin: PathBuf, + repo: PathBuf, + home_file: PathBuf, +} + +impl Fixture { + /// `script` is the body of a `case "$*" in ... esac` dispatch on the + /// fixture's arguments. + fn new(script: &str) -> Self { + let tmp = tempfile::tempdir().expect("tempdir"); + let base = tmp.path().to_path_buf(); + let bin = base.join("bin").join("tool"); + let repo = base.join("repo"); + let home = base.join("home"); + for dir in [bin.parent().expect("bin dir"), &repo, &home] { + fs::create_dir_all(dir).expect("create fixture dir"); + } + write_executable( + &bin, + &format!("#!/bin/sh\ncase \"$*\" in\n{script}\nesac\n"), + ); + let home_file = home.join(".anc.toml"); + fs::write(&home_file, "").expect("write empty home config"); + Self { + _tmp: tmp, + bin, + repo, + home_file, + } + } + + fn repo_config(&self, body: &str) -> &Self { + fs::write(self.repo.join(".anc.toml"), body).expect("write repo .anc.toml"); + self + } + + fn home_config(&self, body: &str) -> &Self { + fs::write(&self.home_file, body).expect("write home .anc.toml"); + self + } + + /// The `id` row of `anc audit --repo --output json`. + fn row(&self, id: &str) -> Value { + let output = Command::cargo_bin("anc") + .expect("anc binary") + .env("AGENTNATIVE_HOME_CONFIG", &self.home_file) + .args(["audit", path_str(&self.bin), "--repo", path_str(&self.repo)]) + .args(["--output", "json"]) + .output() + .expect("spawn anc"); + let stdout = String::from_utf8(output.stdout).expect("utf8 stdout"); + let scorecard: Value = serde_json::from_str(&stdout) + .unwrap_or_else(|e| panic!("scorecard is JSON ({e}): {stdout}")); + scorecard["results"] + .as_array() + .expect("results array") + .iter() + .find(|row| row["id"] == id) + .unwrap_or_else(|| panic!("no {id} row: {stdout}")) + .clone() + } +} + +fn write_executable(path: &Path, body: &str) { + #[cfg(unix)] + { + use std::io::Write; + use std::os::unix::fs::OpenOptionsExt; + let mut file = fs::OpenOptions::new() + .write(true) + .create(true) + .truncate(true) + .mode(0o755) + .open(path) + .expect("open fixture"); + file.write_all(body.as_bytes()).expect("write fixture"); + } + #[cfg(not(unix))] + fs::write(path, body).expect("write fixture"); +} + +fn path_str(path: &Path) -> &str { + path.to_str().expect("utf8 path") +} + +/// A terraform-shaped CLI: `destroy` confirms by prompt, and its help lists +/// the Go-style `-auto-approve` bypass. +const DESTROY_CLI: &str = r#" "destroy --help") printf 'Usage: tool destroy [options]\n\nOptions:\n\n -auto-approve Skip interactive approval.\n' ;; + "--version") echo "tool 1.0.0" ;; + *) printf 'Usage: tool \n\nCommands:\n destroy Destroy everything\n list List things\n' ;;"#; + +#[test] +fn confirm_flags_from_the_repo_file_pass_force_yes_and_name_the_file() { + let fixture = Fixture::new(DESTROY_CLI); + let without = fixture.row("p5-must-force-yes"); + assert_eq!(without["status"], "fail", "row: {without}"); + + let row = fixture + .repo_config("[p5]\nconfirm_flags = [\"-auto-approve\"]\n") + .row("p5-must-force-yes"); + + assert_eq!(row["status"], "pass", "row: {row}"); + assert_eq!( + row["evidence"], "destroy accepts -auto-approve via .anc.toml [p5].confirm_flags", + "row: {row}" + ); +} + +#[test] +fn confirm_flags_in_both_files_credit_the_repo_file() { + let fixture = Fixture::new(DESTROY_CLI); + fixture.home_config("[p5]\nconfirm_flags = [\"-auto-approve\"]\n"); + let home_only = fixture.row("p5-must-force-yes"); + assert_eq!( + home_only["evidence"], + "destroy accepts -auto-approve via $AGENTNATIVE_HOME_CONFIG [p5].confirm_flags", + "row: {home_only}" + ); + + let row = fixture + .repo_config("[p5]\nconfirm_flags = [\"-auto-approve\"]\n") + .row("p5-must-force-yes"); + + assert_eq!( + row["evidence"], "destroy accepts -auto-approve via .anc.toml [p5].confirm_flags", + "row: {row}" + ); +} + +/// A biome-shaped CLI: `clean` removes log files and takes no confirmation +/// flag; `delete` confirms with `--force`. +const CLEAN_CLI: &str = r#" "clean --help") printf 'Usage: tool clean\n\nOptions:\n -h, --help Show help.\n' ;; + "delete --help") printf 'Usage: tool delete \n\nOptions:\n --force Skip the prompt.\n' ;; + "--version") echo "tool 1.0.0" ;; + *) printf 'Usage: tool \n\nCommands:\n clean Remove the log files\n delete Delete an item\n list List items\n' ;;"#; + +#[test] +fn not_destructive_from_the_home_file_passes_force_yes_and_names_the_file() { + let fixture = Fixture::new(CLEAN_CLI); + let without = fixture.row("p5-must-force-yes"); + assert_eq!(without["status"], "fail", "row: {without}"); + + let row = fixture + .home_config("[p5]\nnot_destructive = [\"clean\"]\n") + .row("p5-must-force-yes"); + + assert_eq!(row["status"], "pass", "row: {row}"); + assert_eq!( + row["evidence"], + "declared not destructive: clean via $AGENTNATIVE_HOME_CONFIG [p5].not_destructive", + "row: {row}" + ); +} + +/// A kubectl-shaped CLI: the top-level help names no output format, `get` +/// carries `-o, --output`, the safe probes answer in text, and +/// `version --client -o json` prints JSON. +const KUBECTL_LIKE_CLI: &str = r#" "version --client -o json") echo '{"clientVersion":{"gitVersion":"v1.0.0"}}' ;; + "get --help") printf 'Display one or many resources.\n\nOptions:\n -o, --output='"''"': One of: json, yaml, wide.\n' ;; + "--version") echo "tool 1.0.0" ;; + *) printf 'tool controls the thing.\n\nBasic Commands:\n get Display one or many resources\n version Print the version\n' ;;"#; + +#[test] +fn json_probe_from_the_repo_file_validates_json_output_and_names_the_probe() { + let fixture = Fixture::new(KUBECTL_LIKE_CLI); + let without = fixture.row("p2-must-output-flag"); + assert_eq!(without["status"], "skip", "row: {without}"); + + let row = fixture + .repo_config("[p2]\njson_probe = [\"version\", \"--client\", \"-o\", \"json\"]\n") + .row("p2-must-output-flag"); + + assert_eq!(row["status"], "pass", "row: {row}"); + assert_eq!( + row["evidence"], + "`tool version --client -o json` printed JSON; probe declared via .anc.toml [p2].json_probe", + "row: {row}" + ); +} + +/// The kubectl-shaped CLI with `explain`, its schema surface, under a block +/// heading that carries a parenthetical. +const KUBECTL_EXPLAIN_CLI: &str = r#" "version --client -o json") echo '{"clientVersion":{"gitVersion":"v1.0.0"}}' ;; + "get --help") printf 'Display one or many resources.\n\nOptions:\n -o, --output='"''"': One of: json, yaml, wide.\n' ;; + "--version") echo "tool 1.0.0" ;; + *) printf 'tool controls the thing.\n\nBasic Commands (Intermediate):\n explain Get documentation for a resource\n\nOther Commands:\n get Display one or many resources\n version Print the version\n' ;;"#; + +#[test] +fn schema_command_from_the_repo_file_passes_schema_print_and_names_the_file() { + let fixture = Fixture::new(KUBECTL_EXPLAIN_CLI); + fixture.repo_config("[p2]\njson_probe = [\"version\", \"--client\", \"-o\", \"json\"]\n"); + let without = fixture.row("p2-must-schema-print"); + assert_eq!(without["status"], "fail", "row: {without}"); + + let row = fixture + .repo_config( + "[p2]\njson_probe = [\"version\", \"--client\", \"-o\", \"json\"]\nschema_command = [\"explain\"]\n", + ) + .row("p2-must-schema-print"); + + assert_eq!(row["status"], "pass", "row: {row}"); + assert_eq!( + row["evidence"], + "`tool explain` is the schema command declared via .anc.toml [p2].schema_command; \ + structured output shown by `tool version --client -o json`, the probe declared via \ + .anc.toml [p2].json_probe", + "row: {row}" + ); +}