Fix audit coverage and preserve trustworthy comparison evidence

.github/workflows/test.yml

@@ -21,6 +21,8 @@ jobs:
         run: python3 tests/test_safetensors.py
       - name: Run report rendering self-tests
         run: python3 tests/test_report.py
+      - name: Run CLI and rubric regression tests
+        run: python3 tests/test_cli.py
       - name: Smoke-test CLI help
         run: |
           python3 model_audit.py --help > /dev/null

docs/limitations.md

@@ -9,10 +9,10 @@ eliminate it.
 |---|---|---|
 | SHA-256 (local) | This exact byte stream is what you hashed | That those bytes are safe |
 | `--hf` remote check | Your copy matches what the repo publishes **at that moment/revision** | That the repo's copy was ever safe — you inherit whatever trust the uploader has |
-| `--revision` pin | Match against a specific, immutable commit | Anything about commits *before* it; a repo can publish poison at v1 and swap at v2 |
+| `--revision` pin | A full resolved commit identifies a snapshot; symbolic refs remain mutable | Anything about commits *before* it; a repo can publish poison at v1 and swap at v2 |
 | Template scan | The chat instructions carried inside the file contain no (detected) hostile code, hidden chars, or exfiltration patterns | That the scanner's pattern list is complete. Novel attacks evading these signatures will pass |
 | Metadata checks | Internal labels agree with filename claims; provenance KVs exist | That the labels are truthful — an uploader can set `general.name` to anything |
-| Tensor census | Structure is complete and sane (no missing/empty blocks) | That tensor *values* are what they should be |
+| Tensor census | Implemented inventory checks completed; legal empty safetensors tensors receive an inventory-heuristic WARN but remain structurally valid and baseline-eligible | That tensor *values* are what they should be |
 | safetensors header checks | The header parses, its declared lengths fit the file, each tensor's dtype × shape equals its declared byte span, spans never overlap, and no bytes are unaccounted for (gap or trailing data) | That the weights are what the header says — the header is text the uploader wrote. A perfectly self-consistent file can be a perfectly tampered one |
 | Template sidecar scan (safetensors) | The templates shipped next to the weights (`chat_template.jinja`, every `chat_template` entry in `tokenizer_config.json`) contain no (detected) hostile code, and two disagreeing sources are flagged | That those are the templates that will actually be used — your app may supply its own — or that the pattern list is complete |
 | `*.index.json` check | The shard description matches the shard it describes (weights present, `total_size` right) | That the other shards in the set, or the tensors' contents, are honest |
@@ -34,7 +34,7 @@ eliminate it.
 
 3. **See upstream history that was never pinned.** If nobody recorded the commit SHA
    (or the pre-conversion safetensors state) at download time, a force-push is
-   invisible in hindsight. HF is not git — history can simply vanish. This is why the
+   invisible in hindsight. A remote reference may become unavailable; a failed lookup alone does not establish why. This is why the
    workflow is: pin the commit **at download**, hash immediately, re-check against the
    pin later.
 
@@ -51,17 +51,38 @@ eliminate it.
    re-check them on every re-download. A clean weights hash says nothing about the
    text your model reads its instructions from.
 
-6. **Verify a quantized GGUF against its safetensors source.** Names and shapes can
-   always be matched against the upstream source tree, and `--tensor-hashes` baselines now
-   carry a **value-level** fingerprint as well as a byte-level one, so a lossless conversion
-   is provable across a storage-type change: llama.cpp upcasts bf16 norms to f32, which
-   changes the bytes while preserving the numbers exactly (verified at the bit level — the
-   f32 value is `bf16 << 16`). A real source→BF16 pair matched on 666 of 667 tensors this
-   way, the outlier being a table the converter computes.
-   Block-quantized weights (Q4_K_M and friends) are numerically transformed by design, so
-   their blobs match neither way — only the tensors quantization leaves untouched (norms,
-   usually) still tie the build to its source. `--diff` warns when you compare baselines made
-   from different formats; do not read those differences as tampering.
+6. **Prove model equivalence or conversion history with fingerprints.**
+   Baselines preserve the `tensors`, `values`, and `shapes` maps and add optional
+   per-tensor `value_metadata`: `dtype`, `byte_order: "little"`, and
+   `normalization: "f32-le-bits-v1"`. Only little-endian BF16/F32 value-to-value
+   digests with supported metadata on both sides establish an **exact
+   normalized-F32-fingerprint match**. F32 has an explicit `values` entry too.
+   Big- or unknown-endian GGUF tensors get byte hashes only, with no value claims.
+   F16's existing struct-based value hash remains only a legacy/inconclusive
+   candidate: its NaN payload conversion is not bit-preserving. Typed F32 metadata
+   with contradictory byte and normalized hashes is rejected by baseline validation.
+
+   BF16 widening to F32 is mathematically exact. A normalized fingerprint match
+   establishes consistency of that canonical normalized bitstream, not historical
+   lineage or a lossless whole-model conversion. Known container formats do not
+   prove the dtype of a raw hash: a legacy BF16 normalized hash can match I32 raw
+   bits. Legacy or otherwise ambiguous candidates remain INCONCLUSIVE.
+
+   Cross-format comparison matches bytes first, supported typed normalized hashes
+   second, and ambiguous candidates last, consuming each tensor once and listing
+   both unmatched sides (complete counts and up to eight names each). A scoped PASS requires nonempty, full byte/typed-normalized
+   correspondence. It does not check shapes, semantic tensor roles, numerical/model
+   equivalence, or history. Unknown formats, ambiguous candidates, empty inventories,
+   and partial correspondence stay CONCERN. Generic comparison limits are INFO;
+   actual ambiguous candidates and empty inventories are WARN. Quantized nonmatches
+   are INFO and remain a comparison CONCERN, not evidence of tampering.
+
+7. **Guarantee one immutable snapshot or bounded hostile-input resource use.**
+   Audit stable completed files. Exact-range hashing rejects short reads, but
+   separate reads are not an atomic snapshot against a concurrent writer. Sidecar
+   size budgets, parser isolation, and hostile-directory confinement are not
+   provided. GGUF template inspection is embedded-only; index inspection is
+   limited to the selected local shard, not a complete bundle validation.
 
 ## What strong verification actually looks like
 
@@ -78,8 +99,14 @@ The gold standard, in order:
 
 ## Practical guidance from verdicts
 
-- `DO NOT RUN` — hostile template or structural breakage found. Don't load it.
-- `USE WITH AWARENESS` — read the specific warnings. Tool-calling models trip
-  "file I/O" style patterns legitimately; the explanation text tells you when.
-- `NO RED FLAGS` — the file is honest about what it is, **not** that it is safe.
-  For anything you care about, generate `--tensor-hashes` and keep them.
+- `CRITICAL FINDINGS: REVIEW BEFORE LOADING` — a critical signal, invalid input,
+  or hard failure needs resolution. This is not proof of malicious intent.
+- `WARNINGS: REVIEW THE DETAILS` — read the specific warnings. Tool-calling models
+  trip "file I/O" style patterns legitimately; the explanation text tells you when.
+- `NO RED FLAGS IN COMPLETED CHECKS` — completed checks found no known suspicious
+  patterns. Read the coverage/rubric for anything absent or uninspected. This proves
+  neither honesty nor safety. Preserve a baseline if comparison matters.
+
+Baseline creation refuses an existing destination with WARN and exit 1 (or exit 2
+if another finding is CRIT). Nothing is overwritten; choose a new `--baseline-out`
+destination and retain the original reference. Creating a baseline is not a comparison.

docs/reading-a-report.md

@@ -1,50 +1,99 @@
 # How to read a model-audit report
 
 You don't need to know anything about model files to understand a report. It
-answers one question — *is there anything hidden in this file that could hurt me?*
-— and it is careful about what it claims.
+shows the evidence the scanner collected and which checks completed. It cannot
+answer whether a model is safe in general.
 
 ## The verdict, at the top of each file
 
 Every audited file gets one of three banners:
 
 | banner | meaning | what to do |
 |---|---|---|
-| **NO RED FLAGS FOUND** | nothing hostile was found in the metadata, chat template, or structure | nothing required; save a fingerprint baseline if the model matters to you |
-| **USE WITH AWARENESS** | something needs your attention, but the file isn't clearly hostile | read each warning's plain-language note; many are false positives from tool-calling models |
-| **DO NOT RUN THIS FILE** | a critical problem was found | delete or quarantine it; get the model from a source that documents its build, or make it yourself |
-
-The three are a *scale of evidence*, not a scale of blame. "NO RED FLAGS" means
-nothing hostile was found — it is **not** a promise that the weights are safe, and
-**not** a statement that the uploader is trustworthy. See the bottom of the report.
+| **NO RED FLAGS IN COMPLETED CHECKS** | completed checks found no known suspicious patterns; read coverage too | review any uninspected areas; save a fingerprint baseline if comparison matters |
+| **WARNINGS: REVIEW THE DETAILS** | something needs your attention | read each warning's plain-language explanation |
+| **CRITICAL FINDINGS: REVIEW BEFORE LOADING** | a critical finding needs investigation | resolve it before loading; it does not establish malicious intent |
+
+The three summarize finding severity, not blame. "NO RED FLAGS IN COMPLETED CHECKS"
+means no known suspicious patterns were found in completed checks — it is **not**
+a promise that the weights are safe, and **not** a statement that the uploader is
+trustworthy. See the bottom of the report.
+
+## Evidence rubric and coverage
+
+Four rows separate Structure, Template inspection, Provenance evidence, and
+Baseline comparison. PASS means a specific check completed and matched its
+criteria; CONCERN means suspicious or inconclusive evidence; FAIL means a
+confirmed structural/reference contradiction. NOT CHECKED records why a check
+was absent, unrequested, unavailable, or incomplete. N/A is an explicit exemption.
+
+COMPLETE/INCOMPLETE describes requested applicable coverage, not model safety.
+Optional unrequested remote checks do not count as failures. A template signature
+can be CONCERN while its original severity remains Critical. A checksum PASS
+means only agreement with the named source, not that the source is trusted.
+Same-format baseline PASS is limited to recorded bytes and shapes: dtype, byte
+order, tensor roles, and model behavior remain unverified. Cross-format PASS has
+a different scope: nonempty, full one-to-one correspondence by bytes or supported
+typed normalized fingerprints. It does not check shapes, semantic roles,
+numerical/model equivalence, or conversion history. Unknown formats, ambiguous
+candidates, empty inventories, and partial matches remain CONCERN. There is no
+overall numeric score.
+
+Cross-format comparisons use byte matches first, typed normalized matches second,
+and ambiguous candidates last. Each tensor is used at most once; unmatched tensors
+on both sides are reported with complete counts and up to eight names each.
+An **exact normalized-F32-fingerprint match** requires
+explicit normalized hashes and supported dtype, little-endian byte order, and
+`f32-le-bits-v1` metadata on both sides, for BF16 or F32. BF16 widening to F32 is
+mathematically exact; a matching fingerprint establishes consistency of the
+normalized bitstream, not a lossless whole-model conversion or historical lineage.
+F16's older normalization remains an inconclusive candidate because it does not
+preserve NaN payload bits. Contradictory byte and normalized hashes for typed F32
+metadata are rejected when validating a baseline.
+
+Older baselines remain readable, but an untyped normalized hash matching a raw
+hash is **INCONCLUSIVE**. A known file format does not tell us the raw tensor's
+dtype; I32 bits, for example, could match a BF16 normalized fingerprint. Actual
+ambiguous candidates and empty inventories get Warning findings. General comparison
+limits and quantized nonmatches are Info; a partial quantized comparison can be
+CONCERN even with exit 0 and does not itself suggest tampering. Big- or unknown-endian
+GGUF tensors provide byte fingerprints only, without normalized-value claims.
 
 ## The identity card
 
 A small table describes what the file *claims* to be:
 
 - **Model type** — the architecture and quantization (e.g. `gemma4 · quant Q4_K_M`).
   A mismatch with the filename or the model card is worth noticing.
-- **Chat template** — whether the instructions the model runs on are clean.
+- **Chat template** — whether supported instructions were inspected and known
+  suspicious patterns were found; uninspected is never clean.
 - **SHA-256 fingerprint** and **Size** — the file's identity. Write these down; if
   the file ever changes, they won't match.
 
 ## The finding levels
 
 Below the banner, every finding is tagged:
 
-- 🔴 **Critical** — a concrete hostile pattern (e.g. invisible characters hidden in
-  the chat template). These set the verdict to DO NOT RUN.
+- 🔴 **Critical** — a critical signal, invalid structure, or hard error (for example,
+  invisible template characters). These set CRITICAL FINDINGS: REVIEW BEFORE LOADING;
+  they do not prove intent.
 - 🟠 **Warning** — something unusual that deserves a look. Often benign; read the
   plain-language note under it before worrying.
-- ⚪ **Info** — a fact about the file, recorded for completeness. Not a problem.
+- ⚪ **Info** — context or a limit on the comparison, rather than a warning by itself.
 
 Warnings and Info lines each carry a one-sentence explanation in plain language —
 that note is usually all you need.
 
+An empty safetensors tensor is structurally legal and can be fingerprinted for a
+baseline, but receives an inventory-heuristic Warning worth reviewing. An existing
+baseline output destination also gets a Warning and exit 1, unless an unrelated
+Critical finding sets exit 2. It is never overwritten; choose a fresh destination
+with `--baseline-out`.
+
 ## What the report can and cannot tell you
 
-- It **can** show what the file claims to be, whether anything hostile hides in the
-  instructions it carries, and whether its structure contradicts itself.
+- It **can** show self-reported labels, known suspicious instruction patterns,
+  structural contradictions, and the scope actually inspected.
 - It **cannot** prove the mathematical weights are free of subtle tampering. For
   that, generate per-tensor fingerprints (`--tensor-hashes`) and compare against a
   build you trust.
@@ -53,9 +102,11 @@ that note is usually all you need.
 
 ## A worked example
 
-`examples/report-04-do-not-run-template-swap.html` is a DO NOT RUN report — on a
-model that is **not** suspect. We took a clean model and replaced only its chat
-template with a hostile one, changing nothing else. Compare it with
+The committed files under `examples/` are legacy samples with older banners and
+explanations. `examples/report-04-do-not-run-template-swap.html` has the historical
+DO NOT RUN banner: its chat template was replaced with a hostile one, changing
+nothing else. The example does not establish the safety of the unchanged weights.
+Compare it with
 `examples/report-01-clean-gemma-12b-obliterated.html`: same model, opposite
 verdicts. It is the clearest possible illustration of what the audit is actually
 looking at — and of how much one metadata string can matter.