Fix audit coverage and preserve trustworthy comparison evidence

[am423]

Summary

Makes the existing audit/report pipeline preserve its evidence and distinguish completed checks from uninspected or inconclusive inputs. Adds a four-domain evidence rubric without a numerical safety score.

• Validate safetensors and adjacent JSON before using their fields; reject duplicate keys, invalid dimensions/offsets and non-finite JSON numbers, while preserving legal scalar and empty tensors
• Inspect named and additional chat-template representations, track partial coverage, and compare only competing definitions of the same logical template
• Consume each tensor identity once in cross-format matching; separate byte matches, exact normalized-F32 fingerprint matches supported by typed metadata, and inconclusive candidates, and report both unmatched sides
• Keep existing baselines readable while adding optional per-tensor value_metadata and explicit F32 value hashes; compare recorded shapes while leaving semantic roles, model equivalence and historical lineage unverified
• Refuse baseline overwrites with WARN/exit 1 (--baseline-out selects a new snapshot), preserve unrelated critical findings, reject short tensor reads, and reserve unique exclusive HTML/JSON report pairs
• Keep core JSON stdout as one findings array and return consistent 0/1/2 finding/error statuses from both CLIs
• Add PASS / CONCERN / FAIL / NOT CHECKED / justified N/A evidence rows for structure, templates, provenance and baseline comparison, plus requested-check coverage

No new runtime dependencies or production modules. Existing finding tuples, baseline maps and generate()'s Path return are preserved.

Compatibility changes

Repeated baseline writes now refuse an existing destination. Reports are written into unique run directories rather than minute-resolution filenames. Report CLI finding statuses now matter; the example regeneration script explicitly handles expected statuses and verifies complete pairs. Normal absent safetensors templates remain informational and are shown as NOT CHECKED. Legal empty tensors receive an inventory-heuristic WARN without invalidating the format or preventing baseline generation. The full installation retains a Python 3.10 floor because gguf 0.19.0 declares Requires-Python >=3.10.

A signature-only template finding remains a review concern, not proof of malicious intent; original CRIT severity still returns exit 2. Optional unrequested checks do not become failures. Checksum matches establish consistency with the named reference, not trust in its author.

Existing tests that asserted lossless-conversion/shape-equivalence prose were updated to assert the actual scoped fingerprint evidence. Attack/benign specimen detection remains covered.

Validation

On Python 3.12.14:

• python3 -m unittest discover -s tests: 115 passed, 2 skipped, without gguf and huggingface_hub
• Same suite in a virtual environment with declared dependencies (gguf 0.19.0, huggingface_hub 2.2.0, NumPy 2.5.3): 117 passed
• Both CLI --help smoke tests passed in both environments
• Actual little/big-endian GGUF fixtures, a safetensors BF16-to-GGUF F32 pair, all 65,536 BF16 bit patterns, and dtype-confusion/empty/subset regressions passed
• bash -n examples/regenerate_samples.sh and git diff --check passed

New regressions exercise malformed schemas, template representations/partial scans, exact-range and chunk-boundary hashes, double consumption, comparison coverage, evidence preservation, JSON/exit contracts, rubric states, ambiguous references and report output failures.

The real-model sample regeneration script was not executed because its referenced large model files are not present. Windows, Python 3.10 minimum-version behavior, live HF remote checks and broad adversarial resource testing were not exercised locally. Hosted CI results should be reviewed separately for this exact commit.

Intentionally out of scope

No claim of safe weights, architecture-aware tensor-role equivalence, conversion provenance, complete shard/bundle validation, atomic snapshots or comprehensive denial-of-service resilience. Full manifest reconciliation, input resource budgets, packaging/CI matrix expansion and baseline-schema redesign remain follow-up work.

[ExileAI]

Hi — reviewed on behalf of the maintainer (@grokola). Great PR: it's philosophically aligned with the repo's honesty-first rules, the suite is green on your branch (97 tests on 3.14, including your new test_cli.py), and your compat write-up is unusually candid. We did find one likely regression and two policy questions before this can come out of draft.

1. Did the bf16→f32 value-match proof regress, or was it deliberately re-framed?

Master's tensor_fingerprints() records a value_sha256 = the tensor's values widened to f32 (exact bf16 << 16, verified bit-level against struct in docs/ and the OBLITERATUS specimen: 337 of 667 blobs matched exactly that way). Master's --diff therefore reports an exact value match for a bf16→f32 conversion.

Your branch keeps the widening machinery (_to_f32_bytes, value_sha256 are still there) but the cross-format diff now reports normalized-hash candidates as explicitly INCONCLUSIVE ("numeric equivalence and lossless conversion are not established"), and test_cross_format_upcast_norm_matches_by_value now asserts INCONCLUSIVE where master asserts an exact value match.

Question: is this a deliberate epistemic downgrade (e.g. you consider widening a candidate match because it can't rule out collisions or because the recorded value hash isn't self-describing) or an unintended consequence of the blob_identity rewrite? If deliberate: the OBLITERATUS conversion-chain proof (the repo's flagship validation) depended on that exact-match semantics — if we keep INCONCLUSIVE, we need to know what claim we can still make about source→BF16 pairs, and the README/docs' lossless-conversion prose you removed will need a replacement claim, not just a weaker one.

2. Rubric vs. banner — confirm layering.

The README diff reads as if the four-domain rubric (PASS/CONCERN/FAIL/NOT CHECKED/N-A) layers under the existing NO RED FLAGS / USE WITH AWARENESS / DO NOT RUN banners rather than replacing them ("Finding severity remains independent… retaining exit 2"). Please confirm that's the intent — the banner system is the user-facing contract and we don't want to change it in this PR.

3. Severity demotions worth a second look.

• test_empty_tensor_is_warned → test_empty_tensor_is_informational: a zero-element tensor is exactly the shape of a weights-swap hiding spot (a legit training checkpoint essentially never has one). Why INFO rather than keeping WARN?
• Cross-format --diff gains a blanket WARN that normalized-hash matches are "candidates only" — this will fire on every honest lossless diff too. Concern: a scanner that warns on every legitimate comparison trains users to ignore warnings (house rule #3). Could this be scoped to fire only when a match is relied upon rather than on every cross-format diff?

No rush — it's a draft, we just wanted these on the record early. Happy to hop on the branch and iterate together on 1 and 3 if useful.

[ExileAI]

Follow-up from the maintainer side — we built a selective merge of your branch so the resolution is concrete instead of described in prose. Branch: merge-pr1-selective on this repo (commit 0d03548), based on current master (ac0f1c4). Please review it directly — that's the shape we'd like to land, and the notes below explain every deviation so you can push back where you disagree.

What we kept (the large majority)

Everything, really, except the two items below: the evidence rubric layered under the banners, one-use tensor identity matching with both-side unmatched listing, the stricter safetensors/JSON validation, baseline write refusal + --baseline-out, unique report run directories, tests/test_cli.py, the hardened regenerate_samples.sh, and — per the maintainer's explicit call — your banner rewording ("NO RED FLAGS IN COMPLETED CHECKS" etc.), which we agree is more honest and is now the shipped wording.

Deviation 1 — value-match semantics: exact for known formats, INCONCLUSIVE kept for unknown

We split your cross-format branch in diff_baselines:

• Both baselines are gguf/safetensors → master's exact semantics are restored: "${exact} byte-identical, ${n} more carry identical values in a different storage type." Reason: the value hashes in those baselines are our own widening, and that widening is exact — every f32 value is bf16 << 16, verified bit-level against struct and on a real 667-tensor specimen (329 byte + 337 value matches, one converter-computed table unmatched). It's not a probabilistic candidate; it's arithmetic. Downgrading it to INCONCLUSIVE would also break the repo's flagship validation: the safetensors→BF16 conversion-chain proof that a build carries the source's weights.
• Either side is a legacy/unknown format → your INCONCLUSIVE wording stays unchanged. There we fully agree with you: we can't vouch for what an unknown producer hashed, so matches are candidates-only. Your caution was right — just scoped one level tighter than the PR had it.

If your objection to exact-matching is deeper than provenance (e.g. you see a collision or multiset-ambiguity risk in the bucket matching itself even for known formats), tell us — the fix belongs in match_blobs, not in the wording, and we'll do it there.

Deviation 2 — Python floor stays 3.8+

Your PR bumped the badge and install text to 3.10+. We reverted that: your code parses clean under 3.9 syntax (checked with ast.parse(feature_version=(3,9))), CI runs 3.11, and master promises 3.8+. If some dependency or stdlib call you added actually requires 3.10, point at the line and we'll take the bump — we just don't want an undocumented floor raise.

Validation on the merged branch (this machine, real files)

• All four suites green: scanner 7, safetensors 42, report 22, cli 26.
• Real-model false-positive check: OBLITERATUS Q4_K_M, an mmproj vision file, GPT-OSS MXFP4 — findings identical to master except your additive template-source INFO lines. No new warnings on honest files.
• OBLITERATUS safetensors→BF16 diff: 329 byte + 337 value matches, rope_freqs.weight the only unmatched blob, exit 1 (the format-mismatch WARN) — same as master.
• ComfyUI-ecosystem safetensors (a diffusion model, a VAE, a LoRA — non-HF producers): all clean under your stricter header validation, exit 0.
• Baseline refusal exercised end-to-end on a real file: first write OK, overwrite refused, --baseline-out to a new name OK.
• regenerate_samples.sh (your hardened version) runs the full pipeline including the synthesized CRIT specimen; your collect() guard correctly asserts the template CRIT is present, and the leak check passes.

One polish item, non-blocking

The refused baseline overwrite surfaces as CRIT "input error: [Errno 17] File exists" — safe-by-default, but it means a scripted re-baseline of an unchanged model reads as DO-NOT-RUN-grade. Would you take a follow-up that emits a dedicated "baseline exists — pass --baseline-out to write a new snapshot" message at WARN instead? Happy to do it ourselves if you'd rather not.

The branch is pushed for your review — if the two deviations look right to you (or you fix them differently on your branch), this goes to master with your PR credited.

[am423]

Thanks for the concrete integration branch. Revision 9beae86. I've kept the accepted banners and synced the documentation.

The BF16 widening arithmetic is exact. The issue is interpreting an untyped raw hash: BF16 1.0 widens to bytes 00 00 80 3f, which are also the raw bytes of I32 1065353216. Known container labels cannot distinguish those cases; this does not require a hash collision.

The revision adds optional per-tensor dtype, observed byte order and versioned normalization metadata. Fresh little-endian BF16/F32 baselines can establish exact normalized-F32 fingerprint agreement. Existing baselines remain readable, while ambiguous raw/value matches stay INCONCLUSIVE. F16 and big/unknown-endian GGUF do not receive that stronger claim. Matching consumes each tensor once, with byte matches first.

The follow-up also:

• Covers the selective branch's empty-inventory PASS and omitted unmatched-A cases with regressions, preserving nonempty and both-side checks
• Scopes candidate warnings to actual ambiguous matches; ordinary cross-format limitations and quantized nonmatches are INFO
• Restores empty-safetensors WARN as an inventory heuristic while preserving format validity and baseline eligibility
• Makes only baseline destination conflicts WARN/exit 1, preserving the existing file, completed checks and unrelated CRIT/exit 2
• Retains Python 3.10 for the full installation because gguf 0.19.0 declares Requires-Python >=3.10

Validation on Python 3.12.14: 117 tests pass with gguf 0.19.0, huggingface_hub 2.2.0 and NumPy 2.5.3; 115 pass and two real-GGUF tests skip without gguf and huggingface_hub. This includes actual little/big-endian GGUF fixtures, a safetensors BF16-to-GGUF F32 pair, all 65,536 BF16 bit patterns and the dtype-confusion/empty/subset regressions. Python 3.10 runtime, Windows, large real-model regeneration and live HF checks were not run.

The claim is recorded fingerprint consistency, not tensor-role correspondence, whole-model equivalence or historical lineage. The real-model comparison should be rerun with freshly generated baselines using distinct --baseline-out destinations.

[ExileAI]

Merged. master is now 9717884 (merge commit 3818d71 + your 9beae86), CI green, v0.5.0 shipped. Thank you — genuinely — for one of the best review cycles this repo could have asked for.

Three things worth putting on the record:

You were right, and with the sharper argument. Our "exact for known formats" framing leaned on the widening arithmetic being exact — and it is — but you saw the actual flaw: an untyped raw hash can't distinguish BF16 1.0 from I32 1065353216, because those bytes are identical. No collision involved, and no container label fixes it. The typed-baseline design (dtype + byte order + versioned normalization) is the correct fix: it makes the exact claim earned for fresh little-endian BF16/F32 pairs while correctly keeping everything else INCONCLUSIVE. We verified the behavior directly, not just your test names: a typed BF16(st)↔F32(gguf) pair reports "1 exact normalized-F32 fingerprint matches / 0 candidates → PASS"; the confusion pair reports "0 exact / 1 candidate (INCONCLUSIVE)" + WARN → CONCERN. Both in test_cli.py and reproduced by hand. Your 65,536-bit-pattern and real BE/LE GGUF fixtures are exactly the kind of validation this repo's thesis deserves.

The maintainer's call to skip the 48 GB specimen redownload was made on the strength of your validation writeup plus our local behavior checks; the note in the repo docs flags that the OBLITERATUS numbers (329+337) predate typed baselines and should be regenerated with --baseline-out if anyone wants to re-cite them. Fair warning, honestly labeled.

Housekeeping: our interim merge-pr1-selective branch is deleted (superseded by your revision, as it should be — yours kept everything ours did and added the typed scheme). Changelog credits you: 3818d71 | v0.5.0 — Merge PR #1 (@am423 / mr-r0b0t): evidence rubric under the banners, typed baselines.... Co-author line is on the merge commit.

This PR made the tool more honest than we could have made it alone. whenever you want to pick up the next piece (the README's roadmap has the remote-only audit and quantized-lineage items), the door's open.