Skip to content

Backfill framework_sources for the 61 remaining records, enforce it in CI - #327

Open
chaksaray wants to merge 3 commits into
developfrom
feat/framework-sources-backfill-and-ci
Open

chaksaray wants to merge 3 commits into
developfrom
feat/framework-sources-backfill-and-ci

Conversation

@chaksaray

Copy link
Copy Markdown
Contributor

Closes #326.

Three commits: data, docs, CI.

Before -> after, per framework

Framework Tagged Real pin before Real pin after unknown after
owasp_mcp 87 87 87 0
owasp_asi 74 74 74 0
mitre_atlas 54 4 4 50
nist_ai_rmf 56 0 0 56

66 entries added across 61 records, all {"pin_status": "unknown", "read_date": "2026-10-15"}. Zero real pins added: no citable source exists for any of the 66.

Evidence search

mitre_atlas (10 gaps): git log -S'"mitre_atlas"' on each record back to the commit that added the mapping, then read that commit's message/PR. All 10 verified the technique against MITRE's own ATLAS data at the time ("confirmed against MITRE's own ATLAS data repository," etc.), but none cites a dated snapshot, version, or commit sha. No citable pin for any of them, so all 10 are unknown.

nist_ai_rmf (56 gaps): git log --all -i --grep "AI RMF" across the whole history finds no dated verification event, only the field's own creation commit and two unrelated record-addition commits that happen to mention the framework. No citable pin exists, so all 56 are unknown.

A real finding beyond the backfill itself

docs/terms-relationships.md had a specific, checkable claim about the 40 pre-existing mitre_atlas entries (from #260): that they covered "the ones a dated, citable audit event actually verified." Checked directly against the live records before writing the doc update: every one of those 40 (now 50, after this PR's 10 more of the same kind) has pin_status: "unknown", not a verified pin. That clause was never accurate. Corrected in the docs commit rather than carried forward silently; see that commit's message for the full reasoning.

Mutation-check: the CI step actually bites

Before committing the CI change, temporarily removed AVE-2026-00001's nist_ai_rmf entry and ran --strict:

records/AVE-2026-00001.json | exit: 1 | FAIL: 1 of 87 record(s) carry a framework mapping with no corresponding framework_sources entry: AVE-2026-00001 (nist_ai_rmf)

Then restored the file; git diff --stat records came back empty, confirming nothing was left dirty.

Serialization note (not in the original task brief, found before writing anything)

The records use a custom formatter that collapses short arrays onto one line; plain json.dumps(record, indent=2) does not reproduce that (74 of 87 records fail a byte round-trip under it, checked directly before writing the backfill script). A full-file reserialization would have reformatted every short array in every touched file, far beyond the intended change. Instead, each new entry is spliced into the existing framework_sources block's raw text (all 61 records already had one, just missing a sub-key), matching the file's own object-nesting indentation exactly, confirmed against records/AVE-2026-00001.json's existing entries and against PR #264's own diff (a clean, minimal insertion with zero reformatting elsewhere, the same pattern this follows).

Rebase note

This branch was cut before several other PRs from the same session landed on develop (the ThM-wording correction, the vendor-neutrality cleanup, the main/develop sync). Rebased onto current origin/develop before pushing; the diff below reflects that, not a second round of changes.

Validation

  • python scripts/validate_records.py -> all 87 records valid
  • python scripts/check_framework_sources.py --strict -> exit 0, no warning line
  • python scripts/check_confidence_signal.py -> 9 finding(s) across 8 of 87 record(s), unchanged from the pre-backfill baseline
  • python scripts/check_confidence_signal.py --ratchet-since origin/develop -> all 61 edited records state evidence_vantage where the high band requires it
  • python scripts/check_terms_sync.py -> docs/terms.md matches the live schema
  • pytest tests -q -> 519 passed
  • git status --short dist/ -> empty
  • git diff origin/develop | grep '^+' | grep -c $'—' -> 0

After this merges, the "Regenerate consolidated records JSON" workflow will open the dist/ sync PR on its own; someone should confirm dist/ matches records/ once that lands.

56 records missing nist_ai_rmf, 10 missing mitre_atlas (66 entries
total, matching #245/#260/#264's earlier passes). Every entry added is
{"pin_status": "unknown", "read_date": "2026-10-15"}, per the
honesty rule: a real version/commit pin requires a citable source,
and none exists for any of these 66.

mitre_atlas: checked git log -S'"mitre_atlas"' on each of the 10
records back to the PR that added the mapping. Every one verified the
technique against MITRE's own ATLAS data at the time ('confirmed
against MITRE's own ATLAS data repository', etc.), but none cites a
dated snapshot, version, or commit sha for that data. No citable pin
exists for any of the 10.

nist_ai_rmf: git log --all -i --grep 'AI RMF' finds no dated
verification event anywhere in history, only the field's own creation
commit and two unrelated record additions that happen to mention the
framework in their own mapping. No citable pin exists for any of the
56.

No partial entries existed to overwrite (checked first: every missing
field was either absent from framework_sources entirely or present
with neither a version/commit nor a read_date -- none of the 66 had a
partial, in-between entry).

Surgical insertion, not full-file reserialization: these files use a
custom formatter that collapses short arrays onto one line, which
plain json.dumps(indent=2) does not reproduce (74 of 87 records fail a
byte round-trip under it, discovered before writing anything). Each
new entry is spliced into the existing framework_sources block's raw
text instead, matching the surrounding object-nesting indent exactly.
git diff -U0 records | grep against pin_status/read_date/the new
field names/framework_sources/brace-only lines prints nothing, and
last_updated is untouched on all 61 records.

Signed-off-by: chaksaray <chaksaray@gmail.com>
…laim

docs/terms-relationships.md hand-wrote coverage numbers that were
stale even before this pass (corpus grew from 80 to 87) and, checked
directly against the live records, one clause was simply wrong: it
said the 40 mitre_atlas entries from the original backfill covered
'the ones a dated, citable audit event actually verified.' Every one
of those 40 (now 50, after this pass added 10 more of the same kind)
has pin_status: unknown, not a verified pin -- that was never an
accurate description of what they are. Corrected rather than carried
forward.

Real numbers as of this backfill: owasp_mcp 87/87 and owasp_asi 74/74
real-pinned; mitre_atlas 4/54 real-pinned (AVE-2026-00082/84/86/87,
ATLAS v5.6.0), 50/54 unknown; nist_ai_rmf 56/56 unknown, previously 0.

docs/terms.md regenerated from the schema and the sync check passes.

Signed-off-by: chaksaray <chaksaray@gmail.com>
Added 'framework_sources present for every mapped framework (hard
failure)' to .github/workflows/tests.yml, running
check_framework_sources.py --strict with no --only, right after the
Mutation-ThM trail soft warning step.

Proved it actually bites before committing: temporarily removed
AVE-2026-00001's nist_ai_rmf entry, --strict exited 1 naming that
record and field, then restored the file (git diff --stat records
confirms empty, nothing left dirty).

Updated the script's own header, which previously said corpus-wide
enforcement was deliberately deferred until after the backfill; it is
done as of 2026-10-15, so this now says that plainly instead of still
describing a deferred state. CONTRIBUTING.md's --strict --only line
updated to note CI enforces this for every record now, not just that
a new record needs it.

Signed-off-by: chaksaray <chaksaray@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant