Repository navigation
Conversation
56 records missing nist_ai_rmf, 10 missing mitre_atlas (66 entries total, matching #245/#260/#264's earlier passes). Every entry added is {"pin_status": "unknown", "read_date": "2026-10-15"}, per the honesty rule: a real version/commit pin requires a citable source, and none exists for any of these 66. mitre_atlas: checked git log -S'"mitre_atlas"' on each of the 10 records back to the PR that added the mapping. Every one verified the technique against MITRE's own ATLAS data at the time ('confirmed against MITRE's own ATLAS data repository', etc.), but none cites a dated snapshot, version, or commit sha for that data. No citable pin exists for any of the 10. nist_ai_rmf: git log --all -i --grep 'AI RMF' finds no dated verification event anywhere in history, only the field's own creation commit and two unrelated record additions that happen to mention the framework in their own mapping. No citable pin exists for any of the 56. No partial entries existed to overwrite (checked first: every missing field was either absent from framework_sources entirely or present with neither a version/commit nor a read_date -- none of the 66 had a partial, in-between entry). Surgical insertion, not full-file reserialization: these files use a custom formatter that collapses short arrays onto one line, which plain json.dumps(indent=2) does not reproduce (74 of 87 records fail a byte round-trip under it, discovered before writing anything). Each new entry is spliced into the existing framework_sources block's raw text instead, matching the surrounding object-nesting indent exactly. git diff -U0 records | grep against pin_status/read_date/the new field names/framework_sources/brace-only lines prints nothing, and last_updated is untouched on all 61 records. Signed-off-by: chaksaray <chaksaray@gmail.com>
…laim docs/terms-relationships.md hand-wrote coverage numbers that were stale even before this pass (corpus grew from 80 to 87) and, checked directly against the live records, one clause was simply wrong: it said the 40 mitre_atlas entries from the original backfill covered 'the ones a dated, citable audit event actually verified.' Every one of those 40 (now 50, after this pass added 10 more of the same kind) has pin_status: unknown, not a verified pin -- that was never an accurate description of what they are. Corrected rather than carried forward. Real numbers as of this backfill: owasp_mcp 87/87 and owasp_asi 74/74 real-pinned; mitre_atlas 4/54 real-pinned (AVE-2026-00082/84/86/87, ATLAS v5.6.0), 50/54 unknown; nist_ai_rmf 56/56 unknown, previously 0. docs/terms.md regenerated from the schema and the sync check passes. Signed-off-by: chaksaray <chaksaray@gmail.com>
Added 'framework_sources present for every mapped framework (hard failure)' to .github/workflows/tests.yml, running check_framework_sources.py --strict with no --only, right after the Mutation-ThM trail soft warning step. Proved it actually bites before committing: temporarily removed AVE-2026-00001's nist_ai_rmf entry, --strict exited 1 naming that record and field, then restored the file (git diff --stat records confirms empty, nothing left dirty). Updated the script's own header, which previously said corpus-wide enforcement was deliberately deferred until after the backfill; it is done as of 2026-10-15, so this now says that plainly instead of still describing a deferred state. CONTRIBUTING.md's --strict --only line updated to note CI enforces this for every record now, not just that a new record needs it. Signed-off-by: chaksaray <chaksaray@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #326.
Three commits: data, docs, CI.
Before -> after, per framework
unknownafter66 entries added across 61 records, all
{"pin_status": "unknown", "read_date": "2026-10-15"}. Zero real pins added: no citable source exists for any of the 66.Evidence search
mitre_atlas (10 gaps):
git log -S'"mitre_atlas"'on each record back to the commit that added the mapping, then read that commit's message/PR. All 10 verified the technique against MITRE's own ATLAS data at the time ("confirmed against MITRE's own ATLAS data repository," etc.), but none cites a dated snapshot, version, or commit sha. No citable pin for any of them, so all 10 areunknown.nist_ai_rmf (56 gaps):
git log --all -i --grep "AI RMF"across the whole history finds no dated verification event, only the field's own creation commit and two unrelated record-addition commits that happen to mention the framework. No citable pin exists, so all 56 areunknown.A real finding beyond the backfill itself
docs/terms-relationships.mdhad a specific, checkable claim about the 40 pre-existingmitre_atlasentries (from #260): that they covered "the ones a dated, citable audit event actually verified." Checked directly against the live records before writing the doc update: every one of those 40 (now 50, after this PR's 10 more of the same kind) haspin_status: "unknown", not a verified pin. That clause was never accurate. Corrected in the docs commit rather than carried forward silently; see that commit's message for the full reasoning.Mutation-check: the CI step actually bites
Before committing the CI change, temporarily removed
AVE-2026-00001'snist_ai_rmfentry and ran--strict:Then restored the file;
git diff --stat recordscame back empty, confirming nothing was left dirty.Serialization note (not in the original task brief, found before writing anything)
The records use a custom formatter that collapses short arrays onto one line; plain
json.dumps(record, indent=2)does not reproduce that (74 of 87 records fail a byte round-trip under it, checked directly before writing the backfill script). A full-file reserialization would have reformatted every short array in every touched file, far beyond the intended change. Instead, each new entry is spliced into the existingframework_sourcesblock's raw text (all 61 records already had one, just missing a sub-key), matching the file's own object-nesting indentation exactly, confirmed againstrecords/AVE-2026-00001.json's existing entries and against PR #264's own diff (a clean, minimal insertion with zero reformatting elsewhere, the same pattern this follows).Rebase note
This branch was cut before several other PRs from the same session landed on
develop(the ThM-wording correction, the vendor-neutrality cleanup, the main/develop sync). Rebased onto currentorigin/developbefore pushing; the diff below reflects that, not a second round of changes.Validation
python scripts/validate_records.py-> all 87 records validpython scripts/check_framework_sources.py --strict-> exit 0, no warning linepython scripts/check_confidence_signal.py->9 finding(s) across 8 of 87 record(s), unchanged from the pre-backfill baselinepython scripts/check_confidence_signal.py --ratchet-since origin/develop-> all 61 edited records stateevidence_vantagewhere the high band requires itpython scripts/check_terms_sync.py->docs/terms.md matches the live schemapytest tests -q-> 519 passedgit status --short dist/-> emptygit diff origin/develop | grep '^+' | grep -c $'—'-> 0After this merges, the "Regenerate consolidated records JSON" workflow will open the
dist/sync PR on its own; someone should confirmdist/matchesrecords/once that lands.