Skip to content

docs: correct AVE-2026-00044's README row and the Stats counts, and check the README against records/ - #332

Open
avp9-nexus wants to merge 2 commits into
aveproject:developfrom
avp9-nexus:docs/readme-index-sync-00044
Open

avp9-nexus wants to merge 2 commits into
aveproject:developfrom
avp9-nexus:docs/readme-index-sync-00044

Conversation

@avp9-nexus

Copy link
Copy Markdown

Type of change

  • New AVE record submission
  • Update to existing AVE record
  • Schema change (v1.0.0)
  • New detection rule (YARA / Semgrep / pattern)
  • Crosswalk addition or update
  • Documentation improvement
  • Other: a test that checks the README against records/

Description

The README record index lists AVE-2026-00044 at 6.1 MEDIUM. The record states 8.1 HIGH: #294 raised its thm from 0.75 to 1 and recomputed the score, as the record's aivss.notes explains, and the README was not part of that change. The Stats table has the same lag: it reads 19 HIGH and 65 MEDIUM, and records/ holds 20 and 64.

This PR corrects those three README lines and adds tests/test_readme_sync.py, which reads both sides and fails on drift:

  • every record in records/ has exactly one row in the record index, and the row states the record's own aivss.aivss_score and severity (a row that is there but not in the index's form is reported as unreadable, not as missing);
  • the Stats total and per-band counts, the records badge and the index summary are the counts of records/.

Draft records are left out of the index, the band counts and the totals, since a draft need not carry a score yet; deprecated records are checked like active ones. If either should be counted differently, say so and I will adjust.

The README is edited by hand: nothing in scripts/ or .github/ writes it, and new-record PRs already update it by hand. The test makes a missed update visible in CI, as check_terms_sync.py already does for docs/terms.md. Since that adds a gate, two sentences in CONTRIBUTING.md say when a record change needs a README update, and the CHANGELOG has an entry. The README's own "Adding a new AVE record" steps and docs/specs/researcher-process.md are left as they are; say if you want the same line there. If you would rather not gate CI on the README, I can drop the test and keep the README fix.

On develop at 733974d, against the README as it stands, the two README checks fail with exactly four messages:

AVE-2026-00044: README says score 6.1, the record says 8.1
AVE-2026-00044: README says MEDIUM, the record says HIGH
Stats says 19 HIGH, records/ holds 20
Stats says 65 MEDIUM, records/ holds 64

With the fix, the full suite passes: 543 passed, 519 without the new file plus its 24 tests. The other twenty-two run the checks on fixtures written in the test: each refusal the checks can make, with its exact message expected, and the cases that must pass (a matching README, a bold severity with a trailing space, a draft record, rows outside their section). Run locally on Python 3.14; the test uses no syntax newer than 3.10, and CI runs 3.11.

Out of scope: the index shows display titles, which can differ from the records' title fields, so the test compares score and severity only; the record count in the prose ("87 records. 8 independent crosswalks.") is not checked, nor are the scores quoted in the worked example and in the example scanner output.


AVE record(s)

AVE-2026-00044 (its README row only; the record itself is unchanged).


Checklist

(The record and schema sections are omitted: no record or schema file is touched.)

For all PRs

  • I have read CONTRIBUTING.md
  • ave_id values are immutable - no renumbering
  • Schema version string is 1.0.0 (n/a: no record or schema file touched)
  • No references to SPEC.md (removed), template.json (removed), or bawbel/bawbel-ave (wrong path)
  • I agree my contribution is licensed under Apache 2.0

…heck the README against records/

The README record index lists AVE-2026-00044 at 6.1 MEDIUM. The record states 8.1 HIGH: aveproject#294 raised its thm from 0.75 to 1 and recomputed the score, and the README was not part of that change. The Stats table has the same lag: it reads 19 HIGH and 65 MEDIUM, and records/ holds 20 and 64.

This corrects those three README lines and adds tests/test_readme_sync.py, which reads both sides and fails on drift. Two sentences in CONTRIBUTING.md say when a record change needs a README update, and the CHANGELOG has an entry.
@chaksaray

Copy link
Copy Markdown
Contributor

Thanks, this is a careful PR. The four README lines are right against an independent count of records/ (87 records: 1 CRITICAL, 20 HIGH, 64 MEDIUM, 2 LOW). The new tests fail on current develop with exactly the four messages you list, and the branch passes (543). The test file is stdlib only and reads just README.md and records/.

I then mutated the checks. All ten mutations of the check functions (score, severity, drafts, band counts, total, badge and summary, duplicate rows, unreadable rows, section boundaries, duplicate record files) are caught by your fixtures. Three survive, all in the real-data tests: dropping index_drift from the first, or dropping stats_drift or restated_total_drift from the second. Those tests compose the checks by hand, separate from every_check, so the fixtures prove each check works but nothing proves the real README goes through all of them. Could both real-data tests call one shared function (every_check, renamed if you like), with a case that sends a README carrying each kind of drift through that same function?

On your open questions: counting deprecated records like active ones is fine for now, and leaving drafts out matches the README. One heads-up: two record PRs (#331, #336) edit the same Stats lines, so they will conflict with this one. I plan to merge this first and have those rebase against the new check.

The two real-data tests composed the checks by hand, apart from every_check, so removing a check from them left every test passing (review on aveproject#332). The same held for the duplicate-record problems that load_records returns.

readme_drift() now reads the records and runs every_check, and the real-data test calls it. A fixture writes a README that drifts against each check, plus a second record file with the same id, and expects each message through readme_drift(). Removing any of the three checks from every_check, or the duplicate-record problems or either path argument from readme_drift(), now fails that fixture. The test file still holds 24 tests.
@avp9-nexus

Copy link
Copy Markdown
Author

Thanks for running the mutations, that was the gap. Pushed cfa76f3: the two real-data tests are now one, which calls readme_drift(). That function reads the records and runs every_check, and a fixture sends a README that drifts against each check, plus a second record file with the same id, through that same function and expects all seven messages.

Your three mutants now fail that fixture, and so do three more of the same kind: dropping the duplicate-record problems from the real-data path (which also survived on the first version), or readme_drift() ignoring the README or the records directory it is given. The test file still has 24 tests, one on the real data and twenty-three on fixtures (so "the other twenty-two" in the description is now twenty-three), and the full suite is still at 543.

Thanks as well for the answers on deprecated and draft records, and for the note on #331 and #336.

Avp9.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants