Repository navigation
Three record rules the schema and docs state that no CI step checks #334
Description
The schema and the contributor docs state three rules about records that no step of tests.yml checks today. On a checkout of develop at 733974d, I broke each rule on its own in AVE-2026-00001.json and ran the steps of tests.yml that can fail the run: validate_records.py, validate_crosswalks.py, check_fixtures.py, check_terms_sync.py and the test suite (as python -m pytest tests/ -x -q), plus the confidence ratchet, called through its own ratchet() with the edited record as the change (it reads only confidence_baseline and evidence_vantage). Each broken record passed all six:
severityagrees withaivss.aivss_score. The schema says so on both fields ("Must agree with aivss.aivss_score" onseverity, "Must agree with top-level severity" onaivss.aivss_score), CONTRIBUTING gives the bands, and the PR template lists it.AVE-2026-00001(score 8, HIGH) relabelledLOWpassed.aivss.aivss_severityis the band ofaivss.aivss_score. The schema describes it as "Severity label derived from aivss_score". The same record withaivss_severityset toLOWpassed.- The required text fields are not blank. The PR template's checklist for new records asks that "All 15 required fields are present and non-empty"; the schema sets no
minLengthorpatternon these strings (andminLengthalone would still accept spaces), so the same record withresearcherset to three spaces passed.
All 87 records in records/ meet the three rules today (every one carries an aivss_severity, and it matches its severity), and so do the records that #331 and #327 add or edit, so enforcing them would not change any record.
I have the change ready on a branch (compare with develop) and would like to know whether you want it, and in what shape, before opening a PR:
scripts/check_record_rules.py, laid out likevalidate_records.py: one function per rule,main()overrecords/AVE-*.json, exit 1 on any finding. A value the schema already refuses (an off-enum severity, a blankave_id) is not reported a second time.tests/test_check_record_rules.py, which runs the rules over the shipped records, so the test suite enforces them in CI with no change totests.yml. It also drives each rule on a fixture record with its exact message: the band edges from the README (9.0, 7.0, 4.0), whole-number scores, a draft (held to the eight-field core, as the README says), a deprecated record, andmain()over a directory. I removed the script's guards one at a time (nine of them), and each removal turned at least one test red.validate_records.pystays untouched, since feat: add an optional revisions array with a closed reason_code #323 changes it; the three functions could move into it later if you prefer a single validator.- The PR would also add a CHANGELOG entry under Unreleased and a line for the script in CONTRIBUTING Step 3.
With that test in place, the same three broken records are caught by the test suite, each with its own message, while the untouched tree and a rewrite of the record with no value changed still pass all six steps.
Two questions:
- Where should these rules live? The schema file does not change and no record needs migrating, so I read this as the same kind of change as the AIVSS arithmetic and
verification_basischecks invalidate_records.py, rather than as a schema change. The other way is to write the rules into the schema itself (anif/thenper severity band, apatternon the text fields); that would be a structural change for v1.2.0 (v1.2.0 roadmap #105), and I can propose it there instead if you prefer. - Hard failure, or the graduated pattern of
check_mutation_thm_trail.py(a soft warning over the corpus,--strict --onlyfor a new or edited record)? With the corpus already clean I would make it a hard failure, but it is your call.
thm and schema_version are left out on purpose: the docs do not agree on what the rule is, so there is nothing settled to enforce yet.
Activity