Skip to content

Three record rules the schema and docs state that no CI step checks #334

Open

Description

@avp9-nexus

The schema and the contributor docs state three rules about records that no step of tests.yml checks today. On a checkout of develop at 733974d, I broke each rule on its own in AVE-2026-00001.json and ran the steps of tests.yml that can fail the run: validate_records.py, validate_crosswalks.py, check_fixtures.py, check_terms_sync.py and the test suite (as python -m pytest tests/ -x -q), plus the confidence ratchet, called through its own ratchet() with the edited record as the change (it reads only confidence_baseline and evidence_vantage). Each broken record passed all six:

  1. severity agrees with aivss.aivss_score. The schema says so on both fields ("Must agree with aivss.aivss_score" on severity, "Must agree with top-level severity" on aivss.aivss_score), CONTRIBUTING gives the bands, and the PR template lists it. AVE-2026-00001 (score 8, HIGH) relabelled LOW passed.
  2. aivss.aivss_severity is the band of aivss.aivss_score. The schema describes it as "Severity label derived from aivss_score". The same record with aivss_severity set to LOW passed.
  3. The required text fields are not blank. The PR template's checklist for new records asks that "All 15 required fields are present and non-empty"; the schema sets no minLength or pattern on these strings (and minLength alone would still accept spaces), so the same record with researcher set to three spaces passed.

All 87 records in records/ meet the three rules today (every one carries an aivss_severity, and it matches its severity), and so do the records that #331 and #327 add or edit, so enforcing them would not change any record.

I have the change ready on a branch (compare with develop) and would like to know whether you want it, and in what shape, before opening a PR:

  • scripts/check_record_rules.py, laid out like validate_records.py: one function per rule, main() over records/AVE-*.json, exit 1 on any finding. A value the schema already refuses (an off-enum severity, a blank ave_id) is not reported a second time.
  • tests/test_check_record_rules.py, which runs the rules over the shipped records, so the test suite enforces them in CI with no change to tests.yml. It also drives each rule on a fixture record with its exact message: the band edges from the README (9.0, 7.0, 4.0), whole-number scores, a draft (held to the eight-field core, as the README says), a deprecated record, and main() over a directory. I removed the script's guards one at a time (nine of them), and each removal turned at least one test red.
  • validate_records.py stays untouched, since feat: add an optional revisions array with a closed reason_code #323 changes it; the three functions could move into it later if you prefer a single validator.
  • The PR would also add a CHANGELOG entry under Unreleased and a line for the script in CONTRIBUTING Step 3.

With that test in place, the same three broken records are caught by the test suite, each with its own message, while the untouched tree and a rewrite of the record with no value changed still pass all six steps.

Two questions:

  • Where should these rules live? The schema file does not change and no record needs migrating, so I read this as the same kind of change as the AIVSS arithmetic and verification_basis checks in validate_records.py, rather than as a schema change. The other way is to write the rules into the schema itself (an if/then per severity band, a pattern on the text fields); that would be a structural change for v1.2.0 (v1.2.0 roadmap #105), and I can propose it there instead if you prefer.
  • Hard failure, or the graduated pattern of check_mutation_thm_trail.py (a soft warning over the corpus, --strict --only for a new or edited record)? With the corpus already clean I would make it a hard failure, but it is your call.

thm and schema_version are left out on purpose: the docs do not agree on what the rule is, so there is nothing settled to enforce yet.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions