feat(ctl): add infrahubctl schema format command - #1189
Conversation
Add an opinionated, offline formatter for Infrahub schema YAML files whose job is to normalise the ordering of keys within each node, generic, attribute, relationship and dropdown choice, so hand-authored schemas read consistently and produce small diffs. - New infrahub_sdk/ctl/schema_format.py with the pure formatting logic: canonical key orders, restricted-namespace filtering (core nodes only), list-item order preserved, a PyYAML dumper matching the schema-library layout, literal-block multiline handling, and a semantic-equality guard that aborts rather than risk changing a file's meaning. - New `format` subcommand in schema.py: in-place by default, plus --check (CI gate) and --diff, with warnings for comments that PyYAML cannot preserve. - Unit + CLI tests and the regenerated infrahubctl CLI reference.
github-actions
Bot
added
the
type/documentation
Deploying infrahub-sdk-python with
|
| Latest commit: |
31dbf19
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://8f06d4d0.infrahub-sdk-python.pages.dev |
| Branch Preview URL: | https://schema-format-command.infrahub-sdk-python.pages.dev |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 6 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Switch the schema formatter from PyYAML to ruamel.yaml round-trip mode so that reordering keys no longer discards comments. This also preserves quoting and inline (flow) sequences (e.g. `[manufacturer, name__value]`) for free, so the diff a format run produces is now purely key-ordering. - schema_format.py: reorder keys in place with move_to_end so the comments ruamel attaches to each key travel with it; keep the semantic-equality guard, restricted-namespace filtering, and canonical key orders. The header is preserved (or added when missing). - Drop the comment-drop warning and count_droppable_comments, which existed only because PyYAML lost comments. - schema.py: format from the raw file text; update the command help. - Add ruamel.yaml to the `ctl` / `all` dependency sets.
There was a problem hiding this comment.
All reported issues were addressed across 7 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
The formatter's canonical key ordering is written against a known set of schema properties. When Infrahub adds or removes a property in the published JSON schema, that ordering may need updating (an unrecognised key is preserved, but not ideally placed). Track this without gating releases: - infrahub_sdk/ctl/schema_drift.py compares the live schema's property sets to a committed baseline (schema_properties.json) and reports added/removed properties. It never raises on drift. - `invoke schema-drift-check` emits GitHub ::warning:: annotations for any drift and always exits 0; `invoke schema-drift-update` refreshes the baseline. - .github/workflows/schema-drift.yml runs the check on release publish and manual dispatch, warn-only. - Baseline snapshot + offline unit tests for the drift logic.
|
Added a warn-only schema-drift check so upstream changes to the published JSON schema get surfaced without gating releases.
Rationale for baseline-vs-live (rather than diffing the formatter's key lists directly): the schema has internal fields the formatter deliberately doesn't order ( Note: because it's release-gated + |
…er detection Address code-review findings: - _format_entity: only iterate `attributes`/`relationships` when they are lists, so a parseable-but-malformed schema (e.g. `attributes: 5`) is left untouched instead of crashing. - _ensure_schema_header: detect a real `# yaml-language-server:` directive line via regex rather than an arbitrary substring, so the header is still added when the string only appears in a scalar or unrelated comment. - test_format_preserves_comments: assert exit_code == 0 so the test can no longer pass silently if the format command fails. Add regression tests for the malformed-section and substring-in-scalar cases.
CI lints with ruff 0.15.12 (develop's pinned version), which enforces pydocstyle D413; the branch's local ruff 0.15.0 did not, so this passed locally but failed in CI. Add the required blank line after the final docstring section in the schema formatter/drift modules and the schema format command.
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## develop #1189 +/- ##
===========================================
+ Coverage 82.65% 82.97% +0.32%
===========================================
Files 139 141 +2
Lines 12354 12614 +260
Branches 1851 1909 +58
===========================================
+ Hits 10211 10467 +256
- Misses 1575 1577 +2
- Partials 568 570 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Sync the 361-commit gap so CI (which validates the merge with develop) uses the same ruff (0.15.12) and doc generator as develop.
_print_schema_diff used markup=False (needed so bracketed diff content like [manufacturer, name] stays literal) together with inline [green]/[red] tags, which then printed verbatim instead of colouring the line. Apply the colour with the style= argument instead.
…a format Three off-by-default transforms for `infrahubctl schema format`, keeping the base command purely key-ordering: - --strip-defaults: remove node/attribute/relationship keys whose value equals the schema default (context-aware; grounded in the published JSON-schema defaults). Consequential/internal fields (branch, state, inherited, display) are intentionally not stripped. - --sort-by-order-weight: sort attributes and relationships ascending by order_weight; items without one keep their authored order and go last. - --backfill-order-weight: give attributes/relationships lacking an order_weight a single constant value (1000). The semantic guard now neutralises exactly the requested transforms on both sides of the comparison, so an intended change is allowed while any unintended corruption still aborts. Verified guard-safe and idempotent across all schema-library files for every flag combination.
There was a problem hiding this comment.
All reported issues were addressed across 6 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| options: Opt-in transforms to apply. | ||
|
|
||
| Returns: | ||
| One of ``"error"``, ``"skipped"``, ``"unchanged"`` or ``"changed"``. |
There was a problem hiding this comment.
It would be cleaner if this returned an enum instead, now there's a loose contract between this function and the if-statement in schema_format()
There was a problem hiding this comment.
Good call — done in b787d03. _format_one_schema_file now returns a FormatOutcome enum (ERROR/SKIPPED/UNCHANGED/CHANGED) and the command loop matches on it with is, so the contract is no longer a loose string.
…guard Address PR review comments: - _format_one_schema_file now returns a FormatOutcome enum instead of loose string literals, tightening the contract with the format command loop (per review feedback). - The semantic guard neutralised list reordering by sorting on `name`, which spuriously aborted when two items shared a name but differed in weight. Sort by full item content instead — a total, content-based order permits any intended reorder while still catching a dropped or corrupted item.
Raise patch coverage on the new modules: - schema_drift: test fetch_live_properties (mocked HTTP) and the write/load baseline round-trip — module now fully covered. - schema format CLI: cover the multi-document, invalid-file, and FormatError branches, plus non-dict list items / extension entries. - Drop a now-dead isinstance guard in _strip_default_keys (both callers already pass a mapping). Type-only test imports moved under TYPE_CHECKING to match the repo convention; no private helpers are imported from tests.
| """ | ||
| options = options or FormatOptions() | ||
| yaml_handler = _build_yaml() | ||
| data = yaml_handler.load(raw_text) |
There was a problem hiding this comment.
A file with a duplicate key kills the whole run here, not just that file.
Round-trip YAML() has allow_duplicate_keys=False, so it raises ruamel's DuplicateKeyError, not a FormatError. That slips past the per-file catch and hits @catch_exception, which prints a traceback and exits 1. SchemaFile uses PyYAML safe_load, which allows dup keys (last wins), so a file that schema load accepts will crash here. On a folder run, files before it are already written and the ones after never run.
Can we catch the ruamel YAMLError and raise FormatError instead, so it stays a normal per-file error?
There was a problem hiding this comment.
Good catch — fixed in 31dbf19. format_schema_text now wraps the round-trip yaml_handler.load() in except YAMLError and re-raises as FormatError, so a duplicate-key file (or anything else ruamel parses more strictly than PyYAML) is reported as a normal per-file error and the rest of the run continues. Added a unit test and a CLI test asserting a dup-key file in a folder is reported per-file while the other files still format.
Round-trip ruamel YAML is stricter than the PyYAML safe_load used to discover schema files — notably it rejects duplicate keys, which `schema load` tolerates (last wins). That raised ruamel's YAMLError, which escaped the per-file `except FormatError`, hit @catch_exception, printed a traceback and exited 1 — aborting a whole folder run midway. Catch YAMLError on load and re-raise as FormatError so it is reported per file and the remaining files still format.
4 participants
What
Adds
infrahubctl schema format— an opinionated, offline formatter for Infrahub schema YAML files. Its core job is normalising the ordering of keys (lines) within each node, generic, attribute, relationship, and dropdown choice, so hand-authored schemas read consistently and produce small diffs.Design
name/namespacefirst,attributes→relationshipslast,order_weightlast within each attribute/relationship, choices asname/label/description/color. Unknown keys are preserved (never dropped).RESTRICTED_NAMESPACESnamespace (Core, Builtin, Internal, Profile, Template, …) are left untouched.ruamel.yaml— comments (the# yaml-language-serverheader, standalone notes, and inline comments), quoting style, and flow-style sequences like[manufacturer, name__value]are all preserved. By default a format run produces a diff that is purely key reordering.Opt-in transforms (off by default)
Three flags go further and change file content; each is neutralised in the safety check so only its intended effect is allowed, and the base command stays purely cosmetic:
--strip-defaults— remove keys whose value equals the schema default, context-aware (e.g.optional: trueis stripped on a relationship but kept on an attribute). Grounded in the published JSON-schema defaults; consequential/internal fields (branch,state,inherited,display) are intentionally excluded to avoid coupling a schema to a default that could later change.--sort-by-order-weight— sort attributes and relationships ascending byorder_weight; items without one keep their authored order and go last.--backfill-order-weight— give attributes/relationships lacking anorder_weighta single constant value (1000).Schema drift tracking
The canonical ordering is written against a known set of schema properties. A warn-only check compares the live JSON schema against a committed baseline so upstream additions/removals are surfaced without blocking releases:
infrahub_sdk/ctl/schema_drift.py+ baselineschema_properties.json;invoke schema-drift-checkemits::warning::annotations and always exits 0;invoke schema-drift-updaterefreshes the baseline..github/workflows/schema-drift.ymlruns it onrelease: publishedand manual dispatch.Changes
infrahub_sdk/ctl/schema_format.py(new) — formatting logic (ruamel round-trip, in-place key reordering, opt-in transforms, transform-aware safety guard).infrahub_sdk/ctl/schema.py— theformatsubcommand (--check/--diff/--strip-defaults/--sort-by-order-weight/--backfill-order-weight).infrahub_sdk/ctl/schema_drift.py,schema_properties.json(new) — drift detection + committed baseline.tasks.py—schema-drift-check/schema-drift-updateinvoke tasks..github/workflows/schema-drift.yml(new) — warn-only drift check.pyproject.toml— addruamel.yamlto thectlandalldependency sets.tests/unit/ctl/test_schema_format.py,test_schema_format_app.py,test_schema_drift.py(new) — 36 tests, incl. comment/flow/quote preservation, malformed-input handling, the three opt-in transforms, and drift logic.docs/docs/infrahubctl/infrahubctl-schema.mdx+ changelog fragment.Verification
uv run pytest tests/unit/ctl/test_schema_format*.py tests/unit/ctl/test_schema_drift.py— 36 passed.uv run invoke format lint-code docs-validate— clean.A companion PR (opsmill/infrahub-skills#74) documents the command in the
infrahub-managing-schemasskill.