Proposal: assertion-driven SVG test harness using the resvg suite (no similarity scores) #140

Proposal in one minute

Status: approved by the owner on 2026-09-09; the bounded pilot is merged in gridaco/nothing#141.

The first implementation is available in the SVG assertion tool's contract and setup. It carries two exact rendering assertions and one separately labelled refusal assertion. This issue stays open for separately reviewed expansion and legacy succession; it does not authorize tool retirement.

Audience: the maintainer reviewing the setup and mental model, and contributors who will later implement or use it.

Build an assertion-driven SVG test harness, initially using inputs from the resvg test suite. An assertion is a precise statement we can check: for example, “this rectangle renders exactly like the declared Chromium reference under these settings.”

The result is PASS or FAIL for that statement, not a percentage of visual similarity. If the setup or reference is not trustworthy enough to judge, the result is UNRESOLVED, with a reason. Unreviewed inputs remain observations, not invented tests.

The resvg suite supplies questions. Our harness executes tests. Chromium is the existing browser-compatibility reference; fresh resvg renders and stored suite PNGs are separately identified second opinions. None is automatically correct for every case.

This proposal is for automated comparisons and useful regression gates, not just a gallery. It does not propose an SVG completeness score.

1. Why the unit is a claim, not an image percentage

A similarity percentage depends on how much of the picture the interesting feature occupies. A large correct background can hide a completely missing small feature. Adding empty space can improve that percentage without fixing anything.

For a declared exact-image test, one differing channel in one pixel fails the assertion. That means “this exact assertion was violated,” not “the renderer is entirely wrong.” Pixel locations, differing-pixel counts, and channel deltas explain a failure; they are not grades.

An atlas may exercise several independent behaviors. Preserve the original as an observation, then create focused, separately described cases for its claims where needed. A whole-atlas comparison may still be a valid integration assertion, but it must not masquerade as proof of every feature pictured. Do not crop or hide failing regions after seeing a result to manufacture a pass.

Even binary test results must not be averaged into “SVG support”: duplicate easy cases and missing hard cases would distort that number. No similarity scores, pass-rate percentages, rankings, or conformance aggregates are in this proposal. The independent FLIP decision remains with gridaco/nothing#49, unchanged.

2. The mental model

Pinned upstream SVG + resources
             |
             v
Describe the claim and its environment
             |
             v
Render with n0 strict / best effort, Chromium, and resvg
             |
             v
Preserve images, diagnostics, versions, and repeat checks
             |
             v
Compare named pairs -- without choosing a winner
             |
      +------+------------------+
      |                         |
Not ready to judge       Reviewed assertion + valid setup
      |                         |
Observation / unresolved       PASS or FAIL
      |                         |
Investigate or fix setup       Enforce in regression gate

These are separate concepts:

• Input: the SVG and every dependency needed to interpret it.
• Claim: the specific behavior being tested, including what it does not establish.
• Observation: what one identified renderer actually produced, including errors and omissions.
• Reference: the independently justified expected result for a particular claim.
• Assertion: the executable check against that expectation.
• Verdict: whether that assertion holds, or why it cannot yet be judged.

A description and feature tags live in case metadata, not painted labels that could affect the test. A filename or the presence of an element is only a discovery hint; neither proves the behavior is active.

3. What PASS, FAIL, and UNRESOLVED mean

Keep the kind of assertion separate from its verdict.

A passing refusal assertion is not a rendering-support pass. Its visible label should be “expected refusal verified.” Any fallback pixels it also checks need their own explicit expectation.

UNRESOLVED means a correctness judgment is not available: for example, a missing resource, unmatched font environment, unreviewed reference disagreement, or a case that does not exercise its claimed branch. An input with no reviewed claim is simply an observation. Neither receives a hidden zero or a green badge.

If a required CI case becomes unresolved or cannot run, the job fails closed as an infrastructure/review problem. It must not silently skip that case and succeed. Conversely, exploratory comparisons do not make every existing unsupported feature a new required CI failure.

4. Which picture is the reference?

Keep these observations distinct:

The comparison report shows separate named pairs, such as n0–Chromium and n0–resvg. A pass against either one does not automatically satisfy the other. There is no “take whichever matches n0 best” rule, and the stored PNG plus resvg are not automatically two independent votes.

For a standards disagreement, inspect the applicable specification and a focused control that distinguishes the interpretations. Another browser may provide evidence later. Record the reasoning and obtain the required review before selecting a new reference. This issue does not ratify a general consensus policy or a resvg-derived gate.

For exact comparisons, compare dimensions and decoded RGBA, not compressed PNG-file bytes. Retain file hashes for provenance. Invisible RGB under zero alpha may be reported separately, but it is not silently normalized away. No blanket antialiasing exclusion, resizing, background flattening, or new tolerance is introduced. Existing narrowly justified comparison rules remain confined to their current cases.

5. Worked examples from the scratch spike

These are measured, not celled observations from 2026-09-09, not newly admitted tests. The spike examined 32 selected upstream inputs and 16 follow-up source forms, not a representative conformance denominator. Source revision: d8e064337faf01bc5a9579187a56dbdbe3eacc72; resvg 0.47.0; Chromium 149.0.7827.55; n0 at ef2e125168a610ac0a64456a12c395963a41911e. Full artifacts remain local-only, not published evidence or dependencies of this issue. Reproduction and durable controls are required before promotion.

• Viewport units: shapes/rect/vw-and-vh-values.svg renders the rectangle in Chromium. Current resvg warns and skips it; n0 refuses it by name. The same viewport-unit and explicit-number sources match in Chromium; removing the subject changes the picture. Interpretation: a real exercised branch, not a global resvg-vs-Chromium winner. n0 has no rendering pass here; its refusal can be tested separately.
• Missing external image: filters/feImage/simple-case.svg references a relative PNG. The sealed Chromium environment cannot load it, while resvg can. n0 best effort and Chromium match because both omit it; n0 declares the omission. Interpretation: environment-unaligned, not image support.
• Placeholder: filters/feDisplacementMap/simple-case.svg contains no displacement primitive, filter, or filtered target. Matching its decorative frame proves no displacement behavior. Interpretation: do not promote it as that feature's test.
• Actual n0 mismatch: the star-shaped clip-path case renders without a refusal but differs from Chromium at four pixels, with maximum channel delta 25. Interpretation: preserve and investigate the difference; do not label it harmless or relax a comparison rule to admit it. The renderer fix is separate work.

6. Setup and daily use

Two lanes share the same capture and comparison machinery:

1. Explore: explicitly provision a pinned upstream suite into ignored local storage; select cases by meaningful applicability; run comparisons and inspect images/diagnostics. Broad inputs may remain undescribed observations. This lane finds work; it does not grant support.
2. Gate: review a small set of described assertions, prove their sensitivity, and register reproducible fixtures through the existing admission workflow. CI runs these assertions without requiring someone to inspect every unchanged picture.

A normal feature session becomes: select a claim → run its cases → understand differences → fix or retain a named refusal → add a reviewed regression assertion → let CI protect it. Known mismatches remain visible work items; selection into the required set is explicit and reviewed, not automatically “only whichever cases already match.”

Each run records source/resource/font identities, renderer and capture versions, initial viewport, authored root size, export mapping, device scale, background/color handling, time, commands, diagnostics, and repeated outputs. Matching output dimensions alone does not establish matching environments.

The first implementation slice should be static, self-contained SVGs that fit the existing capture contract. External resource I/O, general font resolution, arbitrary export sizing, animation expansion, and WPT adapters are follow-ups, not hidden prerequisites or permission to add another browser capture route.

The downloaded corpus remains local-only and explicitly provisioned. Required CI fixtures must be available from a clean checkout with their licenses/provenance; they must not depend on a developer's ignored download. Original upstream inputs and references remain immutable. Diagnostic reductions receive separate identities and are rendered independently.

7. Implementation tracking

• Review this model. Settle assertion granularity, result meanings, reference policy, and initial scope here before code or tool retirement.
• Define the case/result contract. Document identity, claim, environment, reference decision, observation, assertion kind, verdict/reason, and promotion history. No similarity field or score aggregation. Keep runner errors separate from semantic refusals.
• Build the smallest runner and viewer. Invoke the actual n0 CLI in both admissions; reuse the sole pinned Chromium capture; record fresh resvg and stored PNGs separately. Preserve repeatable artifacts and display the claim, chosen expectation, diagnostics, original images, and exact differences together. Use bounded execution/concurrency and explicit timeouts. Do not create a new renderer or semantic parser in the harness.
• Exercise the harness's failure modes. Test one-channel perturbation, wrong dimensions, missing output, changed source/reference hash, nondeterminism, an unexpected refusal, a missing expected refusal, a non-discriminating placeholder, and matching images with missing content. No missing required case may yield a successful gate.
• Promote a small reviewed pilot. Start with aligned, self-contained exact cases plus separate refusal assertions. A source control must demonstrate that each claimed branch matters; deliberately breaking the engine path must fail its gate, followed by exact restoration and a green re-gate. Full-image equality and sparse semantic probes are distinct assertions; a probe must not replace a failing exact-image requirement.
• Integrate without weakening existing gates. Register new committed primitive cells using just add, then just bake, just gate, and just status; use the corresponding established workflow for other assertion kinds. Preserve all existing oracles, admitted-slice records, and checklist tick rules. The tool routes reviewed evidence into the existing corpus, not a second authoritative support list.
• Review legacy succession separately. Inventory useful behavior in the old runners, map what is preserved or replaced, verify the replacement, and only then propose retirement. Do not port best-of-oracles scoring or legacy renderer routing. No deletion is approved here.

Acceptance for the pilot: a contributor can reproduce the setup, read a result without guessing what it proves, see why an unresolved case cannot pass, and deliberately make each required check fail. The tool must not compute or inspect conformance scores, change FLIP, silently bless images, or tick feature rows.

8. Relationship to existing records and feedback

This is the approved pilot model and its follow-up tracker, not a new normative support record. The existing admitted slice, Web checklist, and fixture tooling retain their roles. In particular, a passing example does not establish a whole checklist row's listed grammar.

The consolidation program, gridaco/nothing#43 supplies the standing boundaries. Corpus provisioning should coordinate with gridaco/nothing#7, not duplicate it. The old scored runners remain reference-only for this work; the current Chromium-gated corpus is not being retired with them.

The approval settled these initial questions:

• Whether “one described claim, one discrete verdict” is the right unit and is clear enough to use.
• Whether the separation of rendering assertions, refusal assertions, and unresolved observations is understandable.
• Whether the two-lane setup and narrow first pilot are the right starting point.

The owner gave implementation approval after reviewing this issue. Wider resource/font/export profiles, WPT adapters, a general reference-consensus rule, and legacy retirement still require their own scoped work. FLIP remains unchanged.