Skip to content

fix(ci-versioning): classify a record the closure cannot resolve as unverified - #18

Merged
sakanni merged 1 commit into
developfrom
fix/ci-versioning-resolution-classification
Sep 4, 2026
Merged

fix(ci-versioning): classify a record the closure cannot resolve as unverified#18
sakanni merged 1 commit into
developfrom
fix/ci-versioning-resolution-classification

Conversation

@sakanni

@sakanni sakanni commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

What

A record the closure cannot resolve is classified unverified, not real. It keeps its diagnostic, appears in the artefact, and does not gate. A record whose types all resolve and which still fails is a genuine regression and still gates.

#16's attribution policy and the namespace fallback are untouched. The two compose rather than overlap.

The line

ClassifyUnresolvableCause already computed the right answer and threw it away one line later:

if (namedTypes.Any(t => t.StartsWith("BH.", StringComparison.Ordinal)))
    return null;

The intent was "could CI ever load this", and for RevitAPI the proxy held. It cannot hold generally: every downstream BHoM toolkit type also starts with BH.. Measured on BHoM/BHoM run 33849699768, all 145 findings named a BH. type and all 145 were closure artefacts. Fifth instance of the recorded-then-discarded fault in BHoM/internal-tickets#31.

The replacement

Keyed on the loaded set rather than the name:

named type verdict
present in the loaded set not a blocker
absent, but its namespace is loaded an assembly here owns that namespace and the type is gone. A removal. Real, and gates.
absent, and its namespace too nothing in this closure could have supplied it. Unverifiable.

The middle branch is why this is not simply "absent means unverified." That would have classified an undeclared removal as unverified and blinded the check to the thing it exists to catch. Where a record names both a removal and a closure gap the removal wins, and the removed type has its own record besides.

Also parses the two payload event shapes the deserialiser emits at any nesting depth — Failed to convert the string into a type: X and The type X from version V is unknown -> .... Only the signature shape was parsed before, which is why every object-record failure reached the caller with no cause at all.

Separability

Recorded as UnresolvableTypeAbsent, distinct from UnresolvableFromEvents (signature path) and NoMethodEvent (#16's attribution bucket), keyed on which message shape produced the cause rather than on whether a Method event is present — a signature blocker does not always have one. The job summary now breaks unverified down the same way:

### Versioning inconclusive: 98 finding(s) reported, none verified
| Reported unverified                            | 98 |
| — could not be resolved (closure gap)          | 66 |
| — could not be attributed (inferred ownership) | 32 |

Coverage

CoverageCounts gains RecordsUnverified, printed on the Coverage line even when zero. 1191 subject type(s) checked reads as a clean sweep until you know 145 records were not verified.

Verification against real artefacts

Local public-installer closure, 186 assemblies in the runner's index:

Classification: unverified/UnresolvableTypeAbsent=66, unverified/NoMethodEvent=32
Status:   Warning
Failures: 0
Coverage: 1190 subject type(s) across 30 subject assembl(ies) ... 186 assembl(ies) loaded; 98 record(s) attributed but not verified
exit 0

Same test result classified against a 56-assembly index: 98 → 0 real, 70 + 28. The 32 and 28 are types in namespaces BHoM/BHoM co-inhabits with StructuralEngineering_Toolkit (BH.oM.Structure.{Elements,Loads,Results}), so the removal fail-safe correctly declines and #16 catches them instead. That residue is exactly what provenance (#35) removes, and it is why the two changes are ordered rather than alternatives.

Not verified locally, and stated rather than assumed: the CI 145-finding case. FromJsonDataset calls LoadAllAssemblies(), which reads %ProgramData%\BHoM\Assemblies regardless of the runner's --assemblies argument, so the test's closure cannot be reduced locally without mutating the machine's BHoM install. Locally the test always produces its 98-finding result and only the runner's index can be varied. The 47 CI-only findings are the GSA/Sharepoint-poisoned ones, whose blocking namespaces are absent in CI's closure, so they are predicted to land in UnresolvableTypeAbsent — including the 5 Returned null from json records, which are in that 47 and therefore also unobservable locally. First real run on BHoM/BHoM confirms or refutes it.

Suite 132 passed. Verified by mutation: deleting the removal fail-safe fails TypeMissingFromALoadedNamespace_IsReal and ARemovalAlongsideAClosureGap_StaysReal, and nothing else. PowerShell suite 72 passed with the same 15 pre-existing local environment failures as develop.

Refs BHoM/internal-tickets#32, BHoM/internal-tickets#31

…nverified

ClassifyUnresolvableCause already computed the right answer and then discarded it
one line later:

    if (namedTypes.Any(t => t.StartsWith("BH.", StringComparison.Ordinal)))
        return null;

Its intent was "could CI ever load this", and for RevitAPI the proxy held. It
cannot hold generally, because every downstream BHoM toolkit type also starts
with BH.: measured on BHoM/BHoM run 33849699768, all 145 findings named a BH.
type and all 145 were closure artefacts. That is the same recorded-then-
discarded fault as internal-tickets#31, now its fifth instance.

Replace the name test with the one the runner can actually answer, keyed on the
loaded set rather than the namespace:

  present in the loaded set           -> not a blocker
  absent, but its namespace IS loaded -> an assembly here owns that namespace
                                         and the type is gone. A REMOVAL. Real.
  absent, and its namespace too       -> nothing here could have supplied it.
                                         Unverifiable.

The middle branch is the removal fail-safe and is why this is not simply
"absent means unverified", which would have blinded the check to undeclared
removals — the thing it exists to catch. Where a record names both, the removal
wins; the removed type has its own record besides.

Also parse the two payload event shapes the deserialiser emits at any nesting
depth ("Failed to convert the string into a type: X", "The type X from version
V is unknown -> ..."). Only the signature shape was parsed before, which is why
every object-record failure reached the caller with no cause at all.

Recorded as UnresolvableTypeAbsent, distinct from UnresolvableFromEvents
(signature path) and from NoMethodEvent (#16's attribution bucket), so the
artefact and the summary separate "could not resolve it" from "could not
attribute it". Keyed on which message shape produced the cause, not on whether
a Method event is present, because a signature blocker does not always have one.

Coverage gains RecordsUnverified, printed on the Coverage line even when zero:
1191 subject types checked reads as a clean sweep until you know 145 records
were not verified.

#16's attribution policy and the namespace fallback are untouched. The two
compose rather than overlap: measured locally on 98 findings, resolution catches
66 and attribution the remaining 32, total 0 real.

Verified against real artefacts, not only Pester. Local public-installer
closure, 186 assemblies in the runner's index: 98 findings -> 0 real, 98
unverified (66 UnresolvableTypeAbsent + 32 NoMethodEvent), Status Warning,
exit 0. Same run with a 56-assembly index: 98 -> 0 real, 70 + 28. The 32 and 28
are types in namespaces BHoM/BHoM co-inhabits with StructuralEngineering_Toolkit
(BH.oM.Structure.{Elements,Loads,Results}), so the removal fail-safe correctly
declines and #16 catches them instead. That residue is what provenance (#35)
removes.

Suite 132 passed. Verified by mutation: deleting the removal fail-safe fails
TypeMissingFromALoadedNamespace_IsReal and ARemovalAlongsideAClosureGap_StaysReal
and nothing else. PowerShell suite 72 passed with the same 15 pre-existing local
environment failures as develop.

Refs BHoM/internal-tickets#32, BHoM/internal-tickets#31
@sakanni
sakanni merged commit f5ce541 into develop Sep 4, 2026
5 checks passed
@sakanni

sakanni commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

RETRACTED — this comment was wrong. Ignore it.

I predicted the 5 Returned null from json records (BH.oM.Structure.SectionProperties.*) might take the signature path (UnresolvableFromEvents) rather than the payload path. They do not.

Measured on BHoM/BHoM run 33849699768, job 101150230634, the first real run with this PR in (runner source e7d6d5c, #18 merged 31 minutes earlier). All five landed:

record path CountedAsReal cause
AluminiumSection UnresolvableTypeAbsent false BH.oM.Adapters.GSA.MaterialFragments.Fabric
CellularSection UnresolvableTypeAbsent false same
GenericSection UnresolvableTypeAbsent false same
SteelSection UnresolvableTypeAbsent false same
TimberSection UnresolvableTypeAbsent false same

So the PR body was right on both outcome and path label, and this comment added doubt where there was none. The reasoning that produced it — that Returned null from json can arrive via the signature shape, as it does for the ETABS/SAP2000 method records on BHoM/Localisation_Toolkit #181 — is true in general but does not apply to these five, which are object records blocked on a nested fragment.

Full post-merge distribution on that run: 114 UnresolvableTypeAbsent, 31 NoMethodEvent, 145 unverified, 0 real, Status: Warning.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant