fix(ci-versioning): classify a record the closure cannot resolve as unverified - #18
Conversation
…nverified
ClassifyUnresolvableCause already computed the right answer and then discarded it
one line later:
if (namedTypes.Any(t => t.StartsWith("BH.", StringComparison.Ordinal)))
return null;
Its intent was "could CI ever load this", and for RevitAPI the proxy held. It
cannot hold generally, because every downstream BHoM toolkit type also starts
with BH.: measured on BHoM/BHoM run 33849699768, all 145 findings named a BH.
type and all 145 were closure artefacts. That is the same recorded-then-
discarded fault as internal-tickets#31, now its fifth instance.
Replace the name test with the one the runner can actually answer, keyed on the
loaded set rather than the namespace:
present in the loaded set -> not a blocker
absent, but its namespace IS loaded -> an assembly here owns that namespace
and the type is gone. A REMOVAL. Real.
absent, and its namespace too -> nothing here could have supplied it.
Unverifiable.
The middle branch is the removal fail-safe and is why this is not simply
"absent means unverified", which would have blinded the check to undeclared
removals — the thing it exists to catch. Where a record names both, the removal
wins; the removed type has its own record besides.
Also parse the two payload event shapes the deserialiser emits at any nesting
depth ("Failed to convert the string into a type: X", "The type X from version
V is unknown -> ..."). Only the signature shape was parsed before, which is why
every object-record failure reached the caller with no cause at all.
Recorded as UnresolvableTypeAbsent, distinct from UnresolvableFromEvents
(signature path) and from NoMethodEvent (#16's attribution bucket), so the
artefact and the summary separate "could not resolve it" from "could not
attribute it". Keyed on which message shape produced the cause, not on whether
a Method event is present, because a signature blocker does not always have one.
Coverage gains RecordsUnverified, printed on the Coverage line even when zero:
1191 subject types checked reads as a clean sweep until you know 145 records
were not verified.
#16's attribution policy and the namespace fallback are untouched. The two
compose rather than overlap: measured locally on 98 findings, resolution catches
66 and attribution the remaining 32, total 0 real.
Verified against real artefacts, not only Pester. Local public-installer
closure, 186 assemblies in the runner's index: 98 findings -> 0 real, 98
unverified (66 UnresolvableTypeAbsent + 32 NoMethodEvent), Status Warning,
exit 0. Same run with a 56-assembly index: 98 -> 0 real, 70 + 28. The 32 and 28
are types in namespaces BHoM/BHoM co-inhabits with StructuralEngineering_Toolkit
(BH.oM.Structure.{Elements,Loads,Results}), so the removal fail-safe correctly
declines and #16 catches them instead. That residue is what provenance (#35)
removes.
Suite 132 passed. Verified by mutation: deleting the removal fail-safe fails
TypeMissingFromALoadedNamespace_IsReal and ARemovalAlongsideAClosureGap_StaysReal
and nothing else. PowerShell suite 72 passed with the same 15 pre-existing local
environment failures as develop.
Refs BHoM/internal-tickets#32, BHoM/internal-tickets#31
|
RETRACTED — this comment was wrong. Ignore it. I predicted the 5 Measured on
So the PR body was right on both outcome and path label, and this comment added doubt where there was none. The reasoning that produced it — that Full post-merge distribution on that run: 114 |
What
A record the closure cannot resolve is classified unverified, not real. It keeps its diagnostic, appears in the artefact, and does not gate. A record whose types all resolve and which still fails is a genuine regression and still gates.
#16's attribution policy and the namespace fallback are untouched. The two compose rather than overlap.
The line
ClassifyUnresolvableCausealready computed the right answer and threw it away one line later:The intent was "could CI ever load this", and for
RevitAPIthe proxy held. It cannot hold generally: every downstream BHoM toolkit type also starts withBH.. Measured onBHoM/BHoMrun33849699768, all 145 findings named aBH.type and all 145 were closure artefacts. Fifth instance of the recorded-then-discarded fault in BHoM/internal-tickets#31.The replacement
Keyed on the loaded set rather than the name:
The middle branch is why this is not simply "absent means unverified." That would have classified an undeclared removal as unverified and blinded the check to the thing it exists to catch. Where a record names both a removal and a closure gap the removal wins, and the removed type has its own record besides.
Also parses the two payload event shapes the deserialiser emits at any nesting depth —
Failed to convert the string into a type: XandThe type X from version V is unknown -> .... Only the signature shape was parsed before, which is why every object-record failure reached the caller with no cause at all.Separability
Recorded as
UnresolvableTypeAbsent, distinct fromUnresolvableFromEvents(signature path) andNoMethodEvent(#16's attribution bucket), keyed on which message shape produced the cause rather than on whether a Method event is present — a signature blocker does not always have one. The job summary now breaks unverified down the same way:Coverage
CoverageCountsgainsRecordsUnverified, printed on the Coverage line even when zero.1191 subject type(s) checkedreads as a clean sweep until you know 145 records were not verified.Verification against real artefacts
Local public-installer closure, 186 assemblies in the runner's index:
Same test result classified against a 56-assembly index: 98 → 0 real, 70 + 28. The 32 and 28 are types in namespaces
BHoM/BHoMco-inhabits withStructuralEngineering_Toolkit(BH.oM.Structure.{Elements,Loads,Results}), so the removal fail-safe correctly declines and #16 catches them instead. That residue is exactly what provenance (#35) removes, and it is why the two changes are ordered rather than alternatives.Not verified locally, and stated rather than assumed: the CI 145-finding case.
FromJsonDatasetcallsLoadAllAssemblies(), which reads%ProgramData%\BHoM\Assembliesregardless of the runner's--assembliesargument, so the test's closure cannot be reduced locally without mutating the machine's BHoM install. Locally the test always produces its 98-finding result and only the runner's index can be varied. The 47 CI-only findings are the GSA/Sharepoint-poisoned ones, whose blocking namespaces are absent in CI's closure, so they are predicted to land inUnresolvableTypeAbsent— including the 5Returned null from jsonrecords, which are in that 47 and therefore also unobservable locally. First real run onBHoM/BHoMconfirms or refutes it.Suite 132 passed. Verified by mutation: deleting the removal fail-safe fails
TypeMissingFromALoadedNamespace_IsRealandARemovalAlongsideAClosureGap_StaysReal, and nothing else. PowerShell suite 72 passed with the same 15 pre-existing local environment failures asdevelop.Refs BHoM/internal-tickets#32, BHoM/internal-tickets#31