host-testing.md

MTP host testing

RevitDevTool runs tests inside Revit, AutoCAD, and Civil 3D through Microsoft.Testing.Platform. RevitDevTool.TestAdapter is the only public test integration package. NUnit is the default engine. TUnit (tunit-host-testing.md) and MSTest (0038) are opt-in providers. The VSTest adapter and NUnit-specific bridge protocol are not part of the supported product on develop; their final baseline is retained on branch testing/nunit-vstest.

Test project contract

• Reference RevitDevTool.TestAdapter (depends on Microsoft.Testing.Platform.MSBuild 2.5.1). Framework package (default NUnit 5.0.0) is a local choice, not a package dependency. Package props set OutputType=Exe. Declare HostName, HostVersion, optional ForceLaunch, PerTestTimeout, LaunchTimeout. ForceLaunch=true always starts a new host (skip reuse). PerTestTimeout is the per-test budget after the host is ready; testing/run pipe wait = budget × test count. LaunchTimeout waits for the host pipe after process start. TestingFramework (nunit default, tunit or mstest opt-in) overrides the in-host engine without changing the NuGet. An mstest project references MSTest.TestFramework 4.5.1. The in-host closure is that version plus Microsoft.Testing.Platform 2.5.1, private to MSTestRuntime\. An mstest project uses Sdk="MSTest.Sdk", or Microsoft.NET.Sdk with <EnableMSTestRunner>true</EnableMSTestRunner> (the build fails with that hint otherwise). The adapter testhost package is the same 2.5.1.
• testconfig.json is generated from csproj properties; incremental build refreshes [AssemblyName].testconfig.json (no Rebuild after HostName, ForceLaunch, PerTestTimeout, or LaunchTimeout changes). Microsoft.Testing.Platform.MSBuild copies the same file. Adapter reads devtools via MTP IConfiguration (same as mstest / xUnit). Author testconfig.json beside the csproj for platformOptions; no .runsettings.
• dotnet test needs "test": { "runner": "Microsoft.Testing.Platform" } in global.json. Put that runner on the repo root when every dotnet test project in the tree is MTP, or when the only VSTest leftover is isolated with a nested "runner": "VSTest" (this repo: samples/ricaun.NUnit.SampleTests). Scope MTP next to the test project only when many VSTest/dotnet test projects still share the tree. Nearest global.json replaces the whole file (include sdk in the nested file). dotnet test from a folder uses that cwd's file, not the --project path.
• Do not set <RuntimeIdentifier> on the test project. The package uses OutputType=Exe. A RID on net8 / net10 (Revit 2025+) nests testhost output under win-x64, which Test Explorer can bind instead of the current build. The package sets AppendRuntimeIdentifierToOutputPath=false so a leftover RID stays flattened. net48 TUnit needs nothing else — the package supplies the [ModuleInitializer] attribute TUnit generates against. Details: tunit-host-testing.md.
• Packaged modern TFMs copy private build/runtime: exact net{version}-windows7.0 when present, else nearest lower shipped folder (net9 → net8.0-windows7.0). A consumer TFM with no match fails the test project build instead of copying nothing.

Discovery

• Host-free IDE discovery. NUnit uses DevTools.NUnit.MTP + ExploreTests. TUnit lists Sources.TestEntries. MSTest lists with TestApplication --list-tests and does not run test bodies. Host API refs (Revit_All_Main_Versions_API_x64 for Revit) are compile-only (Copy Local false). Build writes $(TargetName).discovery-refs.txt from compile-only NuGet ReferencePath; testhost resolves via AssemblyResolve (API DLLs not beside the exe). With that file, discovery loads an isolated copy of the test assembly and resolves those paths — no host. Parameter/return metadata for TestMethodIdentifier is omitted when those refs cannot load (ref/ NuGet path). That must not abort the leaf list. If ExploreTests cannot build a tree, discovery fails with that NUnit reason; no PE metadata list.
• NUnitAssemblyBuilder uses types that did load instead of marking the whole assembly NotRunnable (testhost uses the same builder → assembly-level attributes and sort order match the host). Sets TestContext.WorkDirectory (generation shadow); early WorkDirectory access throws. net48 has no load context: same-identity product assemblies bind to the host copy, not the generation snapshot — restart the host after deploying a matching add-in, or use net8+ (AssemblyLoadContext).

Identity and filters

• Protocol split: platform TestNode (IDE) vs testing/* (host). NUnit owns discovery tree, identity, filter XML, skip/explicit, parameterized-case naming.
• UID / wire TestId / host <test>: NUnit ITest.FullName. TestNode uid matches except [TestCase(TestName=)] / SetName → Class.Method("DisplayName") (avoids indexing Class.Unit_X beside Named_basis_length_is_one). TestMethodIdentifierProperty.TypeName: C# source type, no namespace, no fixture ctor args, no closed generic args, no ECMA-335 backtick arity. Visual Studio binds syntax: GenericClosedTests`1 and GenericClosedTests<Int32> → "No source available". PDB uses metadata names (GenericClosedTests1, nested +); closed generic args stay on uid / FullName. Display types in TypeNamebreak IDE hierarchy on.in ctor args.TestFileLocationProperty` is the line-accurate fallback when the identifier bind is enough to open the file.
• Filter: UID / --filter-uid / Test Explorer send the TestNode uid. NotRunnable stub Class.Method when [TestFixtureSource] / [TestCaseSource] cannot expand (Revit types at load) keeps that uid. Host filter matches Class("args").Method and TestName / SetName children (ITest.FullName); results fold back onto the requested UID and onto discovered leaf UIDs. Unfiltered runs (no --filter / --filter-uid) remap FullName onto discovered uids; names-only --filter keeps per-leaf host identities; (args) ids stay exact <test> (plus a second <test> with NUnit 4 numeric suffixes d/f/m/L stripped, so testhost -12.3d matches in-host -12.3). Discovery also strips those suffixes from the TestNode uid so Rider keys one leaf per case. Inbound IDE uids resolve through an identity index of strings this testhost published (TestId, FullName, DisplayName-with-args, Ns.Type.Method, Class.Method()). Grouping uids expand to leaves; results publish only those leaf uids. Discover with an empty UID list is discover-all (IDE refresh). The same empty list on a run stays constrained (publish nothing). TreeNodeFilter (--treenode-filter, /ns/type/method) selects those leaves — it does not fall through to the whole assembly. Result nodes reuse discovered TestMethodIdentifier (C# method name). --filter / Name= → Names / <name re="1">. --filter-uid: json TestNode uid (ITest.FullName, or Class.Method("DisplayName") for TestName / SetName). PowerShell: quote " in uids (--filter-uid 'Ns.Class.Method("Unit_X")'). Rider MTP uses --server and TestAncestorCriterion: click on a method group sends Class.Method or Class.Method() (empty parens), not the five (args) leaves. Execution expands to discovered leaves; results publish only those leaf uids. Unreported Failed covers missing leaves, not the grouping uid. Unreported uids publish as Failed (same identity) instead of dropped. MTP TestFrameworkCapabilities stay empty (no VSTest-bridge extras).

Run

• Host launch/reuse only after execution request (testing/hello, testing/run, testing/cancel). Cancel or launch timeout kills only a host this run spawned (not a reused instance). Visual Studio Stop Debugging during that wait cancels when the testhost PID (--debug-parent-pid) exits, then kills the in-flight host. Cancel while a breakpoint is hit in the reused host does not free the idle thread until you Continue (or detach). The next testing/hello starts a new session; it does not inherit a poisoned cancel from the dropped client.
• Visual Studio Test Explorer Debug attaches the testhost; Runner then EnvDTE-attaches that Visual Studio instance to the Autodesk host (0025). Rider and C# Dev Kit attach the host PID and Run (attach does not block test execute). VS Code/forks and PyCharm attach Python via debugpy port 5678 (.vscode/launch.json, .run/Attach.run.xml).

Ownership

Module	Responsibility
DevTools.Testing.Abstractions	Neutral run/result/runtime contracts, plus testhost discovery (ITestDiscoverer). MTP compiles against this assembly, not DevTools.TestAdapter
DevTools.Testing.Transport	testing/* JSON, pipe methods, and TestRunner process client
DevTools.Testing.Host	In-host testing/* handler, generation store, runtime-session lifecycle, and first-party NUnit/TUnit/MSTest providers (closure/version policy, Dynamo-safe NUnit framework sharing, isolated runtime activation, the testhost run mapper's selection → NUnit filter-xml or MSTest --filter-uid)
DevTools.TestAdapter	Published RevitDevTool.TestAdapter. MTP control plane (command line, host launch request, TestNode publish). References the selected DevTools.{NUnit|TUnit|MSTest}.MTP.dll as a testhost assembly. Does not parse NUnit names
DevTools.NUnit.MTP	Authoritative local discovery (NUnitTestAssemblyRunner + ExploreTests), metadata TypeName, DisplayName suffix, host filter XML, and result fold. Build-selected testhost sibling; not ILRepacked into the adapter
DevTools.NUnit.Runtime	Default in-host engine: NUnit execution inside an isolated generation
DevTools.MSTest.MTP	Testhost discovery via TestApplication --list-tests. Does not run test bodies. Maps Names to MTP node-uid test ids
DevTools.MSTest.Runtime	In-host MSTest 4.5.1 session on the assembly the generation already loaded. mstest:output:captureTrace stays off
DevTools.TestRunner.Core	Framework-neutral host locate/launch/reuse, debugger attach, and testing/* pipe client
DevTools.TestRunner	Southbound executable: locate/launch the host and send testing/run. Framework id is a CLI option from the adapter devtools section

The cross-load-context identity is DevTools.Testing.Abstractions. Runtime payloads do not carry a provider-specific transport assembly. Supported runtime targets are net48, net8, and net10; the former netstandard2.0 compatibility target is removed.

Samples and verification

Samples: samples/DevTools.NUnit.SampleTests (Revit), samples/DevTools.NUnit.Civil3D.SampleTests (Civil 3D). Those samples still use NUnit attributes because NUnit is the default engine. samples/DevTools.TUnit.SampleTests and samples/DevTools.TUnit.Civil3D.SampleTests opt in with TestingFramework=tunit. samples/DevTools.MSTest.SampleTests and samples/DevTools.MSTest.Civil3D.SampleTests opt in with TestingFramework=mstest and MSTest.Sdk 4.5.1. samples/ricaun.NUnit.SampleTests is a comparison sample: it links the same HostSmokeTests and runs them through ricaun.RevitTest.TestAdapter (VSTest). It is not the product contract — do not use it as the verify path, and do not try to make it MTP.

Run the generated test executable or use the Microsoft.Testing.Platform dotnet test/IDE surface. This repo puts the MTP runner on the root global.json; samples/ricaun.NUnit.SampleTests overrides with "runner": "VSTest" and must be run from that folder. The adapter copies the selected sibling (DevTools.NUnit.MTP.dll, DevTools.TUnit.MTP.dll, or DevTools.MSTest.MTP.dll) next to the test exe. Consumers reference the framework package; they do not add that sibling as a ProjectReference.

Use an Autodesk configuration (Debug.Autodesk.2024, Release.Autodesk.2024, …). Plain Debug / Release do not set RevitVersion / TargetFramework; the sample does not build, and Test Explorer then shows a source/method tree that is not MTP ExploreTests.

Canonical samples/DevTools.NUnit.SampleTests discovery: test exe --list-tests json leaf count. Measured 71 for Debug.Autodesk.2025 (same UIDs on other Autodesk years):

Fixture	Leaves
BoundingBoxXyzSampleTests	26
ValueSourceTests	21
TestCaseTests	8
LifecycleTests	4
HostSmokeTests	3
NamedFixtureSourceTests("alpha.rvt"|"beta.rvt")	2
InheritedGeometryTests, Nested+Inner, GenericClosedTests<Int32>	1 each
InheritedGeometryTests, Nested+Inner, GenericClosedTests<Int32>, SourceAssetPathTests	1 each
GenericRevitTypeTests<XYZ|BoundingBoxXYZ>	0 (not ExploreTests)

Test Explorer counts are not that leaf count. Visual Studio Real-Time Discovery adds [TestCase(TestName=…)] children that never match MTP uids (TestCaseTests: 8 + 3 = 11 under Named_basis_length_is_one; .SetName cases do not get those extras). Turn off Tools → Options → Test → Discover tests in real time from C# and Visual Basic .NET source files, then refresh. NUnit documents the same gap (nunit3-vs-adapter#1256, #489); adapter code cannot dedupe RTD nodes. A ~32-node tree (methods, GenericRevitTypeTests, TestName leaves as extra methods) is grouping/source discovery, not the 71-leaf CLI list; runs from it do not send expanded FullName UIDs.

Packaging

Two artifacts. Installer workflow does not publish NuGet; adapter workflow does not pack the host bundle.

• NuGet RevitDevTool.TestAdapter — dotnet run --project build -- adapter / PublishTestAdapter.yml. Version is <Version> in source/DevTools.TestAdapter/DevTools.TestAdapter.csproj. Modern targets keep implementation assemblies in a private build/runtime closure. net48 ILRepacks the adapter except DevTools.Testing.Abstractions.dll, which stays beside the test exe (with DevTools.NUnit.MTP.dll) so testhost discovery shares one ITestDiscoverer / TestingDiscovery identity. Consumers see only the platform adapter compile surface.
• Installer / bundle — dotnet run --project build -- pack / PublishRelease.yml. Ships DevTools.TestRunner.exe and the in-host testing stack (Testing.Host with NUnit/TUnit/MSTest providers, NUnit.Runtime / TUnit.Runtime / MSTest.Runtime). Required for live runs; the NuGet does not replace it.

Pack graph and restore constraints: architecture/Testing. Platform-only boundary: 0022. Kernel split: 0021.