ReusesCachedOutputOnRerun always passes, even for a non-incremental generator, because both runs use the same Compilation

[matt-edmondson]

What's wrong

GeneratorHarness.ReusesCachedOutputOnRerun (SourceGeneratorToolkit.Testing/GeneratorHarness.cs:~102-127) runs the driver twice against the same CSharpCompilation instance, with the same AdditionalText instances. Roslyn compares incremental inputs by reference, so on the second run every node, CompilationProvider included, is reported as Cached/Unchanged whatever the generator does.

The doc comment says the check exists to catch "a generator that recomputes everything on every keystroke". A keystroke produces a new Compilation, and this check never creates one, so it cannot detect that. TheGeneratorReusesItsOutputWhenNothingChanged (GeneratorBaseTests.cs:203) therefore asserts nothing meaningful.

Reproduction (ran)

The generator below is non-incremental by construction:

context.RegisterSourceOutput(
    context.CompilationProvider.Select((c, _) => new object()),
    (spc, _) => spc.AddSource("x.g.cs", "// x"));

• ReusesCachedOutputOnRerun(generator) returns true. Expected false.
• For comparison, running it on c1 and then on c1.AddSyntaxTrees(CSharpSyntaxTree.ParseText("")) reports the output step as Modified.

Why it matters

This is a test helper that consumers rely on to prove their generators are incremental. A check that always passes gives false confidence and lets IDE-performance regressions ship.

Suggested fix / acceptance criteria

• Run the second pass on a new but semantically equivalent compilation, for example compilation.AddSyntaxTrees(CSharpSyntaxTree.ParseText(string.Empty)) or compilation.WithAssemblyName(compilation.AssemblyName). Ideally also rebuild the AdditionalText instances with identical content, as an IDE does.
• Treat "no tracked output steps" as not proven (false, or throw) rather than true.
• Add a negative test: the CompilationProvider.Select(_ => new object()) generator above must fail the check.

[matt-edmondson]

Triage

• Category: Bug
• Priority: High. ReusesCachedOutputOnRerun is the helper consumers use to prove a generator is incremental, and it always returns true. A check that can't fail gives false confidence and lets IDE-performance regressions ship in downstream generators. The toolkit's own test TheGeneratorReusesItsOutputWhenNothingChanged currently proves nothing.
• Area / suggested assignee: GeneratorHarness in SourceGeneratorToolkit.Testing. Maintainer: @matt-edmondson.
• Duplicates / in progress: No duplicate and no open PR. Open PR Bump the ktsu group with 10 updates #18 is a dependency bump that doesn't touch this code.
• Next steps: Run the second pass on a new, semantically equivalent Compilation (for example AddSyntaxTrees(ParseText(""))), and rebuild the AdditionalText instances. Return false when no output steps are tracked. Add the negative test from the issue (CompilationProvider.Select(_ => new object()) must fail). Some existing consumer tests may start failing once this lands; that exposes real non-incremental generators, so call it out in the release notes.

---

Generated by Claude Code