Second time coverage gives 0% coverage

I noticed a peculiar behavior; I fire up sbt from the shell and the first time I collect the coverage with clean;coverage;test;coverageAggregate I get about 37% coverage on my example project but if I run it again immediately after I get 0% and also for all consecutive runs. I obviously expected to get the same coverage every time.

If I exit sbt and run coverage again I get the same behavior; first run gives correct coverage but consecutive runs give 0%.

My test project can be found here: https://github.com/olatheander/scala-code-coverage-test

It seems to be a Scala 3 issue, I created an equivalent Scala 2 project and then it worked as expected.

Below is the output of the first run followed by a second run:

sbt:Scala Code Coverage Test> clean;coverage;test;coverageAggregate
[success] Total time: 0 s, completed 18 nov. 2023 13:38:41
[info] Defining ThisBuild / coverageEnabled
[info] The new value will be used by Compile / compile / scalacOptions, libraryDependencies
[info] Reapplying settings...
[info] set current project to Scala Code Coverage Test (in build file:/home/olathe/Git/scala-code-coverage-test/)
[info] compiling 1 Scala source to /home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/classes ...
[info] compiling 1 Scala source to /home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/test-classes ...
com.example.MySuite:
  + Person test 0.036s
  + Person test 2 0.0s
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2
[success] Total time: 4 s, completed 18 nov. 2023 13:38:46
[info] Aggregating coverage from subprojects...
[info] Found 1 subproject scoverage data directories [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-data]
[info] Generating scoverage reports...
[info] Written Cobertura report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/coverage-report/cobertura.xml]
[info] Written XML coverage report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-report/scoverage.xml]
[info] Written HTML coverage report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-report/index.html]
[info] Statement coverage.: 37.50%
[info] Branch coverage....: 100.00%
[info] Coverage reports completed
[info] Aggregation complete. Coverage was [37.50]
[error] Coverage is below minimum [37.50% < 70.00%]: Stmt:Total
[info] All done. Coverage was stmt=[37.50%] branch=[100.00%]
[success] Total time: 0 s, completed 18 nov. 2023 13:38:46
sbt:Scala Code Coverage Test> clean;coverage;test;coverageAggregate
[success] Total time: 0 s, completed 18 nov. 2023 13:38:48
[info] Defining ThisBuild / coverageEnabled
[info] The new value will be used by Compile / compile / scalacOptions, libraryDependencies
[info] Reapplying settings...
[info] set current project to Scala Code Coverage Test (in build file:/home/olathe/Git/scala-code-coverage-test/)
[info] compiling 1 Scala source to /home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/classes ...
[info] compiling 1 Scala source to /home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/test-classes ...
com.example.MySuite:
  + Person test 0.004s
  + Person test 2 0.0s
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2
[success] Total time: 1 s, completed 18 nov. 2023 13:38:49
[info] Aggregating coverage from subprojects...
[info] Found 1 subproject scoverage data directories [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-data]
[info] Generating scoverage reports...
[info] Written Cobertura report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/coverage-report/cobertura.xml]
[info] Written XML coverage report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-report/scoverage.xml]
[info] Written HTML coverage report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-report/index.html]
[info] Statement coverage.: 0.00%
[info] Branch coverage....: 0.00%
[info] Coverage reports completed
[info] Aggregation complete. Coverage was [0.00]
[error] Coverage is below minimum [0.00% < 70.00%]: Branch:Total
[info] All done. Coverage was stmt=[0.00%] branch=[0.00%]
[success] Total time: 0 s, completed 18 nov. 2023 13:38:49
sbt:Scala Code Coverage Test>

Scala version: 3.3.1
sbt-coverage version: 2.0.9

After investigating a bit more; after doing a coverage;test after sbt startup there is scoverage.measurements.**** file created e.g:

$ cd target/scala-3.3.1
$ tree scoverage-data/                                                                                                                                                                                                                                                                                                                           scoverage-data/
├── scoverage.coverage
└── scoverage.measurements.58cb6c3d-238f-4782-90d9-f698d1b8bf19.187

but after a clean and consecutive coverage;test this file is not created anymore, only the scoverage.coverage file is recreated. Without the measurements file, the coverage is reported to be 0%.

I verified this by hard-coding the measurements value in loadCoverage in which I added a log statement to collect the values:

  private def loadCoverage(
      crossTarget: File,
      log: Logger,
      sourceRoot: File
  ): Option[Coverage] = {
     ...
     ..
      log.info(s"Reading scoverage measurements...")
      val measurementFiles = IOUtils.findMeasurementFiles(dataDir)
      // val measurements = IOUtils.invoked(measurementFiles)
      val measurements = Set((0, ""), (2, ""), (1, ""))
      log.info(s"measurements: ${measurements}")
      coverage.apply(measurements)
      Some(coverage)

    } else {
      None
    }
  }

With the hard-coded Set() the reported coverage was 37.50% every time despite the measurements file was missing (obviously since I hard-coded the values).

However, this makes me believe that perhaps it's more of a dotty issue than a sbt-coverage issue?

I also have seen this issue with a recent little project trying with Scala 3.3.3, ZIO 2.0.19, and Scoverage 2.1.0.

On initial setup and run I was getting accurate coverage numbers, but after trying to add some exclusions I began to repeatedly get 0% coverage results. Reverted to original scoverage changeset, and still getting 0%.

I also encountered this issue

Scala 3.7.4, sbt 1.13.0, sbt-scoverage 2.4.4

Looks like this might work with Test / fork = true, seems to be caused by https://github.com/scala/scala3/blob/main/library/src/scala/runtime/coverage/Invoker.scala

We might need to fix it in the compiler.

Potential fix would be #691 but not keen on that. Until we fix it in the compiler setting fork to true as a workaround is probably the way.

Ok, seems it will be fixed by scala/scala3#27087 but using Test / fork = true or Test / classLoaderLayeringStrategy := ClassLoaderLayeringStrategy.Flat should help as well.

Navigation Menu

Repository navigation

Second time coverage gives 0% coverage #522

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Second time coverage gives 0% coverage #522

Description

Activity

olatheander commented on Nov 18, 2023

dillius commented on Jul 27, 2024

mark-kar commented on Sep 9, 2026

tgodzik commented on Sep 15, 2026

tgodzik commented on Sep 15, 2026

tgodzik commented on Sep 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions