Adding another test run halves the code coverage

We’ve recently introduced another vstest step as we’re migrating our unit tests to a newer structure. This means that in our analysis build we run one vstest task pointing at one location, followed by another pointing at a different location.

# Old tests
- task: VSTest@2
  displayName: Run Unit Tests
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
     test\**\*.test*.dll
     !**\obj\**
     !test\bin\**
    codeCoverageEnabled: true
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    runInParallel: true
    pathtoCustomTestAdapters: packages
    testFilterCriteria: TestCategory!=L1&TestCategory!=L2&TestCategory!=L3

# New tests
- task: VSTest@2
  displayName: 'Run L0 Tests'
  inputs:
    testAssemblyVer2: '**/*L0.Test.dll'
    searchFolder: 'test/bin/L0'
    testRunTitle: 'L0 tests'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    diagnosticsEnabled: true
    codeCoverageEnabled: true
    runInParallel: true

When this is run through our SonarCloud definition, it reports our code coverage at 7.8%. Removing the second task reports our code coverage at 14.8%.

Code coverage is enabled for both test tasks, so I would be under the impression that our code coverage should increase, not decrease under these circumstances.

Hi,

What do say the logs ? Are all your test results files and coverages ones well parsed during the analysis ?

There aren’t any errors that I can find during parsing of the files.

I found these two log sections though:

INFO: Sensor C# Tests Coverage Report Import [csharp]
INFO: Parsing the Visual Studio coverage XML report C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Adding this code coverage report to the cache for later reuse: C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=3271ms

INFO: Sensor C# Tests Coverage Report Import [csharp]
INFO: Parsing the Visual Studio coverage XML report C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Adding this code coverage report to the cache for later reuse: C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=3271ms

This step is only present once if the second test is not being run. It seems like the scanner is importing the coverage file twice when there are two test runs.

And do you have specific sonar.cs.vstest.reportsPaths or sonar.cs.vscoveragexml.reportsPaths with all the file you need ?

Not that we’ve explicitly configured, so far it’s just picked up what was run through and published from the vstest tasks.

@mickaelcaro did you have any other thoughts or suggestions for this?

Are both unit test tasks target, in some way, the same unit tests ? I guess so given the pattern of the first one.

I know that our old test task is running tests multiple times, but none of the tests in the new structure should be run in the old task.

The old task excludes the output folder for the new tests now:

     !test\bin\**

The new test task only targets that folder now:

    searchFolder: 'test/bin/L0'