Adding another test run halves the code coverage

We’ve recently introduced another vstest step as we’re migrating our unit tests to a newer structure. This means that in our analysis build we run one vstest task pointing at one location, followed by another pointing at a different location.

# Old tests
- task: VSTest@2
  displayName: Run Unit Tests
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
     test\**\*.test*.dll
     !**\obj\**
     !test\bin\**
    codeCoverageEnabled: true
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    runInParallel: true
    pathtoCustomTestAdapters: packages
    testFilterCriteria: TestCategory!=L1&TestCategory!=L2&TestCategory!=L3

# New tests
- task: VSTest@2
  displayName: 'Run L0 Tests'
  inputs:
    testAssemblyVer2: '**/*L0.Test.dll'
    searchFolder: 'test/bin/L0'
    testRunTitle: 'L0 tests'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    diagnosticsEnabled: true
    codeCoverageEnabled: true
    runInParallel: true

When this is run through our SonarCloud definition, it reports our code coverage at 7.8%. Removing the second task reports our code coverage at 14.8%.

Code coverage is enabled for both test tasks, so I would be under the impression that our code coverage should increase, not decrease under these circumstances.

Hi,

What do say the logs ? Are all your test results files and coverages ones well parsed during the analysis ?

There aren’t any errors that I can find during parsing of the files.

I found these two log sections though:

INFO: Sensor C# Tests Coverage Report Import [csharp]
INFO: Parsing the Visual Studio coverage XML report C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Adding this code coverage report to the cache for later reuse: C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=3271ms

INFO: Sensor C# Tests Coverage Report Import [csharp]
INFO: Parsing the Visual Studio coverage XML report C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Adding this code coverage report to the cache for later reuse: C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=3271ms

This step is only present once if the second test is not being run. It seems like the scanner is importing the coverage file twice when there are two test runs.

And do you have specific sonar.cs.vstest.reportsPaths or sonar.cs.vscoveragexml.reportsPaths with all the file you need ?

Not that we’ve explicitly configured, so far it’s just picked up what was run through and published from the vstest tasks.

@mickaelcaro did you have any other thoughts or suggestions for this?

Are both unit test tasks target, in some way, the same unit tests ? I guess so given the pattern of the first one.

I know that our old test task is running tests multiple times, but none of the tests in the new structure should be run in the old task.

The old task excludes the output folder for the new tests now:

     !test\bin\**

The new test task only targets that folder now:

    searchFolder: 'test/bin/L0'

Hi @StephenLarkin,

Sorry for the delay.

Could you please post the the verbose output of the command (please run SonarScanner.MSBuild.exe begin /k:“MyProject” /d:sonar.verbose=true as the begin step, and please attach the output of END step )
So as we added recently some more logs around coverage, we might get more help for your issue.

Thanks !

Hi there,

Sorry I haven’t been able to provide you with anything earlier than this, things have been pretty manic.

sonarcloud-log-zipped.txt (618.1 KB)

Okay, so that file is actually a zip file, but I had to rename it to get around the upload restrictions. The original txt file was 5879KB… If you rename it back you should be able to unzip it.

Thank you.

So i see only one coverage file that has been parsed, do you run it with both UT tasks ?

If yes, i suggest to try to get differet coverage file for both tasks, otherwise that should lead to weird behavior (i don’t know however what is the guidelines from MS for such cases)

Mickaël

Yeah, we currently have three separate test runs configured. I’ll look into how the coverage files are generated and consumed by Azure DevOps and report back.