Adding another test run halves the code coverage

StephenLarkin · October 4, 2019, 6:10pm

We’ve recently introduced another vstest step as we’re migrating our unit tests to a newer structure. This means that in our analysis build we run one vstest task pointing at one location, followed by another pointing at a different location.

# Old tests
- task: VSTest@2
  displayName: Run Unit Tests
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
     test\**\*.test*.dll
     !**\obj\**
     !test\bin\**
    codeCoverageEnabled: true
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    runInParallel: true
    pathtoCustomTestAdapters: packages
    testFilterCriteria: TestCategory!=L1&TestCategory!=L2&TestCategory!=L3

# New tests
- task: VSTest@2
  displayName: 'Run L0 Tests'
  inputs:
    testAssemblyVer2: '**/*L0.Test.dll'
    searchFolder: 'test/bin/L0'
    testRunTitle: 'L0 tests'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    diagnosticsEnabled: true
    codeCoverageEnabled: true
    runInParallel: true

When this is run through our SonarCloud definition, it reports our code coverage at 7.8%. Removing the second task reports our code coverage at 14.8%.

Code coverage is enabled for both test tasks, so I would be under the impression that our code coverage should increase, not decrease under these circumstances.

mickaelcaro · October 9, 2019, 6:45am

Hi,

What do say the logs ? Are all your test results files and coverages ones well parsed during the analysis ?

StephenLarkin · October 9, 2019, 10:01am

There aren’t any errors that I can find during parsing of the files.

I found these two log sections though:

INFO: Sensor C# Tests Coverage Report Import [csharp]
INFO: Parsing the Visual Studio coverage XML report C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Adding this code coverage report to the cache for later reuse: C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=3271ms

INFO: Sensor C# Tests Coverage Report Import [csharp]
INFO: Parsing the Visual Studio coverage XML report C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Adding this code coverage report to the cache for later reuse: C:\agent\_work\_temp\TestResults\00cf2540-1ebd-4647-b107-c190a451d624\[user_agent] 2019-10-04 13_06_34.coveragexml
INFO: Sensor C# Tests Coverage Report Import [csharp] (done) | time=3271ms

This step is only present once if the second test is not being run. It seems like the scanner is importing the coverage file twice when there are two test runs.

mickaelcaro · October 9, 2019, 11:33am

And do you have specific sonar.cs.vstest.reportsPaths or sonar.cs.vscoveragexml.reportsPaths with all the file you need ?

StephenLarkin · October 9, 2019, 2:18pm

Not that we’ve explicitly configured, so far it’s just picked up what was run through and published from the vstest tasks.

StephenLarkin · October 15, 2019, 11:19am

@mickaelcaro did you have any other thoughts or suggestions for this?

mickaelcaro · October 15, 2019, 11:36am

Are both unit test tasks target, in some way, the same unit tests ? I guess so given the pattern of the first one.

StephenLarkin · October 15, 2019, 1:54pm

I know that our old test task is running tests multiple times, but none of the tests in the new structure should be run in the old task.

The old task excludes the output folder for the new tests now:

     !test\bin\**

The new test task only targets that folder now:

    searchFolder: 'test/bin/L0'

mickaelcaro · October 31, 2019, 2:05pm

Hi @StephenLarkin,

Sorry for the delay.

Could you please post the the verbose output of the command (please run SonarScanner.MSBuild.exe begin /k:“MyProject” /d:sonar.verbose=true as the begin step, and please attach the output of END step )
So as we added recently some more logs around coverage, we might get more help for your issue.

Thanks !

StephenLarkin · November 6, 2019, 7:02pm

Hi there,

Sorry I haven’t been able to provide you with anything earlier than this, things have been pretty manic.

sonarcloud-log-zipped.txt (618.1 KB)

Okay, so that file is actually a zip file, but I had to rename it to get around the upload restrictions. The original txt file was 5879KB… If you rename it back you should be able to unzip it.

mickaelcaro · November 7, 2019, 7:05am

Thank you.

So i see only one coverage file that has been parsed, do you run it with both UT tasks ?

If yes, i suggest to try to get differet coverage file for both tasks, otherwise that should lead to weird behavior (i don’t know however what is the guidelines from MS for such cases)

Mickaël

StephenLarkin · November 7, 2019, 3:54pm

Yeah, we currently have three separate test runs configured. I’ll look into how the coverage files are generated and consumed by Azure DevOps and report back.

czurn · December 20, 2019, 2:47pm

Hello @StephenLarkin,

Were you able to fix this issue on your side or are you still having troubles regarding this?

Best,
-Chris

StephenLarkin · March 2, 2020, 7:20pm

Hey, sorry for being late getting back to you about this. We have since finished moving our test projects around so that we run our .NET Core tests and our .NET Framework tests in two separate VSTest tasks during our SonarCloud analysis job.

We are still seeing some inconsistency with the coverage statistics that are generated from Azure DevOps and what is tracked by SonarCloud. An example of this can be seen below.

Azure DevOps (Downloaded from the code coverage tab saved against the build):

Sonar Cloud:

Andrei_Epure · March 10, 2020, 12:43pm

thanks @StephenLarkin

could you please share with us a reproducer project? It will help us investigate further.

we have an open issue on a similar problem (https://github.com/SonarSource/sonar-dotnet/issues/3191) and any help with more reproducers will help us clarify and pinpoint the root cause

Thanks!

StephenLarkin · March 11, 2020, 2:03pm

Hey, unfortunately our project is private, but I’ve found that this doesn’t get covered for some reason:

using System;

namespace ExampleTest
{
	public static class FnTest
	{
		public static bool Exact(string input1, string input2)
		{
			if (String.IsNullOrEmpty(input1) && String.IsNullOrEmpty(input2))
				return true;
			return input1 == input2;
		}
	}
}

using NUnit.Framework;
using ExampleTest;

namespace TestNs.L0
{
	public class FnTestTest
	{
		[TestCase("", "", ExpectedResult = true, Description = "Blank values")]
		[TestCase("ABC", "ABC", ExpectedResult = true, Description = "Equal")]
		[TestCase("ABC", "ABc", ExpectedResult = false, Description = "Different case")]
		[TestCase("ABC", "CBA", ExpectedResult = false, Description = "Not equal")]
		[TestCase(" ", " ", ExpectedResult = true, Description = "Whitespace only, same")]
		[TestCase(" ", "  ", ExpectedResult = false, Description = "Whitespace only, different")]
		[TestCase("ABC", "ABC ", ExpectedResult = false, Description = "Whitespace difference")]
		public bool Exact(string input1, string input2) => FnTest.Exact(input1, input2);
	}
}

We’re using a .NET Core test project to test a .NET Standard project if that makes a difference too.

StephenLarkin · March 13, 2020, 12:56pm

@Andrei_Epure Was that enough to reproduce or do you need anything else?

Andrei_Epure · March 17, 2020, 10:27am

thanks @StephenLarkin , I didn’t get time to look at it. For milestone 8.6 we focus on improving the code coverage features so I’ll book time to look at your reproducer and come back to you if I need more details

StephenLarkin · April 1, 2020, 10:10am

Hey, I was just wondering how this was going?

I keep getting questions from other developers about why their quality gates aren’t passing even though they’ve written tests, and it would be nice to give them a timeline about when they can expect a fix…

Andrei_Epure · April 1, 2020, 1:32pm

I didn’t get time for the investigation in this sprint, unfortunately, @StephenLarkin.

However, we’ve made improvements in how we parse the VSTest code coverage files, and now if there are multiple coverage entries for a single line, we will consider them. We released it today and it should be deployed to in one of the next working days.

I am not sure, from the snippet you gave that our improvements will fix your problem as a side effect.

To move forward with this, I would need to see the generated code coverage XML for the method that doesn’t get covered (well, ideally it would be the full coverage xml for that assembly… you can replace sensitive strings in the file to avoid leaking namespaces etc). You can send it in a private message on this forum or to andrei.epure at sonarsource dot com via email.

We have seen a similar issue ( #3191) with yield operators - in that case, VSTest doesn’t generate the coverage information. We keep this issue open to investigate more, make a standalone reproducer project and open a ticket on the VSTest side.

To sum up - please share with us the coverage.xml file produced by the VS coverage tool and we’ll check if there’s any entry for the method and if there’s something particular about the coverage file.

Another thing would be to try and use OpenCover and see if the behavior persists. You don’t need to change anything in the tests themselves - this is our change in our pipeline to move from vs.coverage to opencover.