Adding another test run halves the code coverage

[StephenLarkin]

Hey we’ve started to convert some of our projects to use OpenCover, and we’re seeing better statistics being reported now. We hit an issue today though where suddenly our code coverage suddenly dropped dramatically. We haven’t changed anything, but from our Friday master scan to our Monday master scan we dropped From 24.9% coverage overall to 17.8%.

Has anything changed that could have contributed to this difference? It seems to once again affect the projects that still use the Visual Studio coverage more than others…

[Andrei_Epure]

hi @StephenLarkin

I’d need to understand more - have you changed the SQ version or upgraded the C# plugin? If you upgraded the C# plugin, what was the previous version and what’s the current one? Have you upgraded Visual Studio to a newer version?

[StephenLarkin]

This is when running it in Azure DevOps. I’ve checked all of the task versions in there and they’re all the same. We’re using SonarCloud, instead of SonarQube so that is why I was wondering if something could have changed on your end.

[Andrei_Epure]

hi @StephenLarkin, the only thing that may have changed on SonarCloud is the plugin, and you can check the deployment data on https://sonarcloud.io/api/plugins/installed - it has an epoch timestamp - Tuesday, June 30, 2020 11:22:34.454 AM .

[StephenLarkin]

Okay, looking into our SonarCloud settings it seems like something cleared out our typescript settings from SonarCloud, so it no longer was picking up the coverage file for any of our typescript code.

I have since updated our yaml task to include the setting in the extra properties, so that for whatever reason it got cleared out of our sonarcloud settings it will not happen again.

extraProperties: |
          sonar.javascript.lcov.reportPaths=App_Test/Coverage/lcov.info

This after applying this change the PR reported that the code coverage would be set back to what it was before.

However this has somehow broken most of our C# coverage.

Frankly, I’m at a bit of a loss at a few things:

1. Why did the Typescript coverage setting disappear from our SonarCloud settings?
2. Why has our C# coverage suddenly plummeted. It seems like it is no longer picking up the coverage results in the Visual Studio format that it used to retrieve automatically.

In the addition to adding the additional property, I also updated the Visual Studio tasks as it appears to support a different format now:

From this:

    - task: SonarSource.sonarcloud.14d9cde6-c1da-4d55-aa01-2965cd301255.SonarCloudPrepare@1
      displayName: Prepare analysis on SonarCloud
      inputs:
        SonarCloud:#name
        organization: 'wsi-xceptor'
        projectKey: #key
        projectName: #name

To this:

    - task: SonarCloudPrepare@1
      inputs:
        SonarCloud: #name
        organization: #org
        scannerMode: 'MSBuild'
        projectKey: #key
        projectName: #name
        extraProperties: |
          sonar.javascript.lcov.reportPaths=App_Test/Coverage/lcov.info

Is there any way this could have had an impact? I would have expected that it should not.

[Andrei_Epure]

hey @StephenLarkin

@StephenLarkin

Why has our C# coverage suddenly plummeted. It seems like it is no longer picking up the coverage results in the Visual Studio format that it used to retrieve automatically.

I’m not sure on this one, I’ll have to come back to you.

However, looking at our own config, we specify the path to the code coverage file explicitly.

To investigate further, we’d also need the verbose logs. Please add the sonar.verbose=true property at the BEGIN step and give us the logs of the BEGIN and END steps.

[StephenLarkin]

For some unknown reason, without changing anything else, we seem to have gone back to our normal C# code coverage now…

[Andrei_Epure]

thanks for letting us know, @StephenLarkin. That’s strange. AFAIK , we haven’t made any change on our side.

If it happens again, it would be nice to see the verbose logs and the coverage report, to narrow down the problem.

For the Visual Studio coverage, do you control the version of vstest ? Or is it controlled by the platform?

[StephenLarkin]

We use the test platform installer to install the latest version, but between those runs it’s still using the same version: Microsoft.TestPlatform 16.6.1