Two teams sudden drop in "lines to cover" and "covered lines" from Azure DevOps build

Hi,

Yesterday, Nov 13, two of our teams started seeing a sudden drop in their coverage and lines to cover. The first team had:
12 Nov: 149K lines to Cover, 11K Covered lines, 139K uncovered --> 7.2% code coverage
13 Nov: 139K lines to Cover, 6.5K Covered lines, 132K uncovered lines --> 4.7% code coverage

Second team:
12 Nov: 13.7K lines to Cover, 12.5K Covered lines, 1.1K uncovered --> 91.6% code coverage
13 Nov: 8.3K lines to Cover, 7.5K Covered lines, 1K uncovered lines --> 90.0% code coverage

This makes us wonder why these are now excluded? According to the teams, they have not made any changes related to these items. So did something change on SonarCloud? E.g. exclusions that did not work before that work now maybe?

PS: We did see that in the processing of “Run analysis” a lot less files were analyzed than before. But right now we are in a blank whether it is the Microsoft hosted agent or SonarCloud.

Hi,

Which scanner are you using, and which build technology ?

Thank you.

Mickaël

Hi @mickaelcaro,
We use Azure DevOps hosted build agent, latest version of the scanner task. Technologies used: C#, JavaScript, XML, HTML, Typescript and CSS (typical Microsoft Intranet applications) for both applications. T-SQL for the database.

Thank you.So i guess that you are analyzing your code with the Scanner for MSBuild ?

Correct!
PS: We have a team which only has C# and their coverage remained as is during the Nov 12 - 13 period.

If you have access to both logs of the 12th and the 13th, can you send them to me (Run Code Analysis should be sufficient) if you can of course.

Thank you !

Hi, sure can. Attached both dates for both projects.Coverage CreditHub 12 Nov RCA.txt (73.1 KB) Coverage CreditHub 13 Nov RCA.txt (73.6 KB) Coverage eCS 12 Nov RCA.txt (274.2 KB) Coverage eCS 13 Nov RCA.txt (266.9 KB)

Thank you very much !

From what i saw (I looked at the “CreditHub” log files only for now"), there have been an increased number of files indexed by our sensors i.e 1912 files on the 12th and 1949 on the 13th, so to me it’s normal at that only point that the coverage decrease if those new files were not covered by Unit Test.

We can see also that, given the metric of the log itself, it says on the 12th that :

 Coverage Report Statistics: 1305 files, 696 main files, 696 main files with coverage, 609 test files, 0 project excluded files, 0 other language files.

Whereas on the 13th :

Coverage Report Statistics: 979 files, 362 main files, 362 main files with coverage, 617 test files, 0 project excluded files, 0 other language files.

So indeed something has changed here.

As a further reading, can you please provide me, if you have access to, the scanner context that are available in the Background tasks section? Ideally again one of the 12th and the 13th.

Thank you !

Hi,
Indeed, I see the file difference. Weird thing is that both teams did not make any change in their solution/build. Requested scanner files attached. Who determines the nr of files and the coverage? Is that the MS Build or Sonar scanner? As could it also be that previous file exclusions were for instance not taken into account, and they are taken into account now? Or a miscalculation somewhere that is now fixed?

CreditHub 12 nov 2019 scanner context.txt (80.9 KB) CreditHub 13 nov 2019 scanner context.txt (80.9 KB) eCS 12 nov 2019 scanner context.txt (79.2 KB) eCS 13 nov 2019 scanner context.txt (79.2 KB)

Ok so exact same versions of plugins on each side, that is strange.

Do you see such a drop on the Code Coverage results that are published by the VSTest task itself, on the build summary ?

I’ll have a look at files exclusions on my side.

Hi Mickaël,
I downloaded the code coverage reports of both 12 and 13 november of the main branch of both applications, and both went from a 18-19Mb file on the 12th to a 14-15Mb file on the 13th.
So roughly 4Mb less, and also slightly different coverage (but only 1% on eCS, not the 3% in SonarCloud).
Is it the SonarCloud extension telling MS Build which parts to cover for the code analysis, or is this purely Microsoft side?

Hi Richard,

SonarCloud and its subsequent analyzers are only based on what we feed them in terms of code coverage (trx / coveragexml files if VSTest is used), then a match is made between coverage metrics and files that have been analyzed.out
There could be a slight difference between what is uploaded as a summary in the build against the coverage shown on SonarCloud simply because some files might not have been analyzed or are not supported whether the coverage shown in the summary is kinda “raw”

If there have been a drop on the coverage file itself, then there’s not that much we can do unfortunately.