SonarQube V.S. Jest Conditional Code Coverage Discrepencies

SonarQube Server / Community Build
, ,
1

Make sure to tell us:

  • version: 9.9.3.79811 - enterprise edition

  • Sonar Scanner running on Github Actions Runner - Ubuntu latest

  • What’s the issue you’re facing?
    Hi I am running into an issue in my branch where the code coverage looks to be lower that what I would expect based on the coverage reports outputted by Jest and what I am seeing outputted by SonarQube for conditional code coverage.

For our CI/CD process we use Github actions, and for this specific repo we have three test jobs that execute and focus on three different types of tests. Each test job outputs a jest coverage report:
lcov-unit.info
lcov-integration.info
lcov-e2e.info

All three of these coverage reports are then analyzed by SonarQube to determine if the branch passes the code coverage gate we have set in place. However, I am experiencing a discrepancy for the conditional coverage for SonarQube shows 56% percent but in Jest the branch, function, and Statements section show a higher percentage in 2/3 coverage reports

coverage results in jest:
lcov-unit.info - movement-score.entity.ts | 100 | 100 | 100 |
lcov-integration.info - movement-score.entity.ts | 89.85 | 66.66 | 85.71 |
lcov-e2e.info - file does not contain data on this file as it was not tested in this test suite

coverage report in sonarqube:

image
image1909×792 117 KB

I know that SonarQube coverage analysis differs from jest coverage analysis, however I was expecting that because of the one of the coverage reports displays 100% covered so would sonarqube. When providing multiple files in the sonar.javascript.lcov.reportPaths how does SonarQube calculate the conditional coverage ? Because I would expect that SonarQube would take the max of the x coverage reports, however that does not look to be the case here.

2

Hey there.

Can you share these files? I think it will make it clear which files are reporting condition coverage and which aren’t, and why the results look the way they do. I anticipate that one is only reporting line coverage, while the other reports condition coverage.

3

coverage-artifacts.zip (191.7 KB)

Hi Colin I attached the three coverage reports that the sonar analysis is using here in this zip file let me know if you run into any issues downloading the zip/files

4

Thanks!

So a few things:

I don’t think that’s true. In lcov-e2e.info, this file is represented with coverage data.

SF:src/et-customization/domain/entities/movement-score.entity.ts
FN:30,(anonymous_2)
FN:37,(anonymous_3)
FN:105,(anonymous_4) 
...

All 3 files appear to report a different number of conditions that can be covered

lcov-unit.info 40 branches found, 40 branches hit

BRF:40
BRH:40

lcov-integration.info: 26 branches found, 23 branches hit. All of these branches are a subset of lcov-unit.info based on line number.

BRF:26
BRH:23

And finally lcov-e2e.info, 33 branches found, 22 branches hit.

BRF:33
BRH:22

Some of the line numbers referenced in lcov-e2e.info don’t match the line numbers reported for the file in the other reports (specifically for branch coverage). Like these:

BRDA:132,6,0,0
BRDA:132,6,1,0
BRDA:132,6,2,0

So I can only guess that lcov-e2e.info is indicating missing branch coverage data that the other files aren’t. You should figure out why this file is being reported in this test suite that you didn’t think was.

5

Thanks Colin, that’s definitely something we need to look into. (I’m from the same company as Elijah)

Given that the unit tests have 100% unit test coverage in this example, and given that unit tests seem to find the most branches (hopefully all branches) I would have expected that our test coverage would be represented as 100% within sonarqube. My understanding is that it’s not an average, that the coverage should be additive, is that correct? If so, why would we have gotten 56% coverage for this particular file? It would make sense if there are totally new branches in the integration and e2e tests that aren’t found in unit tests, and that’s something we can look into, but assuming that unit tests cover all branches, why wouldn’t this coverage be evaluated by sonarqube at 100%?

Thank you,
Tracy

6

And I was so happy when I thought this thread had died. :laughing: This one gives me a headache, and it’s taken me a moment to refamiliarize myself.

I think the most important thing is this:

If uncovered conditions are reported on line numbers found in no other coverage report, they will not be marked as covered. SonarQube ultimately calculates coverage on individual lines and then sums it up. It discards the file-level measures from the coverage report (I provided them above for illustrative purposes),