Hello all,
The line coverage (percentage) reported by SonarQube seems off:
SonarQube says the Line Coverage is 90.9%, but (25,125 - 1,119) / 25,125 * 100% = 95.5%.
Apparently, SonarQube uses a different formula, but I haven’t been able to find it. The docs only specify the formula for Coverage (Test coverage overview).
This is with SonarQube 9.2, community edition.
Regards, Frank
Hello,
Should I report this as a bug?
Thanks, Frank
Hello all,
Is there anybody from the SonarQube team that can give us insight in how the branch and line coverage percentage is calculated? What is the formula for this? I cannot find this in the documentation.
Thank you in advance,
Sebastiaan
Hi,
The formula in the documentation is the formula that is expected to be used by SonarQube.
I just checked the source code and also a few of the projects we analyze internally and they match the formula:
Coverage = ( lines_to_cover - uncovered_lines ) / lines_to_cover.
I checked with the latest SQ but I couldn’t find anything that impacts the calculation of coverage since 9.2.
Is the number incorrect for one or multiple projects?
Hi Duarte, thank you for your reply. We have upgraded to Community Edition, Version 9.4 (build 54424), but the problem persists. The analysis is of a Kotlin backend project:
The line coverage reported is 97.9% but should be (28,988 - 424) / 28,988 * 100% = 98,5%:
The same SonarQube instance also has a frontend project with TypeScript. There the line coverage is reported correctly.
Would it help to send logging? If so, where?
Thanks, Frank
Hello Frank,
Unfortunately I don’t think there are logs that would be useful. One thing you could do is double check the measures you see in the UI. Do you get the same numbers with
https://{SERVER_HOSTNAME}/api/measures/component?component={PROJEC_KEY}&metricKeys=lines_to_cover,uncovered_lines,coverage?
Also if you go to the Measures Tab, and on the left menu click on Coverage Under the Coverage drop down and see the values per directories and per files, do those numbers make sense?
Looking at the code that performs the calculation I can’t see what could go wrong, so just trying to get some clues on how to reproduce the problem.
The API returns the same values as visible in the UI. Note that I replaced coverage with line_coverage in the API call, cause that’s the value that we’re interested in.
http://sonar.example.org/api/measures/component?component=backend&metricKeys=lines_to_cover,uncovered_lines,line_coverage returns:
{
"component": {
"key": "backend",
"name": "backend",
"qualifier": "TRK",
"measures": [
{
"metric": "lines_to_cover",
"value": "29029"
},
{
"metric": "uncovered_lines",
"value": "426",
"bestValue": false
},
{
"metric": "line_coverage",
"value": "97.9",
"bestValue": false
}
]
}
}
I also added up the uncovered lines reported per file in the Measures tab manually. The number I got was close to 426. What is a bit peculiar and I didn’t notice before is that for some files no line coverage is calculated despite them having uncovered lines:
If I leave out the files without line coverage I get 272 uncovered lines. But (29,029 - 272) / 29,029 * 100% = 99,0% != 97,9% so that doesn’t solve the puzzle.
Hope this helps at least a bit? Please let me know how I can help further, if possible.
Thanks, Frank
Thanks.
I forgot to check: Is the project analyzed regularly (and was it analyzed since the upgrade to 9.4)?
To give some background info, the line coverage is calculate for each node (project, directories, files) independently, based on the numbers of “uncovered lines” and “lines to cover” of each node.
I’m curious if one of the numbers got somehow changed since the project was last analyzed, or if the analysis consistently produces that inconsistency.
Hi Duarte,
Yes, the project is actively worked on and the main branch is analyzed a few times per week.
Current figures (June 15, see screenshot and API response below): lines to cover: 29,320, uncovered lines: 497. So calculated coverage is 98,3% but Sonar reports 97,8%.
Condition coverage is wrong too by the way: conditions to cover: 3,997, uncovered conditions 1,012. So calculated condition coverage is 74,7%. Sonar reports 82,2%.
On new code, the line coverage is correct (706 - 10) / 706 x 100% = 98,6%. But the condition coverage is way off: (52 - 17) / 52 x 100% = 67,3%. Sonar reports 83,3%. Which is a weird number because 43 / 52 x 100% = 83,0% and 44 / 52 x 100% = 84,6% so there’s no way to get 83,3% with 52 conditions to cover.
The UI and the API are consistent. http://sonar.example.org/api/measures/component?component=backend&metricKeys=lines_to_cover,uncovered_lines,line_coverage,conditions_to_cover,uncovered_conditions,branch_coverage returns:
{
"component": {
"key": "backend",
"name": "backend",
"qualifier": "TRK",
"measures": [
{
"metric": "line_coverage",
"value": "97.8",
"bestValue": false
},
{
"metric": "lines_to_cover",
"value": "29320"
},
{
"metric": "uncovered_conditions",
"value": "1012",
"bestValue": false
},
{
"metric": "conditions_to_cover",
"value": "3997"
},
{
"metric": "branch_coverage",
"value": "82.2",
"bestValue": false
},
{
"metric": "uncovered_lines",
"value": "497",
"bestValue": false
}
]
}
}
Cheers, Frank
Thanks, Frank.
Reviewing again the code and the data in the thread, I may have found a way the numbers would be inconsistent.
It looks like when calculating the line coverage, only files not classified as test files (by SonarQube) are taken into account, however the calculation of lines to cover and uncovered lines doesn’t have this constraint. Same thing happens with the corresponding measures for conditions.
Of course that would mean that coverage is being reported for files that SonarQube is classifying as test files, which potentially indicates that the classification is incorrect.
So going back to those files you found with uncovered lines and without line coverage, are those files considered as test files by SonarQube? I can think of 2 ways of knowing whether they are seen as test files:
a) The scanner with debug logs enabled will output that info for each file
b) Test files won’t have a number for ‘lines of code’ in the UI
Hi Duarte,
It seems you’re correct. The files that end with *Test.kt are listed with uncovered lines on the “Coverage → Overall → Uncovered Lines” page, but are not listed at all on the “Size → Lines of Code” page.
Does this mean we can solve the issue by excluding the test files from the coverage measurement via sonar.coverage.exclusions?
Thanks!
Yes, that should fix it.
I created a bug ticket to fix the problem on the SonarQube side: [SONAR-16521] 'Line Coverage' inconsistent with 'Lines to cover' and 'uncovered lines' - SonarSource
Thank you for the support!
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.
