SonarQube and the code coverage

sonarqube
coverage
guide

(Alexandre Frigout) #1

Coverage, why and how

Code coverage is an important quality metric that can be imported in SonarQube.

The coverage report has to be computed by an external tool first and then SonarQube will be provided with informations coming from this report during the analysis.

To get coverage informations in SonarQube, we provide the generic test data format for the coverage and the tests reports.

This is a simple format to gather tests and coverage information to inject into SonarQube and it is what we recommend to use.

Language analysers also support mainstream tools format for the coverage reports like JaCoCo for Java or dotCover, openCover for C# and others.

1 to 1 comparison is not relevant

We are sometimes reported that the code coverage is different between SonarQube and the tool used to gather it.

The reason for this is most often because people are not comparing the same metrics.

SonarQube gets the covered lines from the coverage report given to the analyser. Then it calculates all its coverage metrics from there and the executable lines or also called lines to cover.

What is very often being compared is the Line Coverage, most often displayed by the external tool used to gather the covered lines, and what we define as Code Coverage which is computed from the numbers extracted from the coverage report passed to the analyser.

The metric we promote is the Code Coverage because it is the one that reflects the best the portion of source code being covered by unit tests. This is the metric you can see on the home page of a project.

As you can read in the metric-definitions page, the Code Coverage is computed as follow:

Coverage = (CT + CF + LC)/(2*B + EL)
where
CT = conditions that have been evaluated to ‘true’ at least once
CF = conditions that have been evaluated to ‘false’ at least once
LC = covered lines = lines_to_cover - uncovered_lines
B = total number of conditions
EL = total number of executable lines (lines_to_cover)

Whereas the Line Coverage is computed as follow:

Line coverage = LC / EL
where
LC = covered lines (lines_to_cover - uncovered_lines)
EL = total number of executable lines (lines_to_cover)

By simply looking at the definitions we can already see that the results will be different.

It can also happen that the Line coverage computed by SonarQube differs a little bit from the one calculated by the external tool. This is because the Lines to cover may not be the same according to SonarQube and to the tool.

You can find the definition of what SonarQube considers as a line of code on the metric-definitions page.

The main idea of this article is to highlight the fact that comparing the coverage coming from SonarQube and the coverage coming from other tools is often misleading, SonarQube should be the reference point.

FAQ:

Q: After migrating from 5.6 to 6.7 my coverage shows 0%, why is that ?
R: Since SonarQube 6.2 and the implementation of the MMF-345, if no coverage information is found the coverage is then set to zero by default.

Q: Why my coverage on new code is blank ?
R: Either the coverage report is not found by the analyser or there are no new lines of code. For git users, using shallow clones can also lead to this behaviour, simply use regular clones.

Q: My coverage is loaded but my tests does not show up (or vice versa).
R: Yes, coverage and test results are 2 different metrics, make sure you are loading both.

Q: I provided all the information to gather coverage but it is not loaded.
R: Make sure in a first place that the coverage report exist before the analysis is run, check the analysis logs to get more informations, make sure that coverage report is not empty and contains coverage information that correspond to the sources you are analyzing (files, paths…)

Q: I see the following error when the coverage sensor is kicking in java.lang.IllegalStateException: LineXX is out of range in the file XYZ.
R: The message indicates that the sensor is asked to highlight a line that does not exists any more in the code, the coverage report has to be recomputed to be aligned with the existing code.

SonarSource Support team