SonarQube and code coverage

Coverage, the why and the how

Code coverage is an important quality metric that can be imported into SonarQube.

The coverage report has to be computed by an external tool first and then SonarQube will be provided with information coming from this report during the analysis.

To get coverage information in SonarQube, we provide the generic test data format for the coverage and the test execution reports.

This is a simple format to gather tests and coverage information to inject into SonarQube and it is what we recommend using.

Languages and Their Coverage Reports

Language analyzers also support mainstream tool formats for the coverage reports, especially popular languages:

See Test coverage overview for all the available analysis parameters for the various coverage and test execution reports of other languages.

1-to-1 comparison between SonarQube and coverage report is not relevant

We sometimes have users reporting that the code coverage is different between SonarQube and the tool used to gather it.

The reason for this is most often because people are not comparing the same metrics.

SonarQube gets the covered lines from the coverage report given to the analyzer. Then it calculates all its coverage metrics from there and the executable lines or also called lines to cover.

What is very often being compared is the Line Coverage, most often displayed by the external tool used to gather the covered lines, and what we define as Code Coverage which is computed from the numbers extracted from the coverage report passed to the analyzer.

How to Calculate Code Coverage

The metric we promote is the Code Coverage because it is the one that reflects the best portion of source code covered by unit tests. This is the metric you can see on the home page of a project.

As you can read in the Metric Definitions page, the Code Coverage is computed as follow:

Coverage = (CT + CF + LC) / (2*B + EL)
where

  • CT = conditions that have been evaluated to ‘true’ at least once
  • CF = conditions that have been evaluated to ‘false’ at least once
  • LC = covered lines = lines_to_cover - uncovered_lines
  • B = total number of conditions
  • EL = total number of executable lines (lines_to_cover)

Whereas the Line Coverage is computed as follow:

Line coverage = LC / EL
where

  • LC = covered lines (lines_to_cover - uncovered_lines)
  • EL = total number of executable lines (lines_to_cover)

By simply looking at the definitions we can already see that the results will be different.

Another way to express the above is

Coverage = (Covered Conditions + Covered Lines) / (Conditions to Cover + Lines to Cover)
where

  • Covered Conditions = conditions_to_cover - uncovered conditions
  • Covered Lines = lines_to_cover - uncovered_lines
  • Conditions to Cover = total number of conditions ( conditions_to_cover )
  • Lines to Cover = total number of executable lines ( lines_to_cover )

The calculation can be simplified by the metrics themselves:

Coverage = ( conditions_to_cover - uncovered conditions + lines_to_cover - uncovered_lines ) / ( conditions_to_cover + lines_to_cover )

It can also happen that the Line coverage computed by SonarQube differs a little bit from the one calculated by the external tool. This is because the Lines to cover may not be the same according to SonarQube and to the tool.

You can find the definition of what SonarQube considers as a line of code on the metric-definitions page.

The main idea of this article is to highlight the fact that comparing the coverage coming from SonarQube and the coverage coming from other tools is often misleading, SonarQube should be the reference point.

FAQ

Q: After migrating from 5.6 to 6.7 my coverage shows 0%, why is that?
A: Since SonarQube 6.2 and the simplification of code coverage support, if no coverage information is found the coverage is then set to zero by default.

Q: Why is my coverage on new code blank?
A: Either the coverage report is not found by the analyzer or there are no new lines of code. For git users, using shallow clones can also lead to this behavior, simply use regular clones.

Q: My coverage is loaded but my tests do not show up (or vice versa).
A: Yes, coverage and test results are 2 different metrics, make sure you are loading both.

Q: I provided all the information to gather coverage but it is not loaded.
A: First, make sure that the coverage report exists before the analysis is run, check the analysis logs to get more information, and make sure that the coverage report is not empty and contains coverage information that corresponds to the sources you are analyzing (files, paths, etc.).

Q: I see the following error in the coverage sensor java.lang.IllegalStateException: LineXX is out of range in the file XYZ of my Sonar Scanner logs.
A: The message indicates that the sensor is asked to highlight a line that does not exist anymore in the code, the coverage report has to be recomputed to be aligned with the existing code.

SonarSource Support team

14 Likes

Still, it’s not clear why Sonar uses its own coverage calculation model.
Moreover, I can’t understand where it gets the CT,CF, B values.

For ex., I have a pipeline that runs dotCover and later runs reportGenerator to transform dotCover results into HTML, Cobertura and Sonar formats. Both Cobertura and HTML shows around 46%, but when I import Sonar file into Sonar - it shows 55%. I looked at Sonar file and the only thing I see there - is the lines that were covered or not (e.g. ). There is now any information about conditions, only line coverage.

I wonder where can I read more about the Sonar coverage calculation model? Also, is there a way to instruct Sonar not to use its calculation model but rather use the simple one that other tools leverage?

2 Likes

In addition the SonarQube percentage always includes the testcase code itself, which greatly decreases our code coverage percentage. Is there a way to exclude all testcase code from the percentage?

Hi @scottdickerson,

You can exclude part of your code base from the code coverage calculations either through the global SonarQube configuration (see Narrowing the Focus | SonarQube Docs), or on a per-project basis through the sonar.coverage.exclusions property.

1 Like