SonarQube and code coverage

Clean Code Guides
,
1

Coverage, the why and the how

Code coverage is an important quality metric that can be imported into SonarQube.

An external tool first computes the coverage report, and then, during the analysis, SonarQube is provided with information from this report.

If there is no coverage report format not covered by Test Coverage Parameters for your specific language, we provide the generic test data format for the coverage and the test execution reports. This is a simple format for gathering tests and coverage information when analyzing with SonarQube, and we recommend using it.

Languages and Their Coverage Reports

Language analyzers also support mainstream tool formats for the coverage reports, especially popular languages:

See Test coverage overview for all the available analysis parameters for the various coverage and test execution reports of other languages.

1-to-1 comparison between SonarQube and coverage report is not relevant

We sometimes have users reporting that the code coverage is different between SonarQube and the tool used to gather it.

The reason for this is most often because people are not comparing the same metrics.

SonarQube gets the covered lines from the coverage report given to the analyzer. Then it calculates all its coverage metrics from there and the executable lines also called lines to cover.

What is often compared is the Line Coverage, most often displayed by the external tool used to gather the covered lines, and what we define as Code Coverage, which is computed from the numbers extracted from the coverage report passed to the analyzer.

How to Calculate Code Coverage

We promote the Code Coverage metric because it reflects the best portion of source code covered by unit tests. This is the metric you can see on a project’s home page.

As you can read in the Metric Definitions page, the Code Coverage is computed as follow:

Coverage = (CT + LC) / (B + EL)
where

  • CT = conditions that have been evaluated to ‘true’ at least once
  • LC = covered lines = lines_to_cover - uncovered_lines
  • B = total number of conditions
  • EL = total number of executable lines (lines_to_cover)

Whereas the Line Coverage is computed as follows:

Line coverage = LC / EL
where

  • LC = covered lines (lines_to_cover - uncovered_lines)
  • EL = total number of executable lines (lines_to_cover)

By simply looking at the definitions we can already see that the results will be different.

Another way to express the above is

Coverage = (Covered Conditions + Covered Lines) / (Conditions to Cover + Lines to Cover)
where

  • Covered Conditions = conditions_to_cover - uncovered conditions
  • Covered Lines = lines_to_cover - uncovered_lines
  • Conditions to Cover = total number of conditions ( conditions_to_cover )
  • Lines to Cover = total number of executable lines ( lines_to_cover )

The calculation can be simplified by the metrics themselves:

Coverage = ( conditions_to_cover - uncovered conditions + lines_to_cover - uncovered_lines ) / ( conditions_to_cover + lines_to_cover )

It can also happen that the Line coverage computed by SonarQube differs slightly from the one calculated by the external tool. This is because the Lines to cover may not be the same according to SonarQube and the tool.

You can find the definition of what SonarQube considers as a line of code on the metric-definitions page.

The main idea of this article is to highlight the fact that comparing the coverage coming from SonarQube and the coverage coming from other tools is often misleading, SonarQube should be the reference point.

FAQ

Q: After migrating from 5.6 to 6.7 my coverage shows 0%, why is that?
A: Since SonarQube 6.2 and the simplification of code coverage support, if no coverage information is found the coverage is then set to zero by default.

Q: Why is my coverage on the new code blank?
A: Either the coverage report is not found by the analyzer or there are no new lines of code. For git users, using shallow clones can also lead to this behavior, simply use regular clones.

Q: My coverage is loaded but my tests do not show up (or vice versa).
A: Yes, coverage and test results are 2 different metrics, make sure you are loading both.

Q: I provided all the information to gather coverage but it is not loaded.
A: First, make sure that the coverage report exists before the analysis is run, check the analysis logs to get more information, and make sure that the coverage report is not empty and contains coverage information that corresponds to the sources you are analyzing (files, paths, etc.).

Q: I see the following error in the coverage sensor java.lang.IllegalStateException: LineXX is out of range in the file XYZ of my Sonar Scanner logs.
A: The message indicates that the sensor is asked to highlight a line that no longer exists in the code. The coverage report has to be recomputed to align with the existing code.

SonarSource Support team

14 Likes
2

Still, it’s not clear why Sonar uses its own coverage calculation model.
Moreover, I can’t understand where it gets the CT,CF, B values.

For ex., I have a pipeline that runs dotCover and later runs reportGenerator to transform dotCover results into HTML, Cobertura and Sonar formats. Both Cobertura and HTML shows around 46%, but when I import Sonar file into Sonar - it shows 55%. I looked at Sonar file and the only thing I see there - is the lines that were covered or not (e.g. ). There is now any information about conditions, only line coverage.

I wonder where can I read more about the Sonar coverage calculation model? Also, is there a way to instruct Sonar not to use its calculation model but rather use the simple one that other tools leverage?

2 Likes
3

In addition the SonarQube percentage always includes the testcase code itself, which greatly decreases our code coverage percentage. Is there a way to exclude all testcase code from the percentage?

4

Hi @scottdickerson,

You can exclude part of your code base from the code coverage calculations either through the global SonarQube configuration (see Narrowing the Focus | SonarQube Docs), or on a per-project basis through the sonar.coverage.exclusions property.

1 Like