Test Coverage Discrepancies Between Local Environment and SonarCloud

SonarQube Cloud
1

Title: Test Coverage Discrepancies Between Local Environment and SonarCloud

ALM used: Bitbucket Cloud

CI system used: Bitbucket Cloud

Languages of the repository: Java 17

Error observed:
I’ve encountered an intriguing challenge with test coverage percentages. The issue at hand is that the reported test coverage on SonarCloud (12.3%) is lower than what we observe in the local environment.

Here’s the scoop:

Your insights are highly appreciated!

2

Hi,

Welcome to the community!

For files that are included in your coverage report, we go 100% by what’s in the report. So… I’m not sure what to tell you there. Are you submitting reports from multiple engines that maybe don’t agree? (Even so, we should be treating them additively.)

For files that aren’t listed in your coverage report, this dusty-but-still-relevant blog should help.

 
Ann

3

Hi Ann,

Thanks for your prompt response! To clarify, our coverage report is generated using Jacoco, and the coverage percentages shown in the local reference image were obtained from the ‘jacoco.exec’ file. This file is also utilized by SonarCloud for coverage reporting, ensuring consistency between the local environment and SonarCloud.

Regarding test engines, we’re using a combination of JUnit 4 and JUnit 5 in our project. There’s no parallel use of different engines for the same tests.

Given this, any insights on ensuring alignment between the local and SonarCloud coverage results would be greatly appreciated.

Joan

4

Hi Joan,

Actually, we no longer read the .exec file; the .xml file is what’s supported now.

That’s fine. I just wondered if you were reading multiple types of reports, but as I said, that really shouldn’t have an impact.

Let’s start with this: can you identify a file, or a line that shows up as covered in your JaCoCo xml report that’s not shown as covered in SonarCloud?

 
Ann

5

Hi Ann,

This is an example of a line that shows up covered locally and not shown as covered in SonarCloud.

image
image1067×449 29.2 KB

image
image1141×605 25.4 KB

6

Hi,

Thanks for this example. Can you provide the JaCoCo XML file that shows this coverage?

 
Thx,
Ann

7

Hi Ann,

I hope this message finds you well. I wanted to update you on the recent changes to our Jacoco report configuration to generate the jacoco.xml and jacoco.exec files. This adjustment was motivated by SonarCloud’s exclusive support for the xml format.

Here’s a snippet of our updated configuration:

Dependency:

<dependency>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>0.8.11</version>
</dependency>

Plugin Configuration:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
</plugin>
<plugin>
    <groupId>org.jacoco</groupId>
    <artifactId>jacoco-maven-plugin</artifactId>
    <version>${jacoco-maven-plugin.version}</version>
    <executions>
        <execution>
            <goals>
                <goal>prepare-agent</goal>
            </goals>
        </execution>
        <!-- attached to Maven test phase -->
        <execution>
            <id>report</id>
            <phase>test</phase>
            <goals>
                <goal>report</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <outputDirectory>${project.build.directory}</outputDirectory>
        <dataFile>${project.build.directory}/jacoco.exec</dataFile>
        <output>file</output>
    </configuration>
</plugin>

Reporting:

<reporting>
    <plugins>
        <plugin>
            <groupId>org.jacoco</groupId>
            <artifactId>jacoco-maven-plugin</artifactId>
            <version>${jacoco-maven-plugin.version}</version>
            <reportSets>
                <reportSet>
                    <reports>
                        <report>report</report>
                        <report>xml</report>
                    </reports>
                </reportSet>
            </reportSets>
        </plugin>
    </plugins>
</reporting>

The jacoco.xml file containing the necessary coverage information was generated locally. However, upon testing these changes remotely, the coverage information is no longer displayed in SonarCloud (refer to the attached image).

image
image1109×577 33.6 KB

Unfortunately, I can’t share the jacoco.xml report due to security concerns. Still, I can assure you that the file contains information (see the below image)

image
image1244×798 80 KB

image
image1216×559 39.8 KB

image
image977×203 11.4 KB

Could you kindly assist us in identifying if there is any specific configuration for integrating the jacoco.xml with SonarCloud in the pom.xml or SonarCloud itself to ensure a smooth integration with SonarCloud?

I appreciate your support.

Joan

8

Hi Joan,

Coverage of ‘-’ in SonarCloud indicates no coverage report was uploaded.

Can you review your analysis configuration to make sure you’re pointing analysis to the JaCoCo XML file?

And if so, can you share your analysis log?

The analysis / scanner log is what’s output from the analysis command. Hopefully, the log you provide - redacted as necessary - will include that command as well.

This guide will help you find them.

 
Ann

9

Hi Ann,

I appreciate your guidance. I’ve carefully reviewed the SonarCloud configuration to ensure it aligns with the documentation provided at https://docs.sonarsource.com/sonarqube/latest/analyzing-source-code/test-coverage/java-test-coverage/#coverage-parameter. Specifically, I set the value for “sonar.coverage.jacoco.xmlReportPaths” in SonarCloud settings to target/site/jacoco/jacoco.xml to explicitly point to the JaCoCo XML file.

image
image1489×739 50.3 KB

Additionally, I’ve included the following configuration in the pom.xml file:

<sonar.coverage.jacoco.xmlReportPaths>
    target/site/jacoco/jacoco.xml
</sonar.coverage.jacoco.xmlReportPaths>

Despite these configurations, the discrepancy in coverage persists.

I’m sharing the redacted logs in the attached file to assist with troubleshooting.

[main] INFO org.sonarsource.scanner.maven.SonarQubeMojo - Sensor JaCoCo XML Report Importer [jacoco]
[main] INFO org.sonarsource.scanner.maven.SonarQubeMojo - Importing 1 report(s). Turn your logs in debug mode in order to see the exhaustive list.
[main] INFO org.sonarsource.scanner.maven.SonarQubeMojo - Sensor JaCoCo XML Report Importer [jacoco] (done) | time=222ms

Sonarlogs.txt (8.9 KB)

Your insights into these logs will be invaluable in resolving the coverage issue.

I appreciate your support.

Joan

10

Hi Joan,

Can I have the full analysis log? It seems that parts are missing from this one.

 
Thx,
Ann

11

Hi there,

I wanted to share a quick update regarding the test coverage discrepancies. After a thorough investigation, we’ve identified the root cause.

Root Cause:

The issue stemmed from our Bitbucket Pipeline configuration, which had separate steps for unit tests and integration tests. Since the integration tests take longer to execute, they were overwriting the coverage report generated by the unit tests, leading to discrepancies in the reported coverage percentages.

Ongoing Efforts:

We are actively exploring ways to modify our pipeline to ensure both unit and integration tests are included in the coverage report without one overwriting the other.

Community Input:

As part of this effort, we are seeking your insights. Is it possible to generate two jacoco.xml reports in different paths—one for unit tests and the other for integration tests? Additionally, can SonarCloud read and merge both reports for comprehensive coverage validation?

Thank you for your ongoing support and contributions.

Joan

1 Like
12

Hi Joan,

It looks like you’ve made great progress!

I think the docs on coverage for multi-module projects may be relevant.

 
HTH,
Ann