Title: Test Coverage Discrepancies Between Local Environment and SonarCloud
ALM used: Bitbucket Cloud
CI system used: Bitbucket Cloud
Languages of the repository: Java 17
Error observed:
I’ve encountered an intriguing challenge with test coverage percentages. The issue at hand is that the reported test coverage on SonarCloud (12.3%) is lower than what we observe in the local environment.
Here’s the scoop:
Test coverage on SonarCloud for xxxx repository is showing a lower percentage (12.3%) than what’s reported in the local environment.
Interestingly, some intentionally excluded packages in pom.xml are correctly omitted on SonarCloud, but there are discrepancies in coverage percentages for files that we’ve confirmed to have 100% coverage.
For files that are included in your coverage report, we go 100% by what’s in the report. So… I’m not sure what to tell you there. Are you submitting reports from multiple engines that maybe don’t agree? (Even so, we should be treating them additively.)
For files that aren’t listed in your coverage report, this dusty-but-still-relevant blog should help.
Thanks for your prompt response! To clarify, our coverage report is generated using Jacoco, and the coverage percentages shown in the local reference image were obtained from the ‘jacoco.exec’ file. This file is also utilized by SonarCloud for coverage reporting, ensuring consistency between the local environment and SonarCloud.
Regarding test engines, we’re using a combination of JUnit 4 and JUnit 5 in our project. There’s no parallel use of different engines for the same tests.
Given this, any insights on ensuring alignment between the local and SonarCloud coverage results would be greatly appreciated.
Actually, we no longer read the .exec file; the .xml file is what’s supported now.
That’s fine. I just wondered if you were reading multiple types of reports, but as I said, that really shouldn’t have an impact.
Let’s start with this: can you identify a file, or a line that shows up as covered in your JaCoCo xml report that’s not shown as covered in SonarCloud?
I hope this message finds you well. I wanted to update you on the recent changes to our Jacoco report configuration to generate the jacoco.xml and jacoco.exec files. This adjustment was motivated by SonarCloud’s exclusive support for the xml format.
The jacoco.xml file containing the necessary coverage information was generated locally. However, upon testing these changes remotely, the coverage information is no longer displayed in SonarCloud (refer to the attached image).
Unfortunately, I can’t share the jacoco.xml report due to security concerns. Still, I can assure you that the file contains information (see the below image)
Could you kindly assist us in identifying if there is any specific configuration for integrating the jacoco.xml with SonarCloud in the pom.xml or SonarCloud itself to ensure a smooth integration with SonarCloud?
Coverage of ‘-’ in SonarCloud indicates no coverage report was uploaded.
Can you review your analysis configuration to make sure you’re pointing analysis to the JaCoCo XML file?
And if so, can you share your analysis log?
The analysis / scanner log is what’s output from the analysis command. Hopefully, the log you provide - redacted as necessary - will include that command as well.
I appreciate your guidance. I’ve carefully reviewed the SonarCloud configuration to ensure it aligns with the documentation provided at https://docs.sonarsource.com/sonarqube/latest/analyzing-source-code/test-coverage/java-test-coverage/#coverage-parameter. Specifically, I set the value for “sonar.coverage.jacoco.xmlReportPaths” in SonarCloud settings to target/site/jacoco/jacoco.xml to explicitly point to the JaCoCo XML file.
Despite these configurations, the discrepancy in coverage persists.
I’m sharing the redacted logs in the attached file to assist with troubleshooting.
[main] INFO org.sonarsource.scanner.maven.SonarQubeMojo - Sensor JaCoCo XML Report Importer [jacoco]
[main] INFO org.sonarsource.scanner.maven.SonarQubeMojo - Importing 1 report(s). Turn your logs in debug mode in order to see the exhaustive list.
[main] INFO org.sonarsource.scanner.maven.SonarQubeMojo - Sensor JaCoCo XML Report Importer [jacoco] (done) | time=222ms