Local PHPUnit and sonarqube show different coverage percentages

[rneufeld (Ryan)]

We are using a hosted version of SonarQube, and have configured it to ingest our phpunit coverage several xml files.

We have encountered an odd situation where xdebug/phpunit is showing 100% coverage for a single file locally, but Sonar is reporting approximately 56%.

We are using clover as the XML format.

One thing that may play into this, is that we have to runs of the same test suite, each generating their own coverage files.

I’ve only been able to find dead links to a blog post about “executable lines”, so I’m at my wits end with this.

edit: After some further digging, sonar is complaining about lines of code not being tested, but the tests would not actually pass if that were the case.

[Colin]

Hey there.

As noted in the template post, what version of SonarQube are you using?

@rneufeld

One thing that may play into this, is that we have to runs of the same test suite, each generating their own coverage files.

Are you passing all these coverage files to sonar.php.coverage.reportPaths? You can verify this by looking at the logs:

How to find logs about importing code coverage

When coverage data isn’t appearing as expected in SonarQube / SonarCloud, the first place you should look is the scanner logs. Where do I find the scanner logs? This will likely be broken out into its own guide soon The Scanner logs are the output of executing one of the scanners, either directly or wrapped by one of many integrations, such as the SonarScanner for Jenkins, SonarScanner for Azure DevOps, SonarQube/SonarCloud GitHub Actions or SonarQube / SonarCloud Bitbucket Pipes. …

[rneufeld (Ryan)]

Yes, we have the coverage paths configured.

Strangely enough I enabled debug on the sonar-scanner command and the next run showed the expected output.

I’m wondering now if there is some async process that could have been wedged causing this problem?

[rneufeld (Ryan)]

The debug logs were not helpful.

However looking at the clover results. We’re importing one xml where SomeFile.php is 100% covered in one run of the unit tests.

The next time we run the suite, we don’t run any tests for SomeFile.php and the clover xml shows 0% coverage for that file.

I’ll mention that we’re using Enterprise 9.9.5 and have configured the coverage clover paths from the web ui, and are not passing them using sonar-scanner

[Colin]

@rneufeld

However looking at the clover results. We’re importing one xml where SomeFile.php is 100% covered in one run of the unit tests.

The next time we run the suite, we don’t run any tests for SomeFile.php and the clover xml shows 0% coverage for that file.

Do you end up with multiple reports (XML coverage reports), or just one report where the coverage figure varies?

[rneufeld (Ryan)]

We import multiple coverage reports.

[Colin]

Hey Ryan!

@rneufeld

However looking at the clover results. We’re importing one xml where SomeFile.php is 100% covered in one run of the unit tests.

The next time we run the suite, we don’t run any tests for SomeFile.php and the clover xml shows 0% coverage for that file.

Sorry for the delay.

I ran a little test this morning where I sent two coverage files to SonarQube, one marking the line as covered and one not. I ended up with 100% coverage. Just to be extra sure, I even flipped around the order of the reports being passed into sonar.php.coverage.reportPath.

In the logs, I see both coverage files parsing correctly. It would be good if you confirmed that as well.

11:02:44.049 INFO: Importing /Users/colinmueller/source/sonar-php/sonar-php-plugin/src/test/resources/reports/phpunit/phpunit2.coverage.xml
11:02:44.063 INFO: Importing /Users/colinmueller/source/sonar-php/sonar-php-plugin/src/test/resources/reports/phpunit/phpunit.coverage.xml

I would reckon that any further troubleshooting would require you to provide a sample project this can be reproduced on, since we can’t reproduce it on our side.