Code Coverage not being reported if executed on Jenkins

[jvaronenka (Juozas Varonenka)]

SonarQube version: 9.5.0.56709
SonarScanner version: 4.6.2.2472
Target platform: Mac OS X 11.0
Target language: C++

I have a unit test suite (based on gTest) that I am executing, with a binary that produces coverage data. Then I manipulate the data slightly to produce coverage information that SonarQube can consume, in particular:

./testing/run_cpp_tests.py -c Release -go Coverage -mt unit-tests -p -v
mv build/Makefiles/Release/compile_commands.json compile_commands.json
mv default.profraw unit.profraw
cp build/Makefiles/Release/testing/unitTests/bin/unit-tests .
xcrun llvm-profdata merge unit.profraw -o all.profdata
xcrun llvm-cov show --instr-profile all.profdata unit-tests > lcov.info
./sonar-scanner/bin/sonar-scanner -Dproject.settings=testing/sonar-project.properties

Sonar scanner properties are as such (redacted where needed):

sonar.cfamily.llvm-cov.reportPath=lcov.info
sonar.cfamily.analysisCache.mode=fs
sonar.cfamily.analysisCache.path=analysisCache
sonar.cpp.std=c++11
sonar.projectKey=<project name>
sonar.sources=<list of source folders>
sonar.tests=<list of unit test folders>
sonar.cfamily.compile-commands=compile_commands.json
sonar.host.url=<url>
sonar.login=<login>
sonar.sourceEncoding=UTF-8
sonar.scm.exclusions.disabled=false
sonar.python.version=3.6

When I execute these commands locally, SonarQube produces code coverage data correctly.
If I put these commands into my Jenkins job (not using Jenkins plugin), code coverage produces 0% result.
If I then login to the virtual machine that Jenkins was using and execute just the sonar scanner again manually, results are once again produced correctly.

I am not completely sure where to look for issues here. There are no errors on Jenkins when reading lcov.info:

INFO: Sensor llvm-cov [cpp]
INFO: Parsing /private/tmp/src/workspace/products/UnitTester/lcov.info
INFO: Sensor llvm-cov [cpp] (done) | time=320ms

The only difference in logs that I see is that on Jenkins, it gets auto configured branches and such:

INFO: Load branch configuration
INFO: Detected branch/PR in 'Jenkins'
INFO: Auto-configuring branch 'temp-run'
INFO: Load branch configuration (done) | time=5ms
INFO: Auto-configuring with CI 'Jenkins'

I would appreciate any pointers as to what I could check.

[Colin]

Hey there.

I suppose my first question would be – have you validated the lcov.info file generated in the Jenkins environment (/private/tmp/src/workspace/products/UnitTester/lcov.info) and is it at all different from the one generated locally?

[jvaronenka (Juozas Varonenka)]

Hi.

Yes, I tried to look into lcov.info and the only difference that I saw was that one of the files was counting empty lines, brackets, etc. whereas the other one wasn’t.

What’s interesting though, is that the same lcov.info was used when Jenkins was sending the data to SonarQube and when I tried to send it manually after Jenkins has finished the job - I didn’t regenerate lcov.info. And it worked manually, but not automatically on Jenkins.

[Colin]

@jvaronenka

What’s interesting though, is that the same lcov.info was used when Jenkins was sending the data to SonarQube and when I tried to send it manually after Jenkins has finished the job - I didn’t regenerate lcov.info.

I’m not too surprised by this – as file paths are inevitably going to be different between your local runs and your Jenkins runs.

@jvaronenka

Yes, I tried to look into lcov.info and the only difference that I saw was that one of the files was counting empty lines, brackets, etc. whereas the other one wasn’t.

This… sure sounds like an important difference. I would suggest you first nail down why the reports are being generated differently.

I peeked at the source code and no further logging is available than what you’ve shown… and that seems like something we can improve. I’ve flagged this thread for expert attention to consider if more logging would be useful.

[jvaronenka (Juozas Varonenka)]

Found the issue.

Our Jenkins instance runs tests on macOS from /private/tmp/src folder, however coverage data in lcov.info generated data as if coming from /tmp/src folder. Which is correct, since on macOS /tmp is just a symlink to /private/tmp.

So, when the sonar scanner ran, it expected all of the paths to come from /private/tmp/src, but they were all stored as /tmp/src in lcov.info and thus no coverage was found.

Changing working directory to /tmp/src resolved the issue.

So, it doesn’t look like the issue with sonar scanner, but rather how lcov.info was generated. Still not sure why /tmp/src was chosen for it, but that’s something I will investigate separately.

Thanks for the help and bringing the potential path issue to my attention.