Sonar Scanner does not extract Python coverage measures on Codeship but works on local

SonarQube
, , ,
1

Hi there,

I’m using:

I’m trying to integrate SonarQube with Codeship but as per https://community.codeship.com/t/sonar-qube-integration/853/6 there’s no such integration.

Thus I opted for the manual route: I download and ‘install’ SonarQube Scanner directly into the Codeship box as such

# Download Sonar Scanner
wget https://sonarsource.bintray.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-3.2.0.1227-linux.zip
# Unzip.
unzip sonar-scanner-cli-3.2.0.1227-linux.zip
# Add the Sonar Scanner `bin` directory to PATH.
export PATH=./sonar-scanner-3.2.0.1227-linux/bin:$PATH

All configuration (including the host URL and auth information for the SonarQube server) exists within a sonar-project.properties file committed in the project repo akin to this:

# Obfuscating the host for obvious reasons
sonar.host.url=http://*************
# This is an auth token
sonar.login=***************************
sonar.projectKey=*****
sonar.projectName=*****
sonar.sources=./app
sonar.sourceEncoding=UTF-8
# Set blob for the XUnit XML reports of the unit-testing.
sonar.python.xunit.reportPath=xunit-reports/TEST-*.xml
# Set blob for the Cobertura XML report of test-coverage.
sonar.python.coverage.reportPath=coverage-reports/coverage.xml

And then I perform the scan with SonarQube Scanner from the project repo sending the results to the server as such:

# Delete xunit reports.
if [ -d ./xunit-reports ]; then rm -rf xunit-reports/; fi
# Delete coverage reports.
if [ -d ./coverage-reports ]; then rm -rf coverage-reports/; fi
# Run the unit-tests creating XUnit XML report files.
coverage run --source app run_tests.py ~/google-cloud-sdk
# Generate a Cobertura XML coverage report file.
coverage xml -o coverage-reports/coverage.xml
# Perform a Sonar scan of the code based on the config under `sonar-project.properties`.
sonar-scanner

Now here’s the tricky part. Running sonar-scanner -X on my local machine works just fine. All unit-test and coverage information is sent to the server in addition to the rest of the analysis data.
My issue is that the exact same config on Codeship fails to write coverage measures making coverage appear as 0% on the server when the project gets scanned on Codeship.

Running on local the scanner logs contain entries like this (obfuscated filenames):

15:46:48.841 INFO: Python test coverage
15:46:48.846 INFO: Parsing report '/*****/coverage-reports/coverage.xml'
15:46:48.942 DEBUG: Saving coverage measures for file '******'
15:46:48.947 DEBUG: Saving coverage measures for file '******'

These Saving coverage measures entries however do not appear when running from Codeship. All I get is:

05:45:03.857 INFO: Python test coverage
05:45:03.861 INFO: Parsing report '/home/rof/clone/coverage-reports/coverage.xml'

The coverage files are identical on local and remote. Running coverage report works and produces the same results on both environments.

Has anyone encountered issues like this before? Is there a way to debug this further and figure out what’s missing?

Thanks!

2

Hey Adamos,

If I would have one main advice for further troubleshooting at this stage, it would be: check the filepaths !

  • understand the paths with which the scanner indexes source files (hint: look for indexed with language in the scanner log)
  • understand the paths reported in the coverage file

Then compare those paths, and do that on both local setup and Codeship. It is likely to yield an understand of why the matching is done in one case, but seemingly missing in the other case.

3

Hey Nicolas, sorry for the late response.

Thanks for the tip, I’ll look more into the files and try to verify why no coverage measures are written out.

I’ll update this topic with my findings.

Thanks
Adam

4

Hi Nicolas,

I checked and compared the filepaths between the local and CI environments.

The coverage XML files contains identical paths for the different files and in both environments sonar-scanner -X shows that the files are indexed with the same path and language (something like 'app/exceptions/__init__.py' indexed with language 'py').

The difference is that I can see a log-entry stating Saving coverage measures for file 'app/exceptions/__init__.py' in the local logs but nothing like that in the remote.

I tried testing the older Linux versions of sonar-scanner from https://sonarsource.bintray.com/Distribution/sonar-scanner-cli/ (down to 3.0.1) but the results were the same.

I noticed a sonar-scanner-debug binary in the bin directory of the distro. Could that be used somehow to get a better grasp on the behaviour?

Lastly I’ve noticed these entries in the log:

06:05:49.313 DEBUG: Using pattern '' to find reports
06:05:49.369 DEBUG: No report was found for sonar.python.coverage.itReportPath using default pattern
06:05:49.369 DEBUG: Using pattern '' to find reports
06:05:49.424 DEBUG: No report was found for sonar.python.coverage.overallReportPath using default pattern

AFAIK these variables pertain to deprecated means of designating the coverage report location and shouldn’t be used but I thought I’d check.

Thanks!
Adam

6

Hi Adamos,

Thanks for the additional details.

Nah. Scanner logs in debug mode is the best approach to further troubleshoot.

On that note:

  • mind sharing the full debug logs of analysis in both environments ? that would help further diagnose
  • in the meantime, on Codeship, mind checking whether coverage.xml is still physically present when the scan starts ? asking because of past experiences, with CI frameworks doing their housekeeping in between jobs and workspaces.
7

Hi Nicolas,

Thanks again for the response :slight_smile:. Appreciate it.

mind sharing the full debug logs of analysis in both environments ? that would help further diagnose

I’m attaching the sonar-scanner -X logs here for both the local and Codeship environments. Please keep in mind that I obfuscated the IP of our SonarQube server (replaced with *************) and the name of the project (replaced with xxxxxxxxxxxxxxxxx) given how public this post is :slight_smile:. I hope this doesn’t pose an impediment.

sonar-scanner-local.txt (41.4 KB)

sonar-scanner-codeship.txt (39.4 KB)

in the meantime, on Codeship, mind checking whether coverage.xml is still physically present when the scan starts ?

I tripled check that the coverage.xml files are in place. In fact, I’m debugging the Codeship build directly on their VM so as to avoid any surprising behaviour caused by cleanup.

Thank you!
Adam

8

Thanks Adamos. No key finding after going through the logs, so while we’re at it: could you share the full coverage.xml from both environments ? That way we’ll have the full picture. (and no problem with you obfuscating sensitive data, as long as we can reasonably correlate/compare data/paths/etc.).

9

Was hoping you’d see something obvious :slight_smile:. Attaching the coverage files here (same obfuscation rules):

coverage_codeship.xml.txt (48.0 KB)
coverage_local.xml.txt (48.0 KB)

Thanks!
Adam

10

Nothing obvious in the logs alone. But now that you show the coverage reports there definitely is something that stands out.

Local

Directory where scanner handles sources:
18:52:39.466 INFO: Base dir: /Users/akyriakou004/Development/xxxxxxxxxxxxxxxxx

Directory where coverage reports sources:

	<sources>
		<source>/Users/akyriakou004/Development/xxxxxxxxxxxxxxxxx</source>
	</sources>

It’s a match !

Codeship

Directory where scanner handles sources:
09:07:07.381 INFO: Base dir: /home/rof/clone

Directory where coverage reports sources:

	<sources>
		<source>/home/rof/src/github.com/PwC-Next/xxxxxxxxxxxxxxxxx</source>
	</sources>

It’s a mismatch !

Wrap-up

Seeing how SonarPython does treat the source element in the coverage reports (it’s open-source, see code here), I think the mismatch right here is what you need to focus on for now, to get coverage import working on Codeship,

11

Oh wow what’s going on there. Good catch Nicolas, will look into it ASAP. Thanks!

12

Hi Nicolas,

So I checked it out and it seems you were right, I’ve no idea how or why but the paths pulled into the Cobertura report were erroneous so Sonar-Scanner couldn’t find the files.

I’m yet to figure out the reason for that but a quick sed replace can fix the issue by replacing the erroneous paths with correct ones. Following that the scanner works perfectly and saves the coverage measures.

Thanks heaps for your help!
Adam

1 Like
13

Great ! Happy we figured it out, thanks for the update.