PR decoration not having same % of code coverage results than in coverage report with tox

SonarQube Cloud
, ,
1

Hello, team.

We are having an issue where the test coverage on new lines through PR decoration on Github Actions are not working as expected.

We are gettitng a different total percentages when we are running coverage run and coverage report

For example, we have a PR where we are adding code into this function, and while we run the tox -e py locally we are not getting the same amount of percentage covered as we see on the Github workflow.

Locally:

api_v2/abcd_folder/abcd_file.py 110 50 44 3 47%

Github Workflows:

api_v2/abcd_folder/abcd_file.py 110 86 44 0 16%

Here’s our build.yml

name: PR Decoration
on:
  push:
    branches:
      - develop
      - production
      - pre-production
      - devtestingsonar
  pull_request:
    types: [opened, synchronize, reopened]
jobs:
  sonarcloud:
    name: SonarCloud
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.ref }}
      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Install tox
        run: pip install tox
      - name: Run tox
        run: tox -e py
      - name: SonarCloud Scan
        uses: SonarSource/sonarcloud-github-action@master
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}  # Needed to get PR information, if any
          SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

We would like to validate and see if our sonar-project.properties are fine, find them attached:

sonar.projectKey=xxxxx
sonar.organization=xxxxxxx
sonar.python.version=3.8
sonar.exclusions=external_services/book/tests/files/**/*, **/xsd_model/**/*
sonar.test.inclusions=**/tests/**/*, **test_*.py
sonar.test.exclusions=external_services/book/tests/files/**/*, **/xsd_model/**/*
sonar.qualitygate.wait=true
sonar.python.coverage.reportPaths=coverage.xml

And this is our tox.ini file:

[tox]
envlist = py38
skipsdist = True

[testenv]
deps =
    -r requirements.txt
    coverage
commands =
    coverage run manage.py test --parallel
    coverage xml
    coverage report

[coverage:run]
relative_files = True
branch = True
source = .
omit =
    external_services/book/tests/files/**/*
    **/xsd_model/**/*
    **/*airsub.py

[coverage:report]
omit =
    external_services/book/tests/files/**/*
    **/xsd_model/**/*
    **/*airsub.py

We are mainly concerned on how the paramaters inside sonar-project.properties and tox.ini file should work. See some screenshots attached of the results from the PR decoration

Screenshot 2023-07-14 at 15.26.07
Screenshot 2023-07-14 at 15.26.071184×661 44.4 KB

Please let me know if there are any more details needed to solve this.

Thanks.
Christian.

2

Hi,

To be clear, when you look at the coverage reports generated locally and in your GitHub Action, are they identical?

From this, it looks like they’re not:

And if that’s the case, then it’s going to be your coverage report generation that’s the problem. And that’s out of scope for us.

 
Ann

3

Hello, Ann.

Thanks for the reply. I’m checking that I have a warning that it can be involved in this issue.

I’m getting the following warning during sonar scan:

*** The following error(s) occurred while trying to import coverage report:**
Invalid directory path in ‘source’ element: /home/runner/work/trip_ninja_api/trip_ninja_api

why does my repository name appears twice? This happens when I put my parameter sonar.source=. in sonar-project.properties

Thanks,
Christian.

4

Hi Christian,

What are the paths in your report? Do they start with trip_ninja_api? The assumption is that test execution and analysis are kicked off from the same directory. And it looks like you’re running the tests from a directory above.

 
Ann

5

Hello Ann,

trip_ninja_api is the name of our repository. We don’t have a src/ directory, here’s how our repository structure looks like:

trip_ninja_api
├── api
│   └── tests
│       ├── test_file.py
├── graph_db
│   └── tests
│       ├── test_db_file.py
└── sonar-project.properties
└── buildspec.yml
└── tox.ini
└── coverage.xml

Let me know if you need additional details.

Thank you so much!
Christian

6

Hi Christian,

The question remains: what do the paths in your report start with?

 
Ann

7

Hi Ann,

It’s on the root of our repository, I have the parameter in sonar-project.properties to be:

sonar.python.coverage.reportPaths=coverage.xml

I’ve modified the tree of our repo structure.

8

Hi,

What I’m asking about is the paths you see when you open your coverage report in a text editor.

 
Ann

10

Hi,

The bad path is in your coverage report:

<?xml version="1.0" ?>
<coverage version="7.2.7" timestamp="1689694964079" lines-valid="26351" lines-covered="11586" line-rate="0.4397" branches-covered="0" branches-valid="0" branch-rate="0" complexity="0">
	<!-- Generated by coverage.py: https://coverage.readthedocs.io/en/7.2.7 -->
	<!-- Based on https://raw.githubusercontent.com/cobertura/web/master/htdocs/xml/coverage-04.dtd -->
	<sources>
		<source>/home/runner/work/trip_ninja_api/trip_ninja_api</source>

You need to look into how it’s generated. And unfortunately, that’s out of scope for this community.

 
Ann