How to get the Pytest coverage in sonarqube analysis through Jenkine file for pytest

Im Trying to get coverage report on sonarqube, im able to generate the coverage.xml report but not able to get the coverage in sonarqube.(coverage.xml report not picking by sonarqube), Now getting 0% coverage
code for reference:

                         dir('Main') {
                             echo "Setting up Python environment for pytest..."
                             sh '''
                             pip install --upgrade pip
                             pip install pytest coverage pyspark kafka-python pymemcache

                             export PYSPARK_SUBMIT_ARGS="--packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.1 pyspark-shell"

                             coverage run -m pytest --disable-warnings --maxfail=1 || true
                             coverage xml -o ${WORKSPACE}/coverage.xml || { echo 'Coverage XML generation failed!'; exit 1; }


                             if [ -f ${WORKSPACE}/coverage.xml ]; then
                                 echo "Coverage file exists: ${WORKSPACE}/coverage.xml"
                             else
                                 echo "Coverage file not found at ${WORKSPACE}/coverage.xml"
                                 exit 1
                             fi

                             echo "Coverage file generated:"
                             cat ${WORKSPACE}/coverage.xml
                             '''
                         }
                     } catch (Exception e) {
                         echo "An error occurred during the pytest stage, but continuing the pipeline."
                     }
                 }
             }
         }

         stage('SonarQube Analysis') {
             steps {
                 script {
                     echo "Starting SonarQube analysis..."
                     withSonarQubeEnv() {
                         sh '''
                         podman run --rm \
                             -e SONAR_HOST_URL= \
                             -e SONAR_SCANNER_OPTS="\
                             -Dsonar.projectKey=${ORG}-${APP_NAME} \
                             -Dsonar.projectVersion=${BUILD_NUMBER} \
                             -Dsonar.sources=Main,Scheduler \
                             -Dsonar.exclusions=**/__pycache__/**,**/*.yaml,**/tests/**,Charts/**,Jenkinsfile,Jenkinsfile_dep \
                             -Dsonar.python.version=3 \
                             -Dsonar.python.coverage.reportPaths=/usr/src/coverage.xml \
                             -Dsonar.qualitygate.wait=true \
                             -Dsonar.qualitygate.timeout=300" \
                             -e SONAR_TOKEN="${SONAR_TOKEN}" \
                             -v "${WORKSPACE}:/usr/src" ${SONAR_SCANNER_IMAGE}
                         '''
                     }

Hi,

Welcome to the community!

Could you share your analysis log?

The analysis / scanner log is what’s output from the analysis command. Hopefully, the log you provide - redacted as necessary - will include that command as well.

This guide will help you find them.

 
Thx,
Ann

06:38:58.952 INFO  Project configuration:
06:38:58.952 INFO    Excluded sources: **/__pycache__/**, **/*.yaml, **/tests/**, Charts/**, Jenkinsfile, Jenkinsfile_dep
06:38:59.443 WARN  File '/usr/src/Main/env/lib/python3.7/site-packages/pyspark/jars/hadoop-client-runtime-3.3.4.jar' is bigger than 20MB and as consequence is removed from the analysis scope.
06:38:59.464 WARN  File '/usr/src/Main/env/lib/python3.7/site-packages/pyspark/jars/rocksdbjni-7.9.2.jar' is bigger than 20MB and as consequence is removed from the analysis scope.
06:38:59.734 WARN  File '/usr/src/Main/env/lib64/python3.7/site-packages/pyspark/jars/hadoop-client-runtime-3.3.4.jar' is bigger than 20MB and as consequence is removed from the analysis scope.
06:38:59.752 WARN  File '/usr/src/Main/env/lib64/python3.7/site-packages/pyspark/jars/rocksdbjni-7.9.2.jar' is bigger than 20MB and as consequence is removed from the analysis scope.
06:38:59.825 INFO  1789 files indexed
06:38:59.825 INFO  2585 files ignored because of inclusion/exclusion patterns
06:38:59.825 INFO  1724 files ignored because of scm ignore settings
06:38:59.826 INFO  Quality profile for css: Sonar way
06:38:59.826 INFO  Quality profile for js: Sonar way
06:38:59.826 INFO  Quality profile for json: Sonar way
06:38:59.826 INFO  Quality profile for py: Sonar way
06:38:59.827 INFO  Quality profile for web: Sonar way
06:38:59.827 INFO  Quality profile for xml: Sonar way
06:38:59.827 INFO  ------------- Run sensors on module TA-Alarm_correlation_be
06:38:59.893 INFO  Load metrics repository
06:38:59.910 INFO  Load metrics repository (done) | time=17ms
06:39:00.873 INFO  Sensor Python Sensor [python]
06:39:00.906 INFO  Starting global symbols computation
06:39:00.909 INFO  1243 source files to be analyzed
06:39:02.089 WARN  Invalid character encountered in file /usr/src/Main/env/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py at line 2010 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
06:39:10.446 WARN  Invalid character encountered in file /usr/src/Main/env/lib/python3.7/site-packages/pkg_resources/__init__.py at line 1871 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
06:39:10.914 INFO  830/1243 files analyzed, current file: Main/env/lib/python3.7/site-packages/pip/_vendor/chardet/langbulgarianmodel.py
06:39:14.618 INFO  1243/1243 source files have been analyzed
06:39:15.045 INFO  Starting rules execution
06:39:15.046 INFO  1243 source files to be analyzed
06:39:25.046 INFO  196/1243 files analyzed, current file: Main/env/lib/python3.7/site-packages/pip/_vendor/tenacity/stop.py
06:39:35.047 INFO  418/1243 files analyzed, current file: Main/env/lib/python3.7/site-packages/kafka/producer/kafka.py
06:39:45.047 INFO  702/1243 files analyzed, current file: Main/env/lib/python3.7/site-packages/pip/_vendor/pygments/formatters/img.py
06:39:55.048 INFO  965/1243 files analyzed, current file: Main/env/lib/python3.7/site-packages/_pytest/_io/__init__.py
06:40:03.985 INFO  1243/1243 source files have been analyzed
06:40:03.985 INFO  The Python analyzer was able to leverage cached data from previous analyses for 0 out of 1243 files. These files were not parsed.
06:40:03.986 INFO  Sensor Python Sensor [python] (done) | time=63113ms
06:40:03.986 INFO  Sensor Cobertura Sensor for Python coverage [python]
06:40:04.014 WARN  No report was found for sonar.python.coverage.reportPaths using pattern /export/home/V/workspace/workspace/PR-41/coverage.xml
06:40:04.014 INFO  Sensor Cobertura Sensor for Python coverage [python] (done) | time=28ms
06:40:04.014 INFO  Sensor PythonXUnitSensor [python]
06:40:04.038 INFO  Sensor PythonXUnitSensor [python] (done) | time=24ms
06:40:04.038 INFO  Sensor JaCoCo XML Report Importer [jacoco]
06:40:04.039 INFO  'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
06:40:04.040 INFO  No report imported, no coverage information will be imported by JaCoCo XML Report Importer
06:40:04.040 INFO  Sensor JaCoCo XML Report Importer [jacoco] (done) | time=2ms
06:40:04.040 INFO  Sensor IaC CloudFormation Sensor [iac]
06:40:04.045 INFO  0 source files to be analyzed
06:40:04.052 INFO  0/0 source files have been analyzed
06:40:04.052 INFO  Sensor IaC CloudFormation Sensor [iac] (done) | time=12ms
06:40:04.052 INFO  Sensor IaC Kubernetes Sensor [iac]
06:40:04.054 INFO  0 source files to be analyzed
06:40:04.057 INFO  0/0 source files have been analyzed
06:40:04.057 INFO  Sensor IaC Kubernetes Sensor [iac] (done) | time=5ms
06:40:04.057 INFO  Sensor JavaScript analysis [javascript]
06:40:07.848 INFO  1 source file to be analyzed
06:40:09.205 INFO  1/1 source file has been analyzed
06:40:09.205 INFO  Hit the cache for 0 out of 1
06:40:09.206 INFO  Miss the cache for 1 out of 1: ANALYSIS_MODE_INELIGIBLE [1/1]
06:40:09.206 INFO  Sensor JavaScript analysis [javascript] (done) | time=5149ms
06:40:09.206 INFO  Sensor TypeScript analysis [javascript]
06:40:09.208 INFO  No input files found for analysis
06:40:09.208 INFO  Hit the cache for 0 out of 0
06:40:09.208 INFO  Miss the cache for 0 out of 0
06:40:09.208 INFO  Sensor TypeScript analysis [javascript] (done) | time=2ms
06:40:09.208 INFO  Sensor CSS Rules [javascript]
06:40:09.226 INFO  4 source files to be analyzed
06:40:09.465 INFO  4/4 source files have been analyzed
06:40:09.466 INFO  Hit the cache for 0 out of 0
06:40:09.466 INFO  Miss the cache for 0 out of 0
06:40:09.466 INFO  Sensor CSS Rules [javascript] (done) | time=258ms


showing Zero coverage

Hi,

Can I please have the full analysis log, starting with the analysis command itself, all the way through to process exit?

 
Thx,
Ann

podman run --rm -e SONAR_HOST_URL="https://${SONARQUBE_URL}" -e SONAR_SCANNER_OPTS=" -Dsonar.projectKey=${ORG}-${APP_NAME} -Dsonar.projectVersion=${BUILD_NUMBER} -Dsonar.sources=Main,Scheduler -Dsonar.exclusions=**/__pycache__/**,**/*.yaml,**/tests/**,Charts/**,Jenkinsfile,Jenkinsfile_dep -Dsonar.python.version=3 -Dsonar.python.coverage.reportPaths=$**/coverage.xml -Dsonar.qualitygate.wait=true -Dsonar.qualitygate.timeout=300" -e SONAR_TOKEN="${SONAR_TOKEN}" -v "${WORKSPACE}:/usr/src" ${SONAR_SCANNER_IMAGE}
— Shell Script
22s
+ podman run --rm -e SONAR_HOST_URL=https://swsol-sonar."removed" -e 'SONAR_SCANNER_OPTS=-Dsonar.projectKey=T-correlation_be -Dsonar.projectVersion=7 -Dsonar.sources=Main,Scheduler -Dsonar.exclusions=**/__pycache__/**,**/*.yaml,**/tests/**,Charts/**,Jenkinsfile,Jenkinsfile_dep -Dsonar.python.version=3 -Dsonar.python.coverage.reportPaths=*/coverage.xml                             -Dsonar.qualitygate.wait=true -Dsonar.qualitygate.timeout=300' -e SONAR_TOKEN="token" -v /export/home/virtuora/workspace/workspace/correlation_be_PR-41:/usr/src "imagetag"
13:00:57.729 INFO  Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
13:00:57.734 INFO  Project root configuration file: NONE
13:00:57.755 INFO  SonarScanner CLI 6.1.0.4477
13:00:57.757 INFO  Java 17.0.11 Eclipse Adoptium (64-bit)
13:00:57.757 INFO  Linux 5.15.0-101.103.2.1.el8uek.x86_64 amd64
13:00:57.759 INFO  SONAR_SCANNER_OPTS= -Dsonar.projectKey=T-correlation_be -Dsonar.projectVersion=7 -Dsonar.sources=Main,Scheduler -Dsonar.exclusions=**/__pycache__/**,**/*.yaml,**/tests/**,Charts/**,Jenkinsfile,Jenkinsfile_dep -Dsonar.python.version=3 -Dsonar.python.coverage.reportPaths=*/coverage.xml -Dsonar.qualitygate.wait=true -Dsonar.qualitygate.timeout=300
13:00:57.801 INFO  User cache: /opt/sonar-scanner/.sonar/cache
13:00:59.080 INFO  Communicating with SonarQube Server 9.9.1.69595
13:00:59.468 INFO  Load global settings
13:00:59.552 INFO  Load global settings (done) | time=85ms
13:00:59.554 INFO  Server id: "removed"
13:00:59.559 INFO  User cache: /opt/sonar-scanner/.sonar/cache
13:00:59.562 INFO  Load/download plugins
13:00:59.563 INFO  Load plugins index
13:00:59.602 INFO  Load plugins index (done) | time=39ms
13:01:00.946 INFO  Load/download plugins (done) | time=1384ms
13:01:01.550 INFO  Process project properties
13:01:01.556 INFO  Process project properties (done) | time=6ms
13:01:01.559 INFO  Execute project builders
13:01:01.560 INFO  Execute project builders (done) | time=1ms
13:01:01.564 INFO  Project key: T-correlation_be
13:01:01.564 INFO  Base dir: /usr/src
13:01:01.564 INFO  Working dir: /tmp/.scannerwork
13:01:01.573 INFO  Load project settings for component key: 'T-correlation_be'
13:01:01.598 INFO  Load project settings for component key: 'T-correlation_be' (done) | time=25ms
13:01:01.676 INFO  Load project branches
13:01:01.691 INFO  Load project branches (done) | time=15ms
13:01:01.692 INFO  Load branch configuration
13:01:01.694 INFO  Load branch configuration (done) | time=2ms
13:01:01.763 INFO  Load quality profiles
13:01:01.820 INFO  Load quality profiles (done) | time=57ms
13:01:01.826 INFO  Load active rules
13:01:03.134 INFO  Load active rules (done) | time=1308ms
13:01:03.141 INFO  Load analysis cache
13:01:03.497 INFO  Load analysis cache | time=356ms
13:01:03.549 INFO  Load project repositories
13:01:03.614 INFO  Load project repositories (done) | time=64ms
13:01:03.652 INFO  Indexing files...
13:01:03.652 INFO  Project configuration:
13:01:03.652 INFO    Excluded sources: **/__pycache__/**, **/*.yaml, **/tests/**, Charts/**, Jenkinsfile, Jenkinsfile_dep
13:01:03.762 INFO  20 files indexed
13:01:03.763 INFO  6 files ignored because of inclusion/exclusion patterns
13:01:03.763 INFO  0 files ignored because of scm ignore settings
13:01:03.765 INFO  Quality profile for json: Sonar way
13:01:03.765 INFO  Quality profile for py: Sonar way
13:01:03.765 INFO  ------------- Run sensors on module T-correlation_be
13:01:03.865 INFO  Load metrics repository
13:01:03.884 INFO  Load metrics repository (done) | time=19ms
13:01:05.006 INFO  Sensor Python Sensor [python]
13:01:05.018 INFO  Starting global symbols computation
13:01:05.021 INFO  12 source files to be analyzed
13:01:05.603 INFO  12/12 source files have been analyzed
13:01:05.638 INFO  Starting rules execution
13:01:05.639 INFO  12 source files to be analyzed
13:01:06.527 INFO  12/12 source files have been analyzed
13:01:06.528 INFO  The Python analyzer was able to leverage cached data from previous analyses for 0 out of 12 files. These files were not parsed.
13:01:06.538 INFO  Sensor Python Sensor [python] (done) | time=1532ms
13:01:06.539 INFO  Sensor Cobertura Sensor for Python coverage [python]
13:01:06.543 WARN  No report was found for sonar.python.coverage.reportPaths using pattern */coverage.xml
13:01:06.543 INFO  Sensor Cobertura Sensor for Python coverage [python] (done) | time=4ms
13:01:06.543 INFO  Sensor PythonXUnitSensor [python]
13:01:06.545 INFO  Sensor PythonXUnitSensor [python] (done) | time=2ms
13:01:06.545 INFO  Sensor JaCoCo XML Report Importer [jacoco]
13:01:06.546 INFO  'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
13:01:06.546 INFO  No report imported, no coverage information will be imported by JaCoCo XML Report Importer
13:01:06.546 INFO  Sensor JaCoCo XML Report Importer [jacoco] (done) | time=1ms
13:01:06.546 INFO  Sensor IaC CloudFormation Sensor [iac]
13:01:06.573 INFO  0 source files to be analyzed
13:01:06.582 INFO  0/0 source files have been analyzed
13:01:06.582 INFO  Sensor IaC CloudFormation Sensor [iac] (done) | time=36ms
13:01:06.582 INFO  Sensor IaC Kubernetes Sensor [iac]
13:01:06.585 INFO  0 source files to be analyzed
13:01:06.589 INFO  0/0 source files have been analyzed
13:01:06.589 INFO  Sensor IaC Kubernetes Sensor [iac] (done) | time=7ms
13:01:06.589 INFO  Sensor CSS Rules [javascript]
13:01:06.589 INFO  No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
13:01:06.589 INFO  Sensor CSS Rules [javascript] (done) | time=0ms
13:01:06.589 INFO  Sensor C# Project Type Information [csharp]
13:01:06.590 INFO  Sensor C# Project Type Information [csharp] (done) | time=1ms
13:01:06.590 INFO  Sensor C# Analysis Log [csharp]
13:01:06.599 INFO  Sensor C# Analysis Log [csharp] (done) | time=9ms
13:01:06.599 INFO  Sensor C# Properties [csharp]
13:01:06.599 INFO  Sensor C# Properties [csharp] (done) | time=0ms
13:01:06.599 INFO  Sensor HTML [web]
13:01:06.601 INFO  Sensor HTML [web] (done) | time=2ms
13:01:06.601 INFO  Sensor TextAndSecretsSensor [text]
13:01:06.609 INFO  13 source files to be analyzed
13:01:06.685 INFO  13/13 source files have been analyzed
13:01:06.685 INFO  Sensor TextAndSecretsSensor [text] (done) | time=84ms
13:01:06.685 INFO  Sensor VB.NET Project Type Information [vbnet]
13:01:06.686 INFO  Sensor VB.NET Project Type Information [vbnet] (done) | time=1ms
13:01:06.686 INFO  Sensor VB.NET Analysis Log [vbnet]
13:01:06.696 INFO  Sensor VB.NET Analysis Log [vbnet] (done) | time=10ms
13:01:06.696 INFO  Sensor VB.NET Properties [vbnet]
13:01:06.696 INFO  Sensor VB.NET Properties [vbnet] (done) | time=0ms
13:01:06.696 INFO  Sensor com.github.mc1arke.sonarqube.plugin.scanner.ScannerPullRequestPropertySensor
13:01:06.696 INFO  Sensor com.github.mc1arke.sonarqube.plugin.scanner.ScannerPullRequestPropertySensor (done) | time=0ms
13:01:06.696 INFO  Sensor IaC Docker Sensor [iac]
13:01:06.699 INFO  0 source files to be analyzed
13:01:06.740 INFO  0/0 source files have been analyzed
13:01:06.740 INFO  Sensor IaC Docker Sensor [iac] (done) | time=44ms
13:01:06.743 INFO  ------------- Run sensors on project
13:01:06.801 INFO  Sensor Analysis Warnings import [csharp]
13:01:06.801 INFO  Sensor Analysis Warnings import [csharp] (done) | time=0ms
13:01:06.801 INFO  Sensor Zero Coverage Sensor
13:01:06.813 INFO  Sensor Zero Coverage Sensor (done) | time=12ms
13:01:06.825 INFO  CPD Executor Calculating CPD for 12 files
13:01:06.839 INFO  CPD Executor CPD calculation finished (done) | time=14ms
13:01:06.885 INFO  Load New Code definition
13:01:06.899 INFO  Load New Code definition (done) | time=14ms
13:01:06.903 INFO  Analysis report generated in 61ms, dir size=416.6 kB
13:01:06.931 INFO  Analysis report compressed in 28ms, zip size=179.0 kB
13:01:06.979 INFO  Analysis report uploaded in 48ms
13:01:06.981 INFO  ------------- Check Quality Gate status
13:01:06.981 INFO  Waiting for the analysis report to be processed (max 300s)
13:01:17.060 INFO  EXECUTION FAILURE
13:01:17.061 INFO  Total time: 19.337s
13:01:17.061 ERROR Error during SonarScanner CLI execution
13:01:17.061 ERROR QUALITY GATE STATUS: FAILED - View details on https://swsol-sonar."removed"/dashboard?id=T-correlation_be
13:01:17.061 ERROR 
13:01:17.061 ERROR Re-run SonarScanner CLI using the -X switch to enable full debug logging.
script returned exit code 2

Hi,

Thanks for the log. Here’s your problem:

13:01:06.543 WARN No report was found for sonar.python.coverage.reportPaths using pattern */coverage.xml

Can you give an example path to one of your coverage reports? (I’m assuming multiple, given your use of the wildcard.)

Did you perhaps mean a pattern of **/coverage.xml?

 
Ann

I tried below way, not worked for me. Im missing anything below.

coverage run -m pytest --disable-warnings --maxfail=1 || true
 coverage xml -o ${WORKSPACE}/coverage.xml || { echo 'Coverage XML 
 generation failed!'; exit 1; }

stage('SonarQube Analysis') {
        //     steps {
        //         script {
        //             echo "Starting SonarQube analysis..."
        //             withSonarQubeEnv() {
        //                 sh '''
        //                 podman run --rm \
        //                     -e SONAR_HOST_URL="https://${SONARQUBE_URL}" \
        //                     -e SONAR_SCANNER_OPTS="\
        //                     -Dsonar.projectKey=${ORG}-${APP_NAME} \
        //                     -Dsonar.projectVersion=${BUILD_NUMBER} \
        //                     -Dsonar.sources=Main,Scheduler \
        //                     -Dsonar.exclusions=**/__pycache__/**,**/*.yaml,**/tests/**,Charts/**,Jenkinsfile,Jenkinsfile_dep \
        //                     -Dsonar.python.version=3 \
        //                     -Dsonar.python.coverage.reportPaths=$**/coverage.xml \
        //                     -Dsonar.qualitygate.wait=true \
        //                     -Dsonar.qualitygate.timeout=300" \
        //                     -e SONAR_TOKEN="${SONAR_TOKEN}" \
        //                     -v "${WORKSPACE}:/usr/src" ${SONAR_SCANNER_IMAGE}
        //                 '''
        //             }

Hi,

Is a coverage file actually generated? And if so, where to?

 
Ann