which versions are you using (SonarQube Server / Community Build, Scanner, Plugin, and any relevant extension)
The version of sonarQube Server used is 9.9.2 (build 77730)
The sonar scanner used is sonar-scanner-4.2.0.1873
sonar.python.version=3.9
Ours is a python project … the property file contains
sonar.python.xunit.reportPath
sonar.python.coverage.reportPaths
sonar.python.xunit.skipDetails=true
Despite this, the test_success_density is NOT getting displayed on the sonarQube “Measures” → “Coverage” → “Tests” menu item within the dashboard. The test execution report is generated using the --junit-xml pluggin… and looks normal
What do the logs say about the import of test execution data? Ideally, you should turn on DEBUG level logs (sonar-scanner -X, or setting sonar.verbose=true)
The logs will be between two lines that look like this:
I’ve run some tests, and I think it comes down to you using sonar.python.xunit.skipDetails=true.
With:
16:21:17.303 INFO Sensor PythonXUnitSensor [python]
16:21:17.303 DEBUG Using pattern 'xunit-reports/xunit-result-*.xml' to find reports
16:21:17.317 INFO Sensor PythonXUnitSensor [python] (done) | time=14ms
Without:
16:22:32.512 INFO Sensor PythonXUnitSensor [python]
16:22:32.512 DEBUG Using pattern 'xunit-reports/xunit-result-*.xml' to find reports
16:22:32.528 INFO Processing report '/Users/colinmueller/source/sonar-python/python-commons/src/test/resources/org/sonar/plugins/python/xunit-reports/xunit-result-2.xml'
...
16:22:32.538 INFO Sensor PythonXUnitSensor [python] (done) | time=26ms
I can also reproduce this on the very latest version of SonarQube.
This seems like a bug to me, and I’ll report it as such. In the meantime, can you try removing sonar.python.xunit.skipDetails=true from your analysis parameters? This typically only needs to be set (and even then, need is a strong word) if you have paths in these reports that aren’t in your project.
I had a similar issue and removing the sonar.python.xunit.skipDetails does allow the unit results to process. I want to confirm that the unit test metrics only show up under measures for long lived branches. I don’t see them in my working branch.
I got rid of the property you asked me to sonar.python.xunit.skipDetails=true and there were a couple of undesired side effects
On the Dashboard (landing page) → Overview → under the “Measures” section … in the Overall Code (navigation pane) the “Unit Test” (count) does NOT show any more
Secondly, looking at the Measures (menu item) → Coverage (sub-menu item) the Tests sub section and related metrics DO NOT show… hence… I cannot REMOVE it as we need the unit test count.
14:55:42 14:55:42.759 INFO: Sensor PythonXUnitSensor [python]
14:55:42 14:55:42.759 DEBUG: Using pattern 'checkout/canary/test_execution_report.xml' to find reports
14:55:42 14:55:42.844 INFO: Processing report '/home/jenkins/workspace/proj/Testing/canary-lambda/build/checkout/canary/test_execution_report.xml'
14:55:42 14:55:42.847 DEBUG: Trying to find a SonarQube resource for test case 'tests.test_canary_lambda.TestCanaryLambda'
14:55:42 14:55:42.855 DEBUG: Using the key 'tests/test_canary_lambda/TestCanaryLambda.py' to lookup the resource in SonarQube
14:55:42 14:55:42.856 DEBUG: Using the key 'tests/test_canary_lambda.py' to lookup the resource in SonarQube
14:55:42 14:55:42.856 WARN: The resource for 'tests.test_canary_lambda.TestCanaryLambda' is not found, drilling down to the details of this test won't be possible
14:55:42 14:55:42.857 DEBUG: Trying to find a SonarQube resource for test case 'tests.test_canary_lambda.TestCanaryLambda'
14:55:42 14:55:42.857 DEBUG: Using the key 'tests/test_canary_lambda/TestCanaryLambda.py' to lookup the resource in SonarQube
14:55:42 14:55:42.857 DEBUG: Using the key 'tests/test_canary_lambda.py' to lookup the resource in SonarQube
14:55:42 14:55:42.858 WARN: The resource for 'tests.test_canary_lambda.TestCanaryLambda' is not found, drilling down to the details of this test won't be possible
14:55:42 14:55:42.858 DEBUG: Trying to find a SonarQube resource for test case 'tests.test_canary_lambda.TestCanaryLambda'
14:55:42 14:55:42.858 DEBUG: Using the key 'tests/test_canary_lambda/TestCanaryLambda.py' to lookup the resource in SonarQube
14:55:42 14:55:42.858 DEBUG: Using the key 'tests/test_canary_lambda.py' to lookup the resource in SonarQube
14:55:42 14:55:42.859 WARN: The resource for 'tests.test_canary_lambda.TestCanaryLambda' is not found, drilling down to the details of this test won't be possible
14:55:42 14:55:42.859 INFO: Sensor PythonXUnitSensor [python] (done) | time=100ms
For your reference, these metrics can be reviewed as seen in screenshot-3
The Success percentage metric … I was told is derived by the test_success_density which is NOT getting generated as seen in screenshot-1
I can assure you that the test execution report is getting generated accurately and is referenced by sonar.python.xunit.reportPath=checkout/<proj>/test_execution_report.xml
Now, yesterday when you asked me to toggle sonar.python.xunit.skipDetails=false… Neither did it generate the test success rate… rather it DELETED the entire Tests section … Please look at screenshot-4
In addition, the expected unit test count does NOT (seen here as a “-”) show AND we need these counts to be reported … Please look at screenshot-5
If you look at my previous post on this thread, I have attached the Jenkins logs (after updating that property)
If I make sonar.python.xunit.skipDetails=true Both the unit test count (Refer screenshot-6) and the Test section shows albeit without the success % (Refer screenshot-7)
In a nutshell, for a Python project for the SonarQube and scanner version I mentioned at the beginning of the thread… Please help us figure how to derive the Success % in the Tests section…
Thanks for the clarification! I understand the issue now.
It seems like a bug that we don’t record test success density here, one that I don’t think can be worked around (except maybe converting your report to generic test data and importing it as such) and I’ve flagged it for the team.
I appreciate your reply … Could you please direct me on how to convert the report to generic test data ? I looked at the link posted… and I am not sure if the coverage tool in python generates the report in the format shown on that link also, need help to generate the test execution report.
Could you tell me exactly how you are generating the execution report? I would like to reproduce the problem locally, so it would be easier to guide you through a potential solution.
I came back to this thread to try and provide a reproducer for @david.kunzmann and realized that I could no longer reproduce it… I was analyzing Python files with an XUnit test execution report and saw the Success Density figure reported in SonarQube.
Finally, after a closer look, I think you need to do the following:
Set sonar.python.xunit.skipDetails=false (or remove sonar.python.xunit.skipDetails altogether). This is required for the Success % to show, although I’m not sure why (we should investigate on our side)
Make sure that your Python tests are indexed using sonar.tests This should resolve the warnings you found earlier. It should be as simple as setting sonar.tests=tests
What this means in practice is that tests/test_canary_lambda.py is indexed as a test file. With your scanner at DEBUG level (sonar-scanner -X), you’ll see a message like this:
17:16:35.262 DEBUG: 'tests/test_canary_lambda.py' indexed as test with language 'py'
Hi @Colin, @david.kunzmann,
Thanks for reaching out … I got rid of that property all together…
This is what my property file looks like
sonar.projectKey=org.companyName.projectName:AGS_projectName
sonar.projectName=Project name
sonar.sourceEncoding=UTF-8
sonar.sources=checkout/project/lambda_files/ #Path to the source code
sonar.tests=checkout/project/tests/ #Path to the test code
sonar.python.version=3.12
sonar.language=python
sonar.projectVersion=1.0
sonar.python.coverage.reportPaths=checkout/project/coverage.xml
sonar.python.xunit.reportPath=checkout/project/test_execution_report.xml
If I have that property there … this is what it looks like
This is consistent with the behavior I observed the last time I tried this approach (getting rid of that property)
I have shared the configuration details of the sonarQube server version as well as the sonar scanner version at the beginning of this thread.
Another point noteworthy … I am building this project in Jenkins … AND when the project gets cloned off the repo in a directory called checkout/ … for this project … our source code lives within a sub-directory called checkout/canary/lambda-files/*
Hence, it makes it compelling to reference it as the checkout/src/ since it ALWAYS downloads the project into <jenkins_root>/checkout/
It seems that the analyzer is not able to resolve the test files correctly from the information from the test_execution_report.xml file.
The following log line also seems to confirm that:
14:55:42 14:55:42.858 WARN: The resource for 'tests.test_canary_lambda.TestCanaryLambda' is not found, drilling down to the details of this test won't be possible
Looking at the properties you’re using, I assume you’re running the scanner from the <jenkins_root> dir.
With this assumption, this is most likely due to the analyzer not taking the value of the sonar.tests folder into account.Therefore, it will try to resolve the test file relative to the working directory which is where the scanner was started. This is an oversight on our part and I created a ticket for this (SONARPY-2723).
A possible workaround should be running the scanner in the checkout/project folder and adjust the sonar-project.properties file accordingly.
There is also the sonar.projectBaseDir which in theory should have the same effect, but on 9.9 this doesn’t seem to work.
