SonarPHP doesn't analyze php-unit tests with dataProvider

SonarCloud
1
  • SonarQube Version 7.3 (build 15553), Scanner 3.2.0.1227, SonarPHP 2.14.0.3569, php 7.2.10, php-unit 7.3.5
  • Sonarphp doesn’t analyze php-unit tests with dataProvider.
    In logs i have warnings:
WARN: Test cases must always be descendants of a file-based suite, skipping : Adsterra\Tests\Unit\Core\Helper\ArrHelperTest.testIsStrKeysArray with data set #0 in Adsterra\Tests\Unit\Core\Helper\ArrHelperTest::testIsStrKeysArray
WARN: Test cases must always be descendants of a file-based suite, skipping : Adsterra\Tests\Unit\Core\Helper\ArrHelperTest.testIsStrKeysArray with data set #1 in Adsterra\Tests\Unit\Core\Helper\ArrHelperTest::testIsStrKeysArray
WARN: Test cases must always be descendants of a file-based suite, skipping : Adsterra\Tests\Unit\Core\Helper\ArrHelperTest.testIsStrKeysArray with data set #2 in Adsterra\Tests\Unit\Core\Helper\ArrHelperTest::testIsStrKeysArray
WARN: Test cases must always be descendants of a file-based suite, skipping : Adsterra\Tests\Unit\Core\Helper\ArrHelperTest.testIsStrKeysArray with data set #3 in Adsterra\Tests\Unit\Core\Helper\ArrHelperTest::testIsStrKeysArray
  • steps to reproduce
  1. Create unit test on php with dataProvider;
  2. Run this test;
  3. Scan reults test xml file use SonarQube scanner;
1 Like
2

Ilia,

Is your issue that the tests are not being analyzed, or that the test results are not being imported to SonarQube. Simply scanning a test results file is not enough, you will need to follow the SonarPHP documentation for PHP Unit Test and Coverage Results Import.

Colin

3

Hi Ilia, have your found a solution to this problem by any chance? As I understood, it has to do with the parametrised test cases being generated inside the cascaded testsuite elements in a test report file, and SonarScanner does not like it. But I can’t figure out how to fix that.

4

Hi,

I have a similar issue on a sonarcloud project: https://sonarcloud.io/dashboard?id=Evoluty_google-maps-apis-php-client

@Colin You can check it on travis https://travis-ci.org/Evoluty/google-maps-apis-php-client/jobs/506018115#L581 and the related test is https://github.com/Evoluty/google-maps-apis-php-client/blob/master/tests/ApiExceptionsTest.php

5

“INFO: Analyzing PHPUnit test report: tests/logs/logfile.xml”

Example xml that will show warnings

  • for //testsuite/testsuite/testsuite/testcase
  • but read //testsuite/testsuite/testcase
  • because there is no file attribute on the nested testsuite node
<testsuites>
  <testsuite assertions="3885" errors="0" failures="0" name="classes" skipped="36" tests="1926" time="0.778806">
    <testsuite name="tests\SomeTest" file="/home/.../SomeTest.php" tests="11" assertions="22" errors="0" failures="0" skipped="0" time="0.075715">
      <testcase name="someTest" class="tests\SomeTest" classname="tests.SomeTest" file="/home/.../SomeTest.php" line="17" assertions="3" time="0.032633"/>
      <testsuite name="tests\SomeTest::otherTest" tests="7" assertions="14" errors="0" failures="0" skipped="0" time="0.032672">
        <testcase name="otherTest with data set #0" class="tests\SomeTest" classname="tests.SomeTest" file="/home/.../SomeTest.php" line="87" assertions="2" time="0.004864"/>

Simple python script that you can run on the logfile.xml to patch the file attribute on testsuite nodes (workaround):

7

Greetings guys –

Thanks for reaching out on this (and thanks @black-silence for posting a workaround). Can you clarify the issue that you think should be solved?

  • Warnings appearing in the analysis logs
  • Test Results not appearing in SonarQube / metrics not being calculated correctly
  • …?

Best regards,

Colin

8

Hi Colin,

IMO all testcase nodes in my example are descendants of a file-based suite, just not direct descendants.

There should not be a warning and the testcase should not be ignored if there is a testsuite with a file attribute in ancestors.
Maybe even ignore the testsuite file attribute if the testcase node has a file attribute, could be easier to code. (I am assuming the attribute value is the same on all child nodes for any testsuite that has a file attribute.)

This would include all test cases in the analysis regardless of the nesting level.

Best regards,
bs

9

Without this gist, the test coverage is not imported at all into SonarQube. You can see some of the output here: https://phabricator.wikimedia.org/T208522, but basically it’s a lot of “Test cases must always be descendants of a file-based suite, skipping : AuthManagerStatsdHandlerTest.testHandle with data set “no event” in AuthManagerStatsdHandlerTest::testHandle”.

With this gist, I get a bit closer but if a test has a data provider then I’ll end up with:

java.lang.UnsupportedOperationException: Can not add the same measure twice on tests/phpunit/MentorTest.php: DefaultMeasure[component=tests/phpunit/MentorTest.php,metric=Metric[id=<null>,key=skipped_tests,description=Number of skipped unit tests,type=INT,direction=-1,domain=Coverage,name=Skipped Unit Tests,qualitative=true,userManaged=false,enabled=true,worstValue=<null>,bestValue=0.0,optimizedBestValue=true,hidden=false,deleteHistoricalData=false,decimalScale=<null>],value=0,fromCore=false]

I’ve made a slightly modified version of this gist (https://gerrit.wikimedia.org/r/c/integration/config/+/508019/9/dockerfiles/quibble-stretch-php70/phpunit-junit-edit.py) which will result in data provider test coverage in junit.xml to be ignored, which is better than the other two outcomes I listed above.

10

Hi,

I’ve created a ticket in SonarPHP to support such unit test reports (https://jira.sonarsource.com/browse/SONARPHP-927)

1 Like
11

@Lena do you have a sense of where in the roadmap this issue might fit in? This bug makes SonarPHP a hard sell to many people with respect to code coverage.

12

Sorry, no estimation yet.

13

I have successfully fixed this issue by updating PHP Sensor to 3.5.0.5655. To fix the problem with relative paths in junit-logfile.xml (messages like WARN: Could not resolve 18 file paths in coverage.xml, first unresolved path) I used the snippet below to fix the generated report:

sed -i "s|${PWD}/||g" junit-logfile.xml

1 Like
14

Hello,

I don t know if something supposedly fixed this already.
But to answer @Colin (I am sorry if you completely forgot about these) questions, in my case :

  • I still have warnings
  • the coverage seems imported : I only did simple tests (single line function, and a simple if / else function), but the measures part is showing the coverage, as well as the pull request decoration
  • the tests results are imported : success and failures are updated in the measures part, I did not test errors and skipped tests. The number of unit test by file is correct (only tested with dataProviders doing 2 iterations).

So my only question is why is there a warning in the logs saying the test is skipped ? Or am I missing something else that should be imported ?

This warning is disturbing because we think we have to do something, whereas maybe we do not…

I m using phpunit 9.4.1 (but the junit xml file is still the same), sonarQube 8.3.1, SonarPHP 3.3.0.5166.

If you need more information, let me know. (I did not open a new thread because I think it is really the same problem, except the only problem is the warning, but I might be wrong).

Also, the link in your first answer is outdated, yet many people are still clicking on it, or so it seems?

Thank you

[Edited]
Sorry, I should have read everything, I did not understand Fernando Torres message… so this should be fixed in SonarPHP 3.4, as described in https://jira.sonarsource.com/browse/SONARPHP-927.