We’re using SonarCloud with Bitbucket pipelines for analysis; our projects are predominantly PHP. Starting late last week, SonarCloud began reporting that our test files lacked test coverage despite having sonar.test.exclusions set in our sonar-project.properties file:
The trouble seems to be with tests that are nested a few directories below tests, despite the ** in the exclusions property.
I tried setting sonar.tests=tests/ in place of the sonar.test.exclusions setting, but that prevented SonarCloud from showing any coverage measure. Adding tests/**/* to sonar.coverage.exclusions seems to be my only choice.
Is it expected that my tests need to be explicitly excluded from coverage, or should either sonar.tests or sonar.test.exclusions have accomplished what sonar.coverage.exclusions=tests/**/* would?
The sudden change in the behavior is indeed strange, and I don’t have an explanation for it without more context.
I’m unsure if the meaning of sonar.test.exclusions is clear. Just to be certain, let me try to explain it:
For the scanner there are two types of files: Main source files (specified by sonar.sources), and test files (specified by sonar.tests).
Coverage and “normal” rules’ findings are extracted for the main source files. Now, for PHP, if sonar.tests is not set, everything under the base directory will be considered a main source file (i.e., sonar.sources defaults to . in that case)., and coverage will be calculated for it.
If you specify sonar.tests, you will also need to specify sonar.sources. For example:
sonar.sources=src
sonar.tests=tests
Now, once you mark a directory as the one containing your test code, coverage and “normal” rules’ findings will not be extracted for files in it. Now, what is the effect of sonar.test.exclusions in that case? I kept saying ““normal” rules’ findings” because we also have some rules that are specific to test code (see rules). With sonar.test.exclusions you can ignore findings of those rules (if activated) for specific files.
Please try to use sonar.tests=tests/, and also specify your main source files with sonar.sources=.... That should hopefully resolve your problem.
I tried updating our config to include the following:
sonar.sources=.
sonar.tests=tests
Unfortunately, that results in an error:
File tests/bootstrap.php can’t be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files
Our projects have source files in the root directory and various subdirectories, while tests are generally in the tests directory. Do we need to list every root file, as well as every subdirectory, in sonar.sources?
On a related note, does sonar.tests support wildcards? In one project, source files are in subdirectories according to the package they represent, with tests in a tests directory within each subdirectory. I’m wondering if I can set sonar.tests=**/tests for this project, or if I need to enumerate the relative paths to every test directory.
If the test and source directories are not clearly separated at a specific level, that makes it a bit more tricky. We’ll need to play around with exclusions/inclusions to achieve what the error message you’ve gotten indicates: disjoint sets for main and test files.
This is considering that your tests will always be in some tests sub-directory. In case there are other cases, you might need to add to sonar.exclusions and sonar.test.inclusions (these are comma-separated lists).
Unfortunately not. This post explains very well why and how the above properties work together.
