SonarCloud test inclusions causes source exclusions

  • ALM used: Azure DevOps

  • CI system used: Azure DevOps

  • SonarCloud project setup: Monorepo

  • Relevant applicable language: C#

  • Scanner “prepare” task:

- task: SonarCloudPrepare@1
  inputs:
    SonarCloud: redacted
    organization: redacted
    projectKey: redacted
    projectName: redacted
    scannerMode: 'MSBuild'
    extraProperties: |
      sonar.exclusions=$(sonarCloudExclusions)
      sonar.inclusions=$(sonarCloudInclusions)

The value of $(sonarCloudExclusions) is populated from proprietary per-project config.
The value of $(sonarCloudInclusions) is populated from proprietary per-project config, prefixed with ($slnFolder)/**/*,. Reason for this: Our monorepo is large, with many libraries shared as source, and this ensures that shared code within the solution does not get scanned multiple times - within each consuming solution that has a SonarCloud project.

The solution above achieved its target that no source file was scanned twice. However, a few days ago we recognised that tests applied to said shared code are not excluded. We referred to the docs for file exclusion and inclusion. The usage example attempts to distinguish between source and tests. We have no need of this, because the MSBuild flavour does this excellently automatically, correctly populating the values of sonar.sources and sonar.tests within each module or each project. For us, the aim of these inclusions/exclusions was not to distinguish source from tests, but rather to narrow down the overall scope to certain folders for source and tests alike, so the additional properties for tests inclusions and exclusions seemed appropriate to use, since the docs specifically mention that these value filter the initial scope (which is untouched). So we’ve added the same inclusions and exclusions for tests as for sources:

    extraProperties: |
      sonar.exclusions=$(sonarCloudExclusions)
      sonar.inclusions=$(sonarCloudInclusions)
      sonar.test.exclusions=$(sonarCloudExclusions)
      sonar.test.inclusions=$(sonarCloudInclusions)

Expected result 1: All tests outside the solution folder are removed from each SonarCloud project.
Observed result 1:All tests outside the solution folder are removed from each SonarCloud project. :white_check_mark:

Expected result 2: All sources remain untouched.
Observed result 2: All sources, everywhere, are removed from each SonarCloud project. :x:

===

Additional info:

  • Prepare Analysis Configuration task version: 1.33.0

  • Run code analysis task version: 1.36.0

  • Comparing pipeline logs of two runs, before and after the change, reveal that the new sonar.test.inclusions seems to have been populated also into sonar.exclusions:


    This example project has no configured exclusions, so we expected to see only new test inclusions in the logs.

  • Comparing Project → Background Tasks → Show SonarScanner Context before and after showed all identical parameters, including sonar.sources and sonar.tests for each module, except for the expected additional project-level property sonar.test.inclusions, populated with the correct value. Specifically no change in sonar.inclusions, and in this particular project there are no sonar.exclusions and no sonar.test.exclusions.

===

Would this be an expected behaviour, or a bug?
Am I using an incorrect setup for my goal?

For any solution / workaround you may have for me, please keep in mind that I have a high preference to keep using the scanner’s existing logic for distinguishing sources from tests. If I have to configure this for every project manually - this would be error prone and a high overhead, since our monorepo is very large and very versatile.

Hi,

Welcome to the community!

  1. $(sonarCloudExclusions) = proprietary per-project config.
  2. $(sonarCloudInclusions) = ($slnFolder)/**/*, + proprietary per-project config.
  3. sonar.exclusions=$(sonarCloudExclusions)
  4. sonar.inclusions=$(sonarCloudInclusions)
  5. sonar.test.exclusions=$(sonarCloudExclusions)
  6. sonar.test.inclusions=$(sonarCloudInclusions)

There’s a lot going on here.

Let’s start with #2. You set a pattern to… everything. Plus some stuff.
Then at #4 you apply that as your source inclusion. Okay, so everything is source files.
Then at #6 you apply that same pattern as your test inclusions. So everything is test files. Too.

The thing is, a file must be one or the other.

The surprise here for me is that analysis swallows what you’ve configured. Normally it would error out, saying a file can’t be indexed twice.

You should take a step back and look again at your SonarScanner Context files to make sure each property lists what it should and only what it should.

In general, my advice here is to use the fewest possible properties to achieve the goal. Combining inclusions with exclusions can really confuse the picture. In most cases, only one (or the other) is really needed.

 
HTH,
Ann

Hi there Ann,

Thanks so much for trying to dig into this. Indeed there’s a lot going on. I hope I can shed more light:

Let’s start with #2. You set a pattern to… everything. Plus some stuff.

A key piece of information is this is a monorepo. In a monorepo, a solution is a very small moving part, and certainly not “everything”. I set a pattern to include only the solution scope in the corresponding SonarCloud project. There is one such project per each solution.

Another key piece of information is that these two properties are only applied into 3, 4, 5 and 6. These four properties are what SonarCloud documents as “filters”. See docs for analysis scope → file exclusion and inclusion, excerpt:

Exclusions and inclusions apply on top of the sonar.sources and sonar.tests settings. Both the exclusion and inclusion parameters act as filters. They only ever reduce the number of files in the analyzable set, they never add to the set.

As mentioned in my original post (under additional info, last bullet), sonar.sources and sonar.tests are correctly populated by SonarScanner’s MSBuild flavour for every single module. By “correctly” I mean that SonarScanner is being awesome and fills these properties with a full list of source files and test files, respectively, for each module. This means that users like myself can apply seemingly-large filters on top of them, and all they should do is remove any unnecessary files. I believe this addresses your comment:

The thing is, a file must be one or the other.

So I fully understand your confusion (I went through the same stage myself :slight_smile: ).
Does this new information change how you see this?

Lastly, regarding your suggestion:

In general, my advice here is to use the fewest possible properties to achieve the goal.

Since inclusions and exclusions are just filtering out from the scope, I find they should be able to live together pretty nicely, each one further filtering out the result set. Having said that, even if I were to remove the sonar.exclusions and sonar.test.exclusions from my solution, I don’t suppose this will help me, because it seems to me that the problematic part is populating both of the inclusions for sources and for tests with the same value (which is exactly what we need). So far, our configured exclusions are empty, so I don’t suppose these caused any issue. So I still need a solution how to achieve my goal. Makes sense?

Hi,

Yes, I understand that a project is only a subset of a monorepo.

However, I was talking about the pattern you set:

Specifically *($slnFolder)/**/* means “everything in $slnFolder” (assuming $slnFolder is properly interpolated). ** basically means every directory, recursively, and * means every file.

So you start by creating a pattern that encompasses every file. And then you apply that pattern to both source inclusions and test inclusions.

Now, it’s not clear to me the relationship between $slnFolder and

If $slnFolder is a subset, then - while I don’t understand why you’re building everything when you only want a subset - it makes sense to apply inclusions to narrow the set of source files. But it still doesn’t make sense to apply the same inclusions to both sources and tests. Which you seem to agree with?

 
Ann

Hi Ann,

I’m sorry I’m not getting my point clearly across. I’ll try give a clear example, with images:

For the following folder structure:
image

And the following “MySolution” solution:
image

We would like only these source and test projects to appear in the SonarCloud project for “MySolution”:
image

When setting sonar.inclusions=MySolution/**/*, we get the correct source libraries filtered out:
image

But when setting both sonar.inclusions=MySolution/**/* and sonar.test.inclusions=MySolution/**/*, we get only one test library remaining in the entire SonarCloud project, and all other libraries filtered out:
image

The (much simplified) SonarScanner context looks as expected to us, with SonarScanner MSBuild automatically populating sonar.sources and sonar.tests correctly for each module, and the two filters appearing neatly on project level (above):

So we are wondering:

  1. Why does this context not produce the expected result?
  2. How should we achieve the expected result?

Hi,

Can we back up to what’s being built? Are you building the entire monorepo or just a subset?

 
Ann

As far as SonarCloud is concerned we are building one solution at a time.

I seem to notice a newly submitted topic identical to mine:

Hi,

Okay, so you’re building only the solution you want to analyze. And, as you say, SonarScanner for .NET does a lovely job of identifying which files are source files and which files are test files.

So… why did you go down this road of inclusions and exclusions? What itch were you trying to scratch?

 
Ann

Without inclusion/exclusion filters all source and test files in the solution appear in the SonarCloud project.

In my example, those would include ConsumedService.Contract, Global.SharedLibrary and Global.SharedLibrary.Tests, which we do not want.

Hi,

Okay, so you’re only building a sub-directory, but because that solution references a solution in a sister sub-directory(?) those files automatically get included in analysis. Right?

Your context screenshot shows a sonar.projectBaseDir of C:\MyMonorepo. I.e. the parent directory. Did you set that explicitly? Because it should probably be set to C:\MyMonorepo\MySolution.

 
Ann

Yes, that’s right.

No, I didn’t set it explicitly, perhaps that’s the default for a monorepo.

  1. I don’t think we want that, because we only want to filter the mentioned libraries by default. As mentioned, we allow additional inclusions from across the monorepo via configuration. I suspect that if we set sonar.projectBaseDir=C:\MyMonorepo\MySolution, then we may not be able to include anything outside of it, correct?

  2. Reading again the docs for analysis parameters I actually doubt that would change anything for us:

sonar.sources docs

sonar.sources

Comma-separated paths to directories containing main source files.

Default: for Maven, Gradle, and .NET projects, read from the build system, otherwise, if neither sonar.sources nor sonar.tests is provided, the project base directory.

If I am reading this correctly, it seems that for .NET projects sonar.sources gets populated explicitly, and therefore doesn’t use sonar.projectBaseDir in any way.

Also, on that same page, the docs for sonar.projectBaseDir don’t mention anything about filtering source files:

sonar.projectBaseDir docs

sonar.projectBaseDir

Use this property when you need the analysis to take place in a directory other than the one from which it was launched. For example, analysis begins from jenkins/jobs/myjob/workspace but the files to be analyzed are in ftpdrop/cobol/project1. The path may be relative or absolute. Specify not the source directory, but some ancestor of the source directory. The value specified here becomes the new “analysis directory”, and other paths are then specified as though the analysis were starting from that specified value. Note that the analysis process will need write permissions in this directory; it is where the sonar.working.directory will be created.

Default: The directory from which the SonarScanner is launched.

If either (1) or (2) are correct then we need to find a different solution.

Hi,

Maybe you could try setting an explicit exclusion in the projects you want to omit:

<PropertyGroup>
  <SonarQubeExclude>true</SonarQubeExclude>
</PropertyGroup>

 
Ann

I’m afraid that will not work. We can’t omit these shared projects altogether - they do have their own solutions and their own SonarCloud projects that analyse them. We just don’t want them to be re-analysed in every solution that consumes them as well.

Hi,

Per the docs

If you want to analyze a monorepo that contains more than one project, you need to ensure that you specify the paths to each project for analysis in your azure-pipelines.yml file.

For other CIs this take the shape of specifying the project base dir. For Azure Pipelines (altho it’s not clear to me that’s relevant) you specify the solution file.

 
 
But let’s go back to the beginning:

So in real terms:

sonar.inclusions=MySolution/**/*,&etc
sonar.test.inclusions=MySolution/**/*,&etc

i.e. every file is included as a source file and as a test file.

So then

I was unaware of a code change that appended test inclusions to file exclusions. But it makes sense since, as I’ve said, a file may be only one or the other. If it’s a test file, it can’t be a source file.

I suggest you use as few properties as possible to get your desired result. Ideally you let analysis set the sources and test sets for you automatically. But if you feel you must edit those automatic sets, then

  • look at narrowing via setting the base directory
  • use as few inclusion/exclusion properties as possible.

 
Ann

Yes, our pipeline looks identical to this one. These docs don’t mention sonar.projectBaseDir. I can try setting it to the solution folder, but as I said - if it always excludes everything outside that folder then it isn’t flexible enough for us.

I must disagree. You’re suggesting that the filter defines the files within it as source or test, but a filter is just a filter. If I define:

  • “All files on my disk with extension .abc are text files”
  • “All files on my disk with extension .def are image files”

I can still filter them both to the same folder C:\MyBook\**\*, to get my entire book contents. Even though I use the same filter for both, this filter doesn’t suddenly redefine text as images or vice versa, any more than a solution-directory filter should redefine sources as tests or vice versa.

Thank you for your suggestions. Other than trying setting the sonar.projectBaseDir, which I’ve replied above why I don’t expect it to be a workable solution for us (I’m happy to hear what you think about that), I’m not sure what else to try. I’m happy to use as few settings as possible, I just don’t know how to do that in a way that satisfies our needs. If you have any more concrete suggestions I’ll be happy to hear them!

MSBuild properties can be set in multiple places - as command line arguments, in the project file, in environment variables etc. MSBuild defines an order of precedence to handle the same property being set in multiple places, the highest priority being command line arguments.

So you should be able to set <SonarQubeExclude>true</SonarQubeExclude> in the project files of the shared projects, so they won’t be analysed by default.

Then, in the build pipelines where you do want them to be analysed, override the property from the MSBuild/dotnet command line:
e.g. msbuild ..... /p:SonarQubeExclude=false

1 Like

Hi Duncan,

Thanks for taking a look.

We run msbuild command per solution, I don’t think it is possible to pass /p: for only some of the projects… is it?

@tsemer I was assuming that there was one solution that referenced your shared projects that you would want to analyse, so you would set the command line property when building that solution, so all of the shared projects would be analysed as part of it.

If you want to be more granular about analysing specific projects when building a solution then you could use MSBuild conditions to conditionally set/override the SonarQubeExclude property.

For example, imagine multiple solutions contain shared projects A and B.
Add the following to project file A:

<PropertyGroup>
  <!-- Don't analyse by default -->
  <SonarQubeExclude>true<SonarQubeExclude>
  <SonarQubeExclude Condition=" $(AnalyseProjectA)=='true' ">false</SonarQubeExclude>
</PropertyGroup>

… and the equivalent to B:

<PropertyGroup>
  <!-- Don't analyse by default -->
  <SonarQubeExclude>true<SonarQubeExclude>
  <SonarQubeExclude Condition=" $(AnalyseProjectB)=='true' ">false</SonarQubeExclude>
</PropertyGroup>

By default, A and B will not be analysed. If you are building a solution and you do want to analyse a shared project, simply set the appropriate property on the command line e.g. /p:AnalyseProjectA=true

From the point of view of the Sonar Scanner for .NET, the situation is very simple: if the MSBuild property SonarQubeExclude=true for a project then that project won’t be analysed.
MSBuild gives you lots of flexibility in how you configure your projects, and you can use any of its features to set the property appropriately.

Thanks @duncanp!

That workaround could potentially achieve what we want. It has several noticeable drawbacks:

Architectural:

  • Tightly couples our C# projects to SonarCloud
  • Spreads SonarCloud configuration into 2 locations (config file and csproj file), making it harder for our devs to get “the full picture at a glance”.

Development and maintenance costs:

  • Large amount of repo-wide changes - we currently have 953 C# project files in this repository alone, roughly 200 of which are shared, spread across all solutions (e.g. service contract libraries).
  • We don’t have a pipeline per solution, we have generic pipelines maintained by our CI/CD team reading the sonarcloud configuration, so this needs to be done in a generic way. This unfortunately means we will need to create new configuration structure using project names, and include the AnalyseProjectX configuration, project by project, for all solutions, effectively recreating an entire “inclusions / exclusions” mechanism. A mechanism including wildcards will be even more complex (shared projects are not guaranteed to have anything in particular in common). This also requires us to write and maintain custom pipeline code to “understand” said config file and parse it into an MSBuild command.

Unfortunately it seems to me we are at a place where the issue itself (shared tests appearing in unrelated SonarCloud projects, without the tested code), has less drawbacks - at least so far - than the cost of any of the proposed workarounds.

I’m still happy to hear other solutions, but by now we seem to understand the options in front of us, and have not much faith in resolving this in a satisfying way :frowning: . I wish I could find a way to leave the inclusions and exclusions exactly as configured, but I can’t seem to find any such way. Barring this, I wish I could open a proper issue or bug report to hear what the product team makes of this, but I just found out that as a paying customer we get no product support except this community forum unless we pay a lot more money for just the support component, so I suppose I can’t even do that.

My sincere thanks to you both for looking into this and sharing your perspectives.