SonarCloud not picking up duplication exclusions

SonarQube Cloud
, ,
1

Template for a good new topic, formatted with Markdown:

  • ALM used: GitHub
  • CI system used: GitHub Actions
  • Languages of the repository: C#
  • Only if the SonarCloud project is public, the URL:
    [#249] Corrected duplicate issues · Marvin-Brouwer/FluentSerializer@0b3ec8c · GitHub
  • Command used:
     ./.sonar/scanner/dotnet-sonarscanner begin
      /k:"Marvin-Brouwer_FluentSerializer"
      /o:"marvin-brouwer"
      /d:sonar.login="${{ secrets.SONAR_TOKEN }}"
      /d:sonar.host.url="https://sonarcloud.io"
      /s:"${{github.workspace}}/SonarQube.Analysis.xml"
  • Steps to reproduce:
    Run pipeline, get duplication warning.
  • Potential workaround
    N/A
    Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!

Hi,

I have a duplication warning since I have some mirrored extension methods.
However, it’s on purpose and I’d like to exclude them.
I’m using a SonarQube.Analysis.xml file to configure the scanner.
See here: FluentSerializer/SonarQube.Analysis.xml at 0b3ec8c6ba1750ba2471cc4e5e1be726904feb84 · Marvin-Brouwer/FluentSerializer · GitHub
I tried setting the sonar.cpd.exclusions in both the XML and the SonarCloud admin panel.
It just doesn’t seem to exclude the files.
For additional context I also tried starting paths with ./ but that doesn’t seem to make any difference.

2

Hi,

Welcome to the community!

Are any of your exclusions working?

First, SonarQube.Analysis.xml isn’t meant to be project-level but in the scanner directory, and applied to all projects. So I suspect it’s not being read at all.

Second, you shouldn’t need to specify sonar.sources and sonar.tests when using the SonarScanner for .NET; those values are read from your project automatically.

Third, while it’s possibly different in this context, exclusions values are typically comma-delimited lists. I suspect that if this file is being read, your exclusions are being taking as one value each.

Anything set at the analysis level would override what’s set via the UI.

Can you provide your analysis log? That will help understand what values analysis is getting.

The analysis / scanner log is what’s output from the analysis command. Hopefully, the log you provide - redacted as necessary - will include that command as well.

This guide will help you find them.

 
Ann

3

Hi, let me reply one-by-one:

First, SonarQube.Analysis.xml isn’t meant to be project-level but in the scanner directory, and applied to all projects. So I suspect it’s not being read at all.

In this section it does state it as an option, and it does seem to work.
I’m highly in favor of having these kind of configurations in code instead of in a disconnected UI or having a cli command that just keeps on growing.
So, I’d like to keep using this.

Second, you shouldn’t need to specify sonar.sources and sonar.tests when using the SonarScanner for .NET; those values are read from your project automatically.

This is probably just a remnant of me getting code-coverage to work.
That took quite some trial and error, I’ll try removing those after I get the ignores to work.

Third, while it’s possibly different in this context, exclusions values are typically comma-delimited lists. I suspect that if this file is being read, your exclusions are being taking as one value each.

I can give it a go with comma’s, let me do that in the meantime.

Can you provide your analysis log? That will help understand what values analysis is getting.

5_🧰 Setup SonarCloud scanner.txt (17.8 KB)
6_🛠 Build.txt (19.3 KB)
11_🔬 Perform SonarCloud Analysis.txt (3.0 MB)
12_🔬 SonarCloud Quality Gate.txt (4.8 KB)
Alternatively you can find the complete raw log here:
https://pipelines.actions.githubusercontent.com/serviceHosts/f188b72a-12e8-4872-985f-3584fd73a1cf/_apis/pipelines/1/runs/1302/signedlogcontent/2?urlExpires=2023-01-07T10%3A16%3A43.2787295Z&urlSigningMethod=HMACV1&urlSignature=wJiYJ07483FhMkb%2BkRFFf6JTShHAZtVxA07a9BjWXls%3D

I did a quick ctrl+f on “ignore” but I couldn’t find anything. What am I looking for here?

4

In the mean time, I did a run with comma separated values:

https://pipelines.actions.githubusercontent.com/serviceHosts/f188b72a-12e8-4872-985f-3584fd73a1cf/_apis/pipelines/1/runs/1304/signedlogcontent/6?urlExpires=2023-01-07T10%3A28%3A51.0641522Z&urlSigningMethod=HMACV1&urlSignature=VbsVzMLdMsTFBKvWnxYMxG2ri46sv3ESEeliLofKAcg%3D
Just to be thorough I also tried it without comments:

https://pipelines.actions.githubusercontent.com/serviceHosts/f188b72a-12e8-4872-985f-3584fd73a1cf/_apis/pipelines/1/runs/1306/signedlogcontent/2?urlExpires=2023-01-07T10%3A36%3A47.2958389Z&urlSigningMethod=HMACV1&urlSignature=M6PKNee4aKtgnPhH%2FkJ%2BClMf8ks%2BYtbYpvuz%2F0Tmk8E%3D

It didn’t seem to make a difference.

5

Hi,

I can’t read log 11, and the raw log link has expired. Can you re-post?

 
Ann

6

Oh that’s annoying.

If you go here you can generate a raw log by clicking on the cogwheel:

Could you maybe tell me what I’m looking for in the logs so I can have a look myself too?

Thanks.

7

Here’s what we’re looking for:

10:12:48.303 INFO: Excluded sources: **/build-wrapper-dump.json, FluentSerializer.Xml.Converter.DefaultXml/**/*.cs

10:12:48.303 INFO: Excluded tests: **/GlobalSuppressions.cs

10:12:48.304 INFO: Excluded sources for coverage: **/GlobalSuppressions.cs, ./FluentSerializer.Json.Converter.DefaultJson/**/*.cs, ./FluentSerializer.Xml.Converter.DefaultXml/**/*.cs, *.Tests/**/*.cs

10:12:48.304 INFO: Excluded sources for duplication: **/GlobalSuppressions.cs

**/Extensions/Use*Extensions.cs

Note the linebreak in the duplication expression. What was the result when you tried it with a comma?

 
Ann

8

This is what the value looks like with comma’s and without comments in the xml:

2023-01-07T10:35:08.6188627Z 10:35:08.607 INFO:   Excluded sources: **/build-wrapper-dump.json, **/GlobalSuppressions.cs, *.Tests/**/*.cs, FluentSerializer.Json.Converter.DefaultJson/**/*.cs, FluentSerializer.Xml.Converter.DefaultXml/**/*.cs
2023-01-07T10:35:08.6189204Z 10:35:08.607 INFO:   Excluded tests: **/GlobalSuppressions.cs
2023-01-07T10:35:08.6189986Z 10:35:08.607 INFO:   Excluded sources for coverage: **/GlobalSuppressions.cs, ./FluentSerializer.Json.Converter.DefaultJson/**/*.cs, ./FluentSerializer.Xml.Converter.DefaultXml/**/*.cs, *.Tests/**/*.cs
2023-01-07T10:35:08.6190624Z 10:35:08.607 INFO:   Excluded sources for duplication: **/GlobalSuppressions.cs, **/Extensions/Use*Extensions.cs

Looks fine to me, but the quality gate is still failing on the duplications.

9

Oh well I think removing the comments actually solved it.
I’ll have another go without the comma’s to see if that makes a difference.

10

Okay, another update.
It seems to work perfectly with comments as long as it has comma’s.
I must’ve just missed that there’s something else failing the quality gate.
Sorry for that.

11

I do have yet another hiccup.
Currently I have a duplication in the following file: FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs
I’d also like to exclude this one, because there’s no real way to unduplicate it.
I added this to my xml:

	<Property Name="sonar.cpd.exclusions">
		**/GlobalSuppressions.cs,
		<!-- These extensions are purely for consumers of the package and will be duplicate by design -->
		**/Extensions/Use*Extensions.cs,
		<!-- This converter is different but very similar to the ConvertibleConverter, this is by design -->
		FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs
	</Property>

However now it appears that it’s only taking the last line:

2023-01-09T19:01:16.8716112Z INFO: Project configuration:
2023-01-09T19:01:16.8717131Z INFO:   Excluded sources: **/build-wrapper-dump.json, FluentSerializer.Xml.Converter.DefaultXml/**/*.cs
2023-01-09T19:01:16.8787928Z INFO:   Excluded tests: **/GlobalSuppressions.cs
2023-01-09T19:01:16.8789649Z INFO:   Excluded sources for duplication: FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs

This only appears to happen when I add a path instead of a wildcard.
What could be causing this?

12

Hi,

What happens if you drop the comments?

 
Ann

13

No difference, but this seems to fix it:

	<Property Name="sonar.cpd.exclusions">
		**/GlobalSuppressions.cs,
		<!-- These extensions are purely for consumers of the package and will be duplicate by design -->
		**/Extensions/Use*Extensions.cs,
		<!-- This converter is different but very similar to the ConvertibleConverter, this is by design -->
		*/FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs
	</Property>

Which has the following log:

2023-01-09T19:11:52.6025828Z INFO: Project configuration:
2023-01-09T19:11:52.6027911Z INFO:   Excluded sources: **/build-wrapper-dump.json, **/GlobalSuppressions.cs, FluentSerializer.Json.Converter.DefaultJson/**/*.cs, FluentSerializer.Xml.Converter.DefaultXml/**/*.cs
2023-01-09T19:11:52.6032048Z INFO:   Excluded tests: **/GlobalSuppressions.cs
2023-01-09T19:11:52.6033450Z INFO:   Excluded sources for duplication: **/GlobalSuppressions.cs, **/Extensions/Use*Extensions.cs, FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs
2023-01-09T19:11:52.7539747Z INFO: Indexing files of module 'FluentSerializer.Json.Converter.DefaultJson'

Which is strange because according to the sonarcloud UI the path in the scan is: FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs next to the copy icon.
In fact, I’ve used the copy icon there.

Technically the path is ./src/FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs however, I’m aware the sonarscanner scopes to ./src/ on dotnet by default so it shouldn’t matter.

Sadly, this doesn’t work since the wildcard path doesn’t match like this.

Apart from all of that, it just appears that adding a path without a wildcard breaks the config.

For completion’s sake I also tried this:

	<Property Name="sonar.cpd.exclusions">
		**/GlobalSuppressions.cs,
		<!-- These extensions are purely for consumers of the package and will be duplicate by design -->
		**/Extensions/Use*Extensions.cs,
		<!-- This converter is different but very similar to the ConvertibleConverter, this is by design -->
		./src/FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs
	</Property>

This appears to also only use the last line:

2023-01-09T19:01:16.8716112Z INFO: Project configuration:
2023-01-09T19:01:16.8717131Z INFO:   Excluded sources: **/build-wrapper-dump.json, FluentSerializer.Xml.Converter.DefaultXml/**/*.cs
2023-01-09T19:01:16.8787928Z INFO:   Excluded tests: **/GlobalSuppressions.cs
2023-01-09T19:01:16.8789649Z INFO:   Excluded sources for duplication: FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs

So I’m pretty sure it only uses the path if you add a path without a pattern. Maybe this is on purpose, but it kind of seems like a bug to me.

15

Okay now it gets interesting.

I did a run like this:

	<Property Name="sonar.cpd.exclusions">
		**/GlobalSuppressions.cs,
		**/Extensions/Use*Extensions.cs,
		/FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs
	</Property>

Success

Then I did a run like this:


	<Property Name="sonar.cpd.exclusions">
		**/GlobalSuppressions.cs,
		<!-- These extensions are purely for consumers of the package and will be duplicate by design -->
		**/Extensions/Use*Extensions.cs,
		<!-- This converter is different but very similar to the ConvertibleConverter, this is by design -->
		/FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs
	</Property>

Faillure

And again with this:

 Excluded tests: **/GlobalSuppressions.cs
INFO:   Excluded sources for duplication: /FluentSerializer.Xml/Converting/Converters/FormattableConverter.cs

It appears to me that prepending the absolute path with / solves my issues, but then ALSO having comments breaks it again.

However, having comments without the absolute path, like in the other exclusions cause no issues:

	<Property Name="sonar.exclusions">
		**/GlobalSuppressions.cs,
		<!-- Temporary exclusion, should be re-added in #232 -->
		FluentSerializer.Json.Converter.DefaultJson/**/*.cs,
		<!-- Temporary exclusion, should be re-added in #232 -->
		FluentSerializer.Xml.Converter.DefaultXml/**/*.cs
	</Property>

Since I haven’t touched these between tests.

16

I played around with the order a bit, but that doesn’t seem to help.
It does change the outcome a bit.
If I put a pattern at the end and there’s a path at the start I only get the pattern, so it appears to me that the path being in there just makes the config reading logic only use the last entry.

17

Hi,

To be clear, this functionality was originally built to be supplied via a UI, E.G.:

Selection_686
Selection_686844×312 25.2 KB

(Yes, this is coverage, not duplications, but the idea’s the same.)

Then it was ported to be settable via analysis properties. Now, under the covers, long-previous experimentation has shown me that all exclusions appear to be structured as multi-value properties, like this:

Selection_687
Selection_687914×357 35.7 KB

So some “magic” was applied to allow you to set some exclusions (not that^ one) as a comma-delimited list in a Java key/value property. I.e. one per line.

SonarScanner for .NET is, among other things, a wrapper around the SonarScanner CLI. I suspect (but don’t know for certain) that the values are just pulled straight out of the XML and stuffed into Java properties before being passed to the underlying implementation. So I’m not shocked that when you throw in “exotic” elements like comments and linebreaks it acts a bit odd. This was never meant to be the primary means of setting this stuff. (Ref. our fancy UI^.)

That said, I’ve flagged this for team attention and I’m sure they’ll find your iterations on this very helpful when they take up the topic.

 
:smiley:
Ann

1 Like
18

Ah yes that makes a lot of sense to me.
I figured it’s important to me that the comments are in there. But since it’s not supposed to be a large list of exclusions anyway, I’m happy with the comments next to the nodes for now.
This is what it looks like now: FluentSerializer/SonarQube.Analysis.xml at e01c2e1834728c298ba2bf171a4c74fff7e4819e · Marvin-Brouwer/FluentSerializer · GitHub

I realize that I’m somewhat misusing this for my own needs. But, I hope you do understand why I use it this way, and maybe that this makes you consider using an XML parser, or providing a JSONC config option.
If not I guess the .sonarsource would also suffice, if I can add comments there.
I’d prefer to point to the file explicitly in the CMD though, and perhaps multiline support for exclusions.

I did a last check, prepending the paths with ./ breaks it and removing the commas breaks it.
I hope this helps.

Thanks for your help!

1 Like
19

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.