Second sonar quality gate check is over 1000% slower than first for no apparent reason

SonarQube
12

Hi,

Not necessarily. That’s why I keep asking about file sizes.

Have you successfully excluded that massive, generated file? And if so, was there any impact?

If not, do you need help crafting the exclusion pattern?

 
Ann

13

So far the patterns I tried seem to have no effect at all. Even excluding everything seemed to have no effect. I currently have the suspission ** matches only directories and not the content within those directories. If that is true the patterns I posted before could never have worked.

14

Hi,

Yep. You would need to exclude **/enormous-generated-file.ext. Could you try that & get back to me with the results?

 
Ann

15

ok, I tried now to exclude everything with:

<SonarQubeAnalysisProperties  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://www.sonarsource.com/msbuild/integration/2015/1">
  <Property Name="sonar.exclusions">**/*</Property>
</SonarQubeAnalysisProperties>

plus:
Project Setting Source File Exclusions **/*

The projects have now 0 lines of code but the sonar quality gate check still needs so super long.

One thing that is still a bit strange: The duplicate statistic refers to 400k lines. Clicking on it that there are serval really huge files in TestResults folders. For example in each project there is a huge coverage.opencover.xml file with over 100k each.

Probably from some earlier tests the code duplication exclusion was already set to **/*.* in the project settings. Strangly it had no effect.

16

Hi,

Can you run it one more time, please? The first time you set **/* SonarQube still had the big files from the previous analysis to compare.

 
Ann

17

The file setting xml file I posted before does not seem to be enough alone. So I used the project settings again for further tests even though I don’t like changing the setting project wide. If there is a way to do it via configuraiton it would be great.

So I tried now this:

  1. I deleted two branches (called a and b now) in sonarcube I used for testing before
  2. Triggered a build of each branch => The sonar gate check was in each case fast (branch a 24s, branch b 23s), even though the new code page shows 0% duplication on 400k compared to the refrenence branch (develop in our case)
  3. configured branch b as reference branch of a in the settings
  4. triggered a build of branch a => This time the sonar gate check took 13min 15s.
  5. The new code tab of sonar shows now in the title the hint “Compared to branch b” and has also 0% duplication for 400k. Otherwise branch a and b both have only 0s in new code tab and overall code tab.

=> Since the 400k get also displayed for the fast builds they might not be the cause
=> strangly the 400k lines did not vanish

18

Hi,

Based on your OP, this is not about duplications, but about detection of moved / renamed files.

Since you appear to be analyzing C#(???) then you’re using the Scanner for .NET(???) which means sonar.properties isn’t used.

Ideally you should be excluding generated files for all branches of the project. Thus setting an exclusion for your large, generated file in the UI would be normal and reasonable. But since you don’t want to do that, can you please try setting /d:sonar.exclusions=**/[name of large generated file] in the begin step command line?

Once analysis is complete, you can check the Scanner Context ([project homepage] → Project Settings → Background Tasks → [analysis row cog menu] → Show Scanner Context) to verify the exclusions the analysis actually ran with.

 
Ann

19

As I wrote before, sonar.properties did not work so I tried a SonarQube.Analysis.xml file, which however seems also to have no impact. We use sonar for .NET 6 and C# 10.0

There are multiple such generated files like the large one and they all end with .Designer.cs (which is very common for C# projects). So I added this exclude rule:
/d:sonar.exclusions=**/*.Designer.cs

The run was not faster, but probably one of the the slowest runs ever. the scanner context showed:

Project server settings:
  - sonar.cpd.exclusions=**/*.*
  - sonar.exclusions=**/SomeFolderWeExcludedBefore/*.*
Project scanner properties:
  - sonar.exclusions=**/*.Designer.cs
  ...

So I guess the project server settings were overriden by the new project scanner properties.

I did some further tests with combined exclusions like **/*.Designer.cs,**/SomeFolderWeExcludedBefore/*.* but they were still slow.

20

Hi,

Yes, there’s a hierarchy of parameters, as described here. That’s why I asked you to set the exclusion on the analysis command line.

I’m eager to hear how it goes once you’ve successfully excluded your **/*.Designer.cs files.

 
Ann

21

Well as I just wrote I tried excluding the **/*.Designer.cs files via command line already and it was rather slower than faster.

Just to be sure the seeting has effect I also tried excluding everything via command line. It had an effect on the numbers (all 0) but still took long.

22

Hi,

Thanks. That wasn’t clear to me.

Going back to the beginning, you observed that the detection of file moves was the time-hog during analysis report processing. Since the first run without *.Designer.cs files would still be comparing against the *.Designer.cs files from the previous run, could we have two runs in a row where *.Designer.cs files are excluded?

And then I’d like to see the ce.log lines for the processing of that second analysis report. And why not the analysis log too.

 
Ann

23

Since I assumed those *.Designer.cs files get automatically excluded, I just checked it. According to sonar-.net-documentation/appendix-2.md at master · SonarSource/sonar-.net-documentation · GitHub documentation xxx.Designer.cs files are automatically excluded.

I already have spend quite some time in preparing some log files with log file references to a fast and two slow runs on debug and trace level some time ago. Based on some earlier requests via communication via email which lead me then finally to this forum. I explictly had to repalce some sensitive strings that were included, so it was just not copy and paste.

I don’t want to post them here. If those logs are fine for you I could send them to you via email.

25

Hi,

I’ve initiated a private thread to transmit those logs in.

And I think it’s worth noting those docs you cited are in an archived repo - i.e. not incurrent use - and haven’t been updated since 2016. Since that exclusion isn’t in the current docs, I wouldn’t count on it being there.

 
Ann

26

I found a workaround that speeds it up dramatically: From 14-15min to 4s for a PR and 30s for branches => So it’s about 20000% faster in some cases.

The workaround is to specifiy project specific source and test folders in the sonar scanner begin call

/d:<guid of prod project>.sonar.sources=SourceFolder1/,SourceFolder2/
/d:<guid of test project>.sonar.tests=SourceFolder1/,SourceFolder2/

where the overall structure is at our project something like this:

MainFolder/ProdProject/SourceFolder1/sub/folders/SomeFile1.cs
MainFolder/ProdProject/SourceFolder2/sub/folders/SomeFile2.cs
MainFolder/TestProject/SourceFolder1/sub/folders/SomeFile1.cs
MainFolder/TestProject/SourceFolder2/sub/folders/SomeFile2.cs

The duplication detection is now also reporting much less lines with this workaround and not anymore the 400k as before.

Very helpful is in this regards is the generated .sonarcube/out/sonar-project.properties file which was containing for each project a <project.gui>.sonar.sources or <guid of prod project>.sonar.tests property with a HUGE list of long absolute file names. Those file paths explicitly listed as source included HUGE coverage.opencover.xml files and some other large resource files that are not relevant for sonar.

Note that the workaround is not perfect, as when new source folders get added they will not be analysed by sonar but will be silently ignored.

So in summary I think there are multiple bugs:

  • SonarScanner for MSBuild 5.10 includes too many not relevant files and possibly creates a too large source list for sonar
  • SonarCube 9.8.0.63668 Developer does not apply it’s own excludes to ignore not relevant data to speed up file movement detection
  • SonarCube 9.8.0.63668 Developer may be ineffiecent in file movement detection when dealing with large source lists
  • SonarCube 9.8.0.63668 Developer can be as initially described much slower on second analysis for no good reason (As a user I want to see how the PR compares to the reference branch and not to the previous analysis. There were no freshly closed/opened issues so there should have been no reason for the sonar quality gate check to compare to the reference branch for both first and second quality gate check).
32

Hi @florian4s,

I read all the thread. I think it’s important to first give you the knowledge about how things work::

  • The SonarScanner for MSBuild includes files based on what is in .csproj/.sln files. So if there are in your VisualStudio project, they will be in your analysis, and the scanner can’t decide whether or not a project file is relevant, ie. they all are from a scanner perspective.
  • The step Detect file moves doesn’t execute during the first analysis (ie. there is no baseline)
  • The step Detect file moves doesn’t execute on PR analysis (there is another dedicated one instead). So you’re obviously analyzing your code as a branch, right? If you are analyzing a PR, you should make sure your CI (Jenkins if I get it right) runs the analysis as a PR, not a branch.
  • Regardless of the comparison method (reference branch, previous analysis, etc.), at the end of the day, SonarQube has to compare a previous report to the current one. It necessarily involves executing Detect file moves, we can’t infer file move/rename in any other way. So it’s not a bug that it executes, it’s by design.
  • The step Detect file moves has to compute the difference between 2 sets of files. This takes time when files content are overly long, like an XML coverage report for example. This is not a bug, performances on such huge files can’t be easily improved, those files should not be in analysis scope in the first place.

In the end, you had the good approach: to scope the analysis to only files that are relevant. However, instead of hardcoding sonar.sources and sonar.tests, you’d be better using sonar.exclusions to exclude irrelevant files, as @ganncam mentioned. Indeed as you spotted “when new source folders get added they will not be analyzed by sonar but will be silently ignored”, this will not happen if you use exclusions.
You could also choose to restructure your project to not have those files there in the first place.

So as a conclusion, there is no bug or misbehavior here, the solution is to not have huge files in the analysis scope, especially if this is not source code (XML/JSON/SQL/… exports, generated code, etc.)

I hope it makes sense.

Cheers,
Antoine

1 Like
34

Hello @Antoine

Thanks for taking the time to read the whole post and for taking the time to composing a reply:

  1. About SonarScanner for MSBuild: The coverage and other test result files are not in the project source. We also don’t have special code that creates them. We have normal csharp xunit test projects for the test code. I think the coverage file is even generated by/for the sonar scanner. So I would say it is a bug they are included. There are different test frameworks in csharp, could it be that sonar scanner can’t handle xunit projects yet?

  2. About detect file moves point 1: Agree there should be no Detect file moves if it is the first build on a branch without a baseline configured to compare against

  3. For point 2 about Detect file moves. For the explicit tests I made, I used a test branches. We had the performance issue however also for PRs.

  4. If I understand your third point about the step Detect file moves corectly it has to run when I start a branch build with a reference branch configured. => So it also has to start for the first build of that branch as there is another branch to compare it to. => It is strange that it is not slow for the first build of a branch when there is a reference branch configured!

  5. Sadly the excludes do not help with the performance. Wild guess: Maybe the excludes are not applied for the file list used by the file movement detection. Or another wild guess: The long list of included source file paths is the source of the slow down (e.g. because it compares each file against the huge list).

37

Florian, it all boils down to this fact: why are those files included in the first place? I doubt that this is a bug but there must be an explanation. Let’s find out.

Could you please turn on debug mode for analysis (add the /d:sonar.verbose=true parameter to your begin step) and share those logs with me? That way, I could try to understand what happening. Note that I need the full analysis logs: begin, build and end.

Cheers
Antoine

38

Hello @Antoine ,

how can I send the log files to you?

I don’t want to post them in this public thread.

39

Hello @florian4s,

This is a Community forum and your case could be useful to others as well. Of course, I understand that you don’t want to leak any sensitive information. To comply with these 2 concerns, could you please redact what’s necessary from your logs and still share them here? That would be the ideal solution.

Cheers,
Antoine

1 Like
40

Hello @Antoine ,
here are the log lines that I think are relevant with edits to remove sensitive data.

Lines from begin part in which sonar.cs.opencover.reportsPaths is used to pass in the location of the coverage files:

C:\Jenkins_Home\workspace\demo-branch>dotnet "C:\Jenkins_Home\tools\hudson.plugins.sonar.MsBuildSQRunnerInstallation\Sonar_.NET_6.0\SonarScanner.MSBuild.dll" begin /k:SolutionName /n:SolutionName /d:sonar.verbose=true /v: /d:sonar.cs.opencover.reportsPaths=SolutionName/**/TestResults/**/coverage.opencover.xml /d:sonar.coverage.exclusions=**/SolutionName/*Tests/**/Test*.cs,**.xaml.cs,**/SomeMoreExplictExcludes/Example.cs 
SonarScanner for MSBuild 5.10
Using the .NET Core version of the Scanner for MSBuild
Default properties file was found at C:\Jenkins_Home\tools\hudson.plugins.sonar.MsBuildSQRunnerInstallation\Sonar_.NET_6.0\SonarQube.Analysis.xml
Loading analysis properties from C:\Jenkins_Home\tools\hudson.plugins.sonar.MsBuildSQRunnerInstallation\Sonar_.NET_6.0\SonarQube.Analysis.xml
sonar.verbose=true was specified - setting the log verbosity to 'Debug'
16:08:32.075  Updating build integration targets...
...
16:08:33.192  Pre-processing succeeded.

Actual build:

C:\Jenkins_Home\workspace\demo-branch>"C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Current\Bin\MSBuild" SolutionName/SolutionName.sln /p:Configuration=Debug 
Microsoft (R) Build Engine version 17.1.0+ae57d105c for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Building the projects in this solution one at a time. To enable parallel build, please add the "-m" switch.
Build started 3/13/2023 4:08:33 PM.
Project "C:\Jenkins_Home\workspace\demo-branch\SolutionName\SolutionName.sln" on node 1 (default targets).
...
Build succeeded.
...
Time Elapsed 00:02:37.18

The end phase with several references to the coverage file and other test result files

C:\Jenkins_Home\workspace\demo-branch>dotnet "C:\Jenkins_Home\tools\hudson.plugins.sonar.MsBuildSQRunnerInstallation\Sonar_.NET_6.0\SonarScanner.MSBuild.dll" end 
SonarScanner for MSBuild 5.10
Using the .NET Core version of the Scanner for MSBuild
Post-processing started.
16:11:12.523  16:11:12.304  Uninstalling target: C:\Jenkins_Home\workspace\demo-branch\.sonarqube\bin\targets\SonarQube.Integration.targets
16:11:12.523  16:11:12.523  sonar.verbose=true was specified - setting the log verbosity to 'Debug'
16:11:12.524  Loading the SonarQube analysis config from C:\Jenkins_Home\workspace\demo-branch\.sonarqube\conf\SonarQubeAnalysisConfig.xml
...
16:11:12.787  Dumping content of sonar-project.properties
------------------------------------------------------------------------
...
test-project-guid.sonar.tests=\
...
"C:\\Jenkins_Home\\workspace\\demo-branch\\SolutionName\\SomeProjectTests\\TestResults\\some-guid\\coverage.opencover.xml",\
"C:\\Jenkins_Home\\workspace\\demo-branch\\SolutionName\\SomeProjectTests\\TestResults\\PROJECT-BUILD$_PROJECT-BUILD_2023-03-13_16_07_39\\In\\PROJECT-BUILD\\coverage.opencover.xml",\
"C:\\Jenkins_Home\\workspace\\demo-branch\\SolutionName\\SomeProjectTests\\TestResults\\SomeProjectTest_Results.xml"
...
sonar.cs.opencover.reportsPaths=SolutionName/**/TestResults/**/coverage.opencover.xml
sonar.coverage.exclusions=**/SolutionName/*Tests/**/Test*.cs,**.xaml.cs,...
...
project-guid.sonar.cs.analyzer.projectOutPaths=\
"C:\\Jenkins_Home\\workspace\\demo-branch\\.sonarqube\\out\\3"
project-guid.sonar.cs.roslyn.reportFilePaths=\
"C:\\Jenkins_Home\\workspace\\demo-branch\\.sonarqube\\out\\3\\Issues.json"
...
sonar.visualstudio.enable=false
...
16:11:28.098 DEBUG:   Test paths: some/file/paths/ExampleTest.cs, .. TestResults/some-guid/coverage.opencover.xml, TestResults/PROJECT-BUILD$_PROJECT-BUILD_2023-03-13_16_07_39/In/PROJECT-BUILD/coverage.opencover.xml, TestResults/SomeProjectTest_Results.xml
16:11:28.098 INFO:   Excluded sources: **/SomeFolder/*.*
16:11:28.098 INFO:   Excluded sources for coverage: **/SolutionName/*Tests/**/Test*.cs, **.xaml.cs, **/SomeMoreExplictExcludes/Example.cs 
16:11:28.098 INFO:   Excluded sources for duplication: **/*.*
...
16:11:29.400 DEBUG: 'SomeProjectTests\TestResults\some-guid\coverage.opencover.xml' indexed as test with language 'xml'
16:11:29.400 DEBUG: File SomeProjectTests/TestResults/some-guid/coverage.opencover.xml excluded for duplication
...
16:11:29.409 DEBUG: 'SomeProjectTests\TestResults\PROJECT-BUILD$_PROJECT-BUILD_2023-03-13_16_07_39\In\PROJECT-BUILD\coverage.opencover.xml' indexed as test with language 'xml'
16:11:29.409 DEBUG: File SomeProjectTests/TestResults/PROJECT-BUILD$_PROJECT-BUILD_2023-03-13_16_07_39/In/PROJECT-BUILD/coverage.opencover.xml excluded for duplication
16:11:29.422 DEBUG: 'SomeProjectTests\TestResults\SomeProjectTest_Results.xml' indexed as test with language 'xml'
16:11:29.422 DEBUG: File SomeProjectTests/TestResults/SomeProjectTest_Results.xml excluded for duplication
...
16:11:41.038 INFO: Sensor C# Properties [csharp] (done) | time=0ms
16:11:41.038 INFO: Sensor XML Sensor [xml]
16:11:41.039 INFO: Sensor XML Sensor [xml] (done) | time=1ms
16:11:41.039 INFO: Sensor Text Sensor [text]
16:11:41.039 INFO: x source files to be analyzed
...
16:11:41.415 DEBUG: 'SomeProjectTests/TestResults/SomeProjectTest_Results.xml' generated metadata as test  with charset 'UTF-8'
...
16:11:42.561 DEBUG: 'SomeProjectTests/TestResults/some-guid/coverage.opencover.xml' generated metadata as test  with charset 'UTF-8'
...
16:11:43.999 INFO: x/x source files have been analyzed
...
16:12:46.561 INFO: EXECUTION SUCCESS
...
16:12:47.088  Post-processing succeeded.

Also interesting that with our base branch using the workaround the test branch had now also a faster quality gate check. Even though the test brach I used for this logs again was unchanged and was not using the workaround. Probably because with the large coverage files being gone from the base branch the compare between test and base branch was thus faster.