Enabling SonarQube analysis cause dotnet build to be very slow

I have a dotnet core build within an Azure DevOps pipeline that usually takes 3 minutes. This build is for a collection of production code and unit test projects

If I enable SonarQube analysis, using the SonarWay profile, this build time jumps up to around 3 hours.

If I disable the code smells, reducing the rules from approx 250 to 110, the build time drops to just under 2 hours.

I have tried splitting my production code build from unit tests, as it seems to be the unit tests that are the slow part. Doing this means my build time doubles when I enable SonarQube (which is much better) but I obviously loose code coverage which is a major limitation.

Is there anything I can do to bring this build time down to something more reasonable, I expect enabling SonarQube to increase the build time but 3 minutes to 3 hours is unexpected.

Hi,

Could you share your version and edition of SonarQube?

 
Ann

It is Version 8.5.1.38104 Developer, 2M Loc License

1 Like

hi @rfennell

to see what rules are taking time during build

  • msbuild /p:reportanalyzer=true /v:d > build.log

Get the most consuming rules for each project.

select-string -path “build.log” -pattern “NOTE: Elapsed time may be less than analyzer execution time because analyzers can run concurrently.” -Context 1,30 > analyzer_times.txt

And we’d appreciate if you could share the analyzer_times.txt with us.

Does pipe increase only during the msbuild compilation (where our roslyn analyzers run), or also during the END step of the Scanner for MSBuild (where our injection vulnerability detection runs)? for the latter, we’d need another set of logs

  • please run SonarScanner.MSBuild.exe begin /k:“MyProject” /d:sonar.verbose=true as the begin step, and please attach the output of END step

Hi @Andrei_Epure

My slow build step is not being built with MSBuild but also using dotnet.exe. I built my production (APPX package) with MSBuild but the unit tests with dotnet.exe

So, is there an equivalent way to get the detailed log from dotnet.exe?

The key bits of my build pipeline is as follows

- task: SonarSource.sonarqube.15B84CA1-B62F-4A2A-A403-89B77A063157.SonarQubePrepare@4
      displayName: 'Prepare analysis on SonarQube'
      inputs:
        SonarQube: Sonarqube
        projectKey: 'xxxx'
        projectName: 'xxxx'
        projectVersion: '${{parameters.major}}.${{parameters.minor}}'
        extraProperties: |
         # Additional properties that will be passed to the scanner, 
         sonar.cpd.exclusions=**/AssemblyInfo.cs,**/*.g.cs
         sonar.cs.vscoveragexml.reportsPaths=$(System.DefaultWorkingDirectory)/**/*.coveragexml
         sonar.cs.vstest.reportsPaths=$(System.DefaultWorkingDirectory)/**/*.trx

# Build the APPX package - this builds in reasonable time  with (15min) or without  (26min) SonarQube
- task: VSBuild@1
      displayName: 'Build solution for Configuration $(BuildConfiguration)'
      inputs:
        solution: src/xxx.csproj
        msbuildArgs: '/p:PackageCertificateKeyFile="$(tempSigningCert.secureFilePath)" /p:AppxPackageDir=$(AppxOutputPath) /p:PackageCertificatePassword="$(signingCert.password)"'
        platform: '$(BuildPlatform)'
        configuration: '$(BuildConfiguration)'
        clean: true
        maximumCpuCount: true

# Build the Unit tests - this builds in  is the one that jumps from 3min without SonarQube up to  120min with SonarQube
- task: DotNetCoreCLI@2
      displayName: 'dotnet build unit tests'
      inputs:
        command: build
        projects: 'src/xxx/**/*UnitTests.csproj'
        arguments: '--no-restore' 

    - task: VisualStudioTestPlatformInstaller@1
      inputs:
        packageFeedSelector: 'nugetOrg'
        versionSelector: 'latestPreRelease'

    - task: VSTest@2
      inputs:
        testSelector: 'testAssemblies'
        testAssemblyVer2: |
          **\*UnitTests.dll
          !**\*TestAdapter.dll
          !**\obj\**
        searchFolder: 'src/xxx'
        resultsFolder: '$(System.DefaultWorkingDirectory)\TestResults'
        runInParallel: true
        codeCoverageEnabled: true
        rerunFailedTests: true
        runTestsInIsolation: true
        runOnlyImpactedTests: false

    - task: BlackMarble.CodeCoverage-Format-Convertor-Private.CodeCoverageFormatConvertor.CodeCoverage-Format-Convertor@1
      displayName: 'CodeCoverage Format Convertor'
      inputs:
        ProjectDirectory: '$(System.DefaultWorkingDirectory)'
    
    - task: SonarSource.sonarqube.6D01813A-9589-4B15-8491-8164AEB38055.SonarQubeAnalyze@4
      displayName: 'Run Code Analysis'
 

The dotnet build command uses MSBuild under the hood, so you can pass the same parameters.

dotnet build ... -v:d -p:reportanalyzer=true

In case you have large classes in your codebase, it’s possible you will suffer from a performance issue like in this thread and this thread. We’ve fixed that issue, which will be released in SonarQube 8.6 (this week).

But first let’s see the logs.

The logs show that my build is re-building a lot of referenced projects. The analysis on most are quick, but a good few show this pattern. I have not found any take over 300sec yet (log sample below)

So I don’t think it is one rule causing the issue.

I am looking to see why there are so many rebuilds, but it remains strange how much the time leaps when SonarQube is enabled.

2020-12-07T13:26:54.2774727Z          Total analyzer execution time: 258.341 seconds.
2020-12-07T13:26:54.2775695Z          NOTE: Elapsed time may be less than analyzer execution time because analyzers can run concurrently.
2020-12-07T13:26:54.2776467Z          
2020-12-07T13:26:54.2776997Z          Time (s)    %   Analyzer
2020-12-07T13:26:54.2777831Z           220.402   85   SonarAnalyzer.CSharp, Version=8.13.1.0, Culture=neutral, PublicKeyToken=c5b62af9de6d7244
2020-12-07T13:26:54.2778912Z            41.030   15      SonarAnalyzer.Rules.SymbolicExecution.SymbolicExecutionRunner
2020-12-07T13:26:54.2779809Z            37.284   14      SonarAnalyzer.Rules.CSharp.TokenTypeAnalyzer
2020-12-07T13:26:54.2780598Z            16.972    6      SonarAnalyzer.Rules.CSharp.SymbolReferenceAnalyzer
2020-12-07T13:26:54.2781518Z            13.503    5      SonarAnalyzer.Rules.CSharp.PropertiesAccessCorrectField
2020-12-07T13:26:54.2782474Z            11.269    4      SonarAnalyzer.Rules.CSharp.CbdeHandlerRule
2020-12-07T13:26:54.2783349Z             8.245    3      SonarAnalyzer.Rules.CSharp.InfiniteRecursion
2020-12-07T13:26:54.2784260Z             5.549    2      SonarAnalyzer.Rules.CSharp.BeginInvokePairedWithEndInvoke
2020-12-07T13:26:54.2785111Z             5.394    2      SonarAnalyzer.Rules.CSharp.DisposableNotDisposed
2020-12-07T13:26:54.2786009Z             4.899    1      SonarAnalyzer.Rules.CSharp.InsecureEncryptionAlgorithm
2020-12-07T13:26:54.2786880Z             4.126    1      SonarAnalyzer.Rules.CSharp.MetricsAnalyzer

We’re really interested in finding out more on this, as performance is a feature.

Are you re-building the projects when running the unit tests?

Is it possible to reduce the performance issue to a small reproducer project and share it with us (I can send a private message for that, if you wish, to avoid public eyes).

Andrei

I have been doing more digging on this too. The basic logic of my Azure Pipelines build is

  • Use VSBuild Task to build and package a UWP based csproj. With no SonarQube Analysis this takes 15min and with analysis 26min as expected.
  • Use dotnet Task to build all the unit test projects (inc re-build the DLLs it depends on). With no SonarQube Analysis this takes 3min, but with analysis 1hr 43 min

On looking at the logs I can see the same DLLs are being built and I think the issue is just the volume of analysis going on. None of the analysis tasks are taking that long individually.

I am happy to provide the logs via a private channel if it helps.

Richard

I’ll send a private message for the perf logs.

Please note that we made some important perf improvements in the C# analyzer, which got shipped in SQ 8.6 (the latest release).

I think we have got to the bottom of this issue. It was a misconfiguration of the build tasks. Note we did not yet updated to SonarQube 8.6.

The problem appears to have been with the build --configuration passed into the .NET Core build stage. There was a mismatch between this and a setting on a prior build step where we build the Xamarin project. Once these values matched the build time for the .NET Core build task when SonarQube was enabled dropped back to just a little over the time taken when SonarQube was not enabled i.e. dropped from 3hrs to 3min.

It is strange that this mismatch was only exposed when SonarQube was enabled, I would have expect the build without SonarQube to be equally as slow.

Any way it is sorted now - thanks everyone for the suggestions

I’m glad you sorted things out, @rfennell. From what I understand, your SQ analysis (issues, code coverage) works fine, right?