Long runtime SonarCloudAnalyze@4 - Help needed

SonarQube Cloud
, , , ,
1

Template for a good new topic, formatted with Markdown:

  • ALM used: Azure DevOps
  • CI system used:Azure DevOps (self-hosted agents, 8 cores, dynamic up to 32 GB RAM), Java 25 preinstalled, using build wrapper
  • Scanner command used when applicable (private details masked)
  • Languages of the repository: C++23 (mainly), C#, Powershell, YAML
  • Error observed (wrap logs/code around with triple quotes ``` for proper formatting)
    Time for code analysis is much longer than for build.
    In log I see ##[warning]Free memory is lower than 5%; Currently used: - in all runs
  • Steps to reproduce
- task: SonarCloudPrepare@4
  inputs:
    SonarCloud: $(SonarQubeServiceConnection)
    organization: $(SonarQubeOrganization)
    projectKey: $(projectKey)
    extraProperties: |
      sonar.exclusions=external/\*\*,source/vcpkg_installed/\*\*
      sonar.cs.vscoveragexml.reportsPaths=$(Agent.TempDirectory)/\*\*/\*.xml
      sonar.cs.vstest.reportsPaths=$(Agent.TempDirectory)/\*\*/\*.trx
      sonar.cfamily.vscoveragexml.reportsPath=$(Agent.TempDirectory)/TestResults/gtest/\*\*/\*.coveragexml
      sonar.cfamily.reportingCppStandardOverride=c++23
      sonar.cfamily.enableModules=true
      sonar.cfamily.threads=6
      sonar.cfamily.compile-commands=$(SonarOutDir)/compile_commands.json
      sonar.sca.enabled=false
      sonar.sca.cfamily=true
      sonar.sca.exclusions="source/\*.Tests/\*\*"
      sonar.sca.resolveAsRoot=true
      sonar.sca.resolveDependencies=true
      sonar.sca.resolveDependencySources=false
      sonar.architecture.enable=false 

- task: PowerShell@2
  inputs:
    pwsh: true
    targetType: inline
    script: |
      & $vcpkgPath install --triplet x64-windows-v143 `
      --overlay-triplets ".\triplets" `
      --x-manifest-root ".\" `
      --x-install-root ".\vcpkg_installed\x64-windows-v143"

- task: PowerShell@2
  inputs:
    pwsh: true
    targetType: inline
    script: |
      $vswhere = "${env:ProgramFiles(x86)}\\Microsoft Visual Studio\\Installer\\vswhere.exe"
      $msbuild = & $vswhere -latest -requires Microsoft.Component.MSBuild -find MSBuild\\\*\*\\Bin\\MSBuild.exe | select-object -first 1

      & $msbuild MySolution.sln /t:restore /nologo
      & "$(BuildWrapper)" --out-dir "$(SonarOutDir)" \`
      "$msbuild" MySolution.sln \`
      /t:Rebuild \`
      /m \`
      /p:Configuration=$(BuildConfiguration) \`
      /nodeReuse:False \`
      /nologo

- task: SonarCloudAnalyze@4
  inputs:
    jdkVersion: 'JAVA_HOME'
  env:
    SONAR_SCANNER_JAVA_OPTS: "-Xms24G -Xmx24G -XX:+UseG1GC -XX:MaxGCPauseMillis=200"
  • Potential workaround
    • Tried to increase number of cores (4 to 8, 16 GB):
      • Build time dropped 10m 47s → 6m 39s
      • Analyze 41 m 4s → 1h 11, 35 increased, limit number of threads to 4: 30m 33s
    • Increase memory (8 cores, 16 GB → 32 GB) - nearly no change
    • different settings on SONAR_SCANNER_JAVA_OPTS. In system diagnostics the effect on the system diagnostics Memory: Used 24056.00 MB out of 30749.00 MB, - both numbers increase, the number of messages remain about 17 times.

For vcpkg binary caching on the build server is used - if this has any effect on the analyze process.

I observe, that the number of cache hits are in general quiet low if any C++ code was changed. In a build I got 44 hits [177/283] Module scan cache hit for - and in this case only one module and 5 test files were changed, analyze run on the same build server.

I would like to get the analyze time to be below the build time. Any recommendation are welcome.

2

Hi @milbrandt,

Thanks for the detailed report. There are a few things to address:

1. JVM heap appears to be too large

Setting -Xms24G -Xmx24G on a ~30GB machine leaves very little memory for the OS and the CFamily analyzer, which runs a separate native subprocess outside the JVM heap. Your ##[warning]Free memory is lower than 5% confirms this. This is probably why increasing cores made analysis slower because more parallel CFamily threads competed for the remaining memory.

The JVM scanner itself probably doesn’t need 24GB. Try reducing to:

SONAR_SCANNER_JAVA_OPTS: "-Xmx6G -XX:+UseG1GC"

This frees ~18GB for the CFamily native process and the OS, which should reduce swapping.

2. CFamily cache hit rate is too low (~15%)

44/283 hits means 85% of modules are being re-analyzed from scratch every run. The most likely causes with a vcpkg + Azure DevOps setup:

  • vcpkg reinstalls between runs change paths in compile_commands.json, causing cache misses
  • The cache path may not be persisted correctly $(Agent.TempDirectory) is wiped between pipeline runs on Azure DevOps agents

To diagnose cache misses, add sonar.verbose=true to your extraProperties and look for Cache miss for lines, they’ll tell you exactly which property changed.

For the cache path, ensure you’re pointing to a stable location that persists between runs (as you noted in a 2021 thread on exactly this topic):

sonar.cfamily.analysisCache.mode=fs
sonar.cfamily.analysisCache.path=$(Agent.WorkFolder)/$(Build.Repository.Name)/.cfamily-cache

3. C++20 modules add significant re-analysis cost with low cache hits

With sonar.cfamily.enableModules=true, a cache miss requires rebuilding the full BMI tree for all dependent modules, not just the changed file. With an 85% miss rate, this multiplies the overhead considerably. As a diagnostic step, try disabling this temporarily (sonar.cfamily.enableModules=false) to see how much it contributes to runtime.

Here’s a suggested order of actions:

  1. Reduce JVM heap to -Xmx6G and rerun
  2. Add sonar.verbose=true and look at cache miss reasons in the logs
  3. Fix the cache path to a persistent location
  4. Try disabling sonar.cfamily.enableModules as a diagnostic
  5. Consider setting sonar.sca.cfamily=false or sonar.sca.resolveDependencies=false to eliminate SCA overhead while tuning

Let us know what the verbose logs show for the cache misses, that will confirm whether the vcpkg paths are the culprit.

Best regards,

Stevan

3

Thanks @stevan.vanderwerf for your input.

  1. JVM heap appears to be too large
    I already tried before with number of cores and amount of memory (Xmx 4G, 12G, 24G).
    Your proposal of further reduction didn’t change much, it was on the lower end of the scattering (23m44s - 26m32s with 8 cores). In all cases, the warning Free memory is lower than 5% is emitted, that was the reason to increase the value.
    Is the .NET scanner taking the `SONAR_SCANNER_JAVA_OPTS´ into account at all?

  2. CFamily cache hit rate is too low

  • In 2021 there was only file system caching available. Now Analyze task reports “Loading cache from: server”. But I switch to fs and this seems to increase the cache hits.
  • For comparison I mainly use one build server, as only there I got there additional memory and cores from out IT department :wink:
  • .vcpkg is installed each time in the same location in the source folder. Therefore paths in compile_commands.json
  • binary caching of vcpkg on the build server is enabled.
  • My observation is, that no test files (all cpp, gtest) are found in the cache. I will not judge, if this is because they test code from C++ modules.
  • Log for cache hits indicates module dependency scanning cache issue. The C Family time 1387s (more than 23m7s of 24m2s of the task)
11:12:05.098  INFO: 0 compilation units were partially analyzed: 0 stopped on Config, 0 stopped on Parsing, 0 stopped on AST, 0 stopped on Symbolic Execution, and 0 stopped on Symbolic Execution with Parsing Error
11:12:05.098  INFO: 54 rules with 'symbolic-execution' tag raised 1 out of 279 issues
11:12:05.098  INFO: 253 compilation units were scanned for C++ modules
11:12:05.098  INFO: C++ module dependency scanning cache: 33/286 hits
11:12:05.098  INFO: 160 BMIs were built for C++ modules
11:12:05.176  INFO: Performance statistics per module scanning stage (Total, Average) across 253 compilation units: 130947ms, 517.58ms
11:12:05.176  INFO: Performance statistics per module building stage (Total, Average) across 160 compilation units: 1001944ms, 6262.15ms
11:12:05.176  INFO: Size statistics for successfully built BMIS, in bytes (Total, Maximum, Average): 1486572360, 53122588, 9291077
11:12:05.192  INFO: Performance statistics per analysis stage (Total, Average, Percentage) across 178 compilation units:
config: 2556ms, 14ms, 0.03%
parsing: 5954555ms, 33452ms, 80.53%
annotation: 487666ms, 2739ms, 6.6%
astRules: 132076ms, 742ms, 1.79%
symbolicExecution: 817179ms, 4590ms, 11.05%
ctuPreparation: null
11:12:05.192  INFO: PCH: unique=0 use=0 (forceInclude=0,throughHeader=0,firstInclude=0) out of 178 (forceInclude=0,throughHeader=0)
11:12:05.192  INFO: Z3 refutation rate: 1 out of 5
11:12:05.192  INFO: Percentage of files indexed with CFamily languages: 77.81% (C: 19, C++: 251, ObjC: 0, AnyLang: 347)
11:12:05.223  INFO: C and C++ analysis quality score: 96.41/100
11:12:05.223  INFO: Analysis measures statistics:
92.19% of classes were parsed successfully (21 out of 269 have parsing errors)
96.6% of functions were parsed successfully (55 out of 1619 have parsing errors)
96.85% of statements were parsed successfully (171 out of 5433 have parsing errors)
100% of the project includes directives were resolved (0 out of 307 were not resolved)
0 external includes directives were not resolved
0 modules imports were not resolved
0 header units were not resolved
11:12:05.520  INFO: Incremental symbolic execution internal implementation statistics (across analyzed compilation units):
Successful cache reads, writes: 165, 165 for 165 target compilation unit(s) (success rates 100%, 100%)
Unsupported cases in USR conversion decoding, encoding: 0, 61
Unsupported cases in issue-relocation decoding, encoding: 0, 0
Unsupported cases in abs. line no. decoding, encoding: 0, 0
Top-level declarations dropped decoding, encoding: 0, 0
Replayed path diagnostics: 0
Recorded path diagnostics: 61
Overwritten path diagnostic targets: 0
11:12:05.567  INFO: Telemetry analyzers aggregated counts: {boost=2, coroutines=0}
11:12:06.005  INFO: Sensor CFamily [cpp] (done) | time=1387240ms

3. C++20 modules add significant re-analysis cost with low cache hits
Disabling C++ modules sonar.cfamily.enableModules=false dropped the time dramatically to 2m30s (1m42s for CFamily) and the task log is about 900 lines shorter. But for sure the analysis depth will be reduced.

Current conclusions:

  • Upgrade servers to 8 cores (not only for testing, this generally speed up all builds)
  • reduce JVM Xmx as proposed
  • use less threads than cores (6/8)
  • file system caching seems to be better than server side caching for modules
  • sonar.cfamily.enableModules=false for faster feedback in PR, true for full build