I’ve been doing some performance testing trying to understand why it takes Sonar so long to scan files locally and if there is anything I can do to speed it up. We have a few projects with total of around 12k Java source files having ~1m LOC and it takes ~37 minutes to do scan with the default rules and settings. SQ 6.7 is being used.
Scan timings are different per project ranging from 1.7 to 7.5 files per second, with average being 5.2 files/sec which seems pretty slow. During the scan average CPU usage is at 25%.
Is there anything I can do to improve the local performance? Should I get any other metrics or provide any other info for a better picture?
Hi,
The first thing I would do is upgrade to a modern version. 6.7 is a couple years out of date by now. We release approximately every 2 months and pack lots of features and improvements (sometimes including analysis performance!) into each version.
Your upgrade path is:
6.7 → 7.9.5 (current LTS) → 8.6 (8.7 imminent. This step optional)
If you’re still having performance issues after the upgrade, please come back to us.
Ann
Sorry, it was a typo. I’m using 8.7
Okay! Well that’s a different story then.
You should be getting some timing out of the various steps in your analysis log. Can you start by sharing that log here? (Code-formatted, please! 
)
Ann
Out of 110 seconds total Sonar run, the majority of time (85 seconds) takes metadata generation:
‘###.java’ generated metadata with charset ‘windows-1252’
…
898/898 source files have been analyzed
This is roughly 10 files/second. Is that considered a normal scan performance? Is there a way to improve it?
Hi,
I’m confused. Your OP says 37 min & your most recent post says just under 2. Could you post an analysis log, please?
Ann
As I noted in my initial post, I have a couple of projects that are scanned at the same time. So these 2 minutes is timeline for one of them. I can do the same for other projects or for all of them combined but this one is enough to see the general picture. Hope that that clarifies.
Hi,
Not without the analysis log.
Ann
00:00:04.246 [sonar:sonar] Apache Ant(TM) version 1.10.1 compiled on February 2 2017
00:00:04.247 [sonar:sonar] SonarQube Ant Task version: 2.7.0.1612
00:00:04.248 [sonar:sonar] Loaded from: file:/c:####/WebContent/WEB-INF/lib/sonarqube-ant-task-2.7.0.1612.jar
00:00:04.733 [sonar:sonar] User cache: ####\.sonar\cache
00:00:04.872 [sonar:sonar] Default locale: "en_US", source code encoding: "windows-1252" (analysis is platform dependent)
00:00:05.269 [sonar:sonar] SonarScanner will require Java 11 to run starting in SonarQube 8.x
00:00:05.659 [sonar:sonar] Load global settings
00:00:05.801 [sonar:sonar] Load global settings (done) | time=141ms
00:00:05.811 [sonar:sonar] Server id: 86E1FA4D-AXWQGZXW8jNkJQJr8yoI
00:00:05.817 [sonar:sonar] User cache: ####\.sonar\cache
00:00:05.824 [sonar:sonar] Load/download plugins
00:00:05.825 [sonar:sonar] Load plugins index
00:00:05.961 [sonar:sonar] Load plugins index (done) | time=136ms
00:00:06.198 [sonar:sonar] Load/download plugins (done) | time=373ms
00:00:06.438 [sonar:sonar] Loaded core extensions: developer-scanner
00:00:07.052 [sonar:sonar] JavaScript/TypeScript frontend is enabled
00:00:07.504 [sonar:sonar] Process project properties
00:00:07.528 [sonar:sonar] Process project properties (done) | time=24ms
00:00:07.528 [sonar:sonar] Execute project builders
00:00:07.535 [sonar:sonar] Execute project builders (done) | time=5ms
00:00:07.545 [sonar:sonar] Project key: ####
00:00:07.548 [sonar:sonar] Base dir: ####
00:00:07.549 [sonar:sonar] Working dir: ####\.scannerwork
00:00:13.110 [sonar:sonar] Load project settings for component key: '####'
00:00:13.127 [sonar:sonar] Load project settings for component key: '####' (done) | time=17ms
00:00:13.133 [sonar:sonar] Load project branches
00:00:13.151 [sonar:sonar] Load project branches (done) | time=18ms
00:00:13.152 [sonar:sonar] Load project pull requests
00:00:13.162 [sonar:sonar] Load project pull requests (done) | time=9ms
00:00:13.162 [sonar:sonar] Load branch configuration
00:00:13.164 [sonar:sonar] Load branch configuration (done) | time=3ms
00:00:13.211 [sonar:sonar] Load quality profiles
00:00:13.298 [sonar:sonar] Load quality profiles (done) | time=87ms
00:00:13.305 [sonar:sonar] Auto-configuring with CI 'Jenkins'
00:00:13.312 [sonar:sonar] Load active rules
00:00:14.423 [sonar:sonar] Load active rules (done) | time=1111ms
00:00:14.559 [sonar:sonar] Indexing files...
00:00:14.559 [sonar:sonar] Project configuration:
00:00:15.971 [sonar:sonar] 910 files indexed
00:00:15.972 [sonar:sonar] 8 files ignored because of scm ignore settings
00:00:15.972 [sonar:sonar] Quality profile for java: Sonar way
00:00:15.972 [sonar:sonar] Quality profile for web: Sonar way
00:00:15.972 [sonar:sonar] ------------- Run sensors on module ####
00:00:16.218 [sonar:sonar] JavaScript/TypeScript frontend is enabled
00:00:16.243 [sonar:sonar] Load metrics repository
00:00:16.272 [sonar:sonar] Load metrics repository (done) | time=27ms
00:00:20.015 [sonar:sonar] Sensor JavaSquidSensor [java]
00:00:20.229 [sonar:sonar] Configured Java source version (sonar.java.source): none
00:00:20.254 [sonar:sonar] JavaClasspath initialization
00:00:20.296 [sonar:sonar] JavaClasspath initialization (done) | time=42ms
00:00:20.297 [sonar:sonar] JavaTestClasspath initialization
00:00:20.298 [sonar:sonar] JavaTestClasspath initialization (done) | time=3ms
00:00:20.330 [sonar:sonar] Java Main Files AST scan
00:00:20.335 [sonar:sonar] 898 source files to be analyzed
00:00:20.350 [sonar:sonar] Load project repositories
00:00:20.406 [sonar:sonar] Load project repositories (done) | time=57ms
00:00:30.337 [sonar:sonar] 14/898 files analyzed, current file: ####.java
00:00:40.338 [sonar:sonar] 71/898 files analyzed, current file: ####.java
00:00:50.338 [sonar:sonar] 189/898 files analyzed, current file: ####.java
00:01:00.340 [sonar:sonar] 289/898 files analyzed, current file: ####.java
00:01:10.340 [sonar:sonar] 414/898 files analyzed, current file: ####.java
00:01:20.341 [sonar:sonar] 540/898 files analyzed, current file: ####.java
00:01:30.342 [sonar:sonar] 622/898 files analyzed, current file: ####.java
00:01:40.342 [sonar:sonar] 727/898 files analyzed, current file: ####.java
00:01:50.343 [sonar:sonar] 825/898 files analyzed, current file: ####.java
00:01:54.743 [sonar:sonar] Unable to run check class org.sonar.java.se.SymbolicExecutionVisitor - on file '####.com/
00:01:54.743 [sonar:sonar] org.eclipse.jdt.internal.compiler.problem.AbortCompilation: Pb(324) The type #### cannot be resolved. It is indirectly referenced from required .class files
pretty long stack trace goes here
00:01:55.829 [sonar:sonar] 898/898 source files have been analyzed
00:01:55.830 [sonar:sonar] Java Main Files AST scan (done) | time=95502ms
00:01:55.830 [sonar:sonar] Java Test Files AST scan
00:01:55.831 [sonar:sonar] 0 source files to be analyzed
00:01:55.832 [sonar:sonar] Java Test Files AST scan (done) | time=1ms
00:01:55.832 [sonar:sonar] Java Generated Files AST scan
00:01:55.833 [sonar:sonar] 0 source files to be analyzed
00:01:55.833 [sonar:sonar] 0/0 source files have been analyzed
00:01:55.834 [sonar:sonar] Java Generated Files AST scan (done) | time=1ms
00:01:55.834 [sonar:sonar] Sensor JavaSquidSensor [java] (done) | time=95821ms
00:01:55.834 [sonar:sonar] Sensor CSS Rules [cssfamily]
00:01:55.834 [sonar:sonar] 0/0 source files have been analyzed
00:01:59.731 [sonar:sonar] 1 source files to be analyzed
00:01:59.994 [sonar:sonar] 1/1 source files have been analyzed
00:01:59.994 [sonar:sonar] Sensor CSS Rules [cssfamily] (done) | time=4161ms
00:01:59.994 [sonar:sonar] Sensor C# Properties [csharp]
00:01:59.995 [sonar:sonar] Sensor C# Properties [csharp] (done) | time=2ms
00:01:59.995 [sonar:sonar] Sensor SurefireSensor [java]
00:01:59.996 [sonar:sonar] parsing [####\target\surefire-reports]
00:01:59.997 [sonar:sonar] Sensor SurefireSensor [java] (done) | time=2ms
00:01:59.998 [sonar:sonar] Sensor JavaXmlSensor [java]
00:02:00.007 [sonar:sonar] Sensor JavaXmlSensor [java] (done) | time=8ms
00:02:00.007 [sonar:sonar] Sensor HTML [web]
00:02:00.094 [sonar:sonar] Sensor HTML [web] (done) | time=87ms
00:02:00.095 [sonar:sonar] Sensor VB.NET Properties [vbnet]
00:02:00.096 [sonar:sonar] Sensor VB.NET Properties [vbnet] (done) | time=1ms
00:02:00.096 [sonar:sonar] Sensor JaCoCo XML Report Importer [jacoco]
00:02:00.104 [sonar:sonar] No coverage report can be found with sonar.coverage.jacoco.xmlReportPaths='build/coverage-results/jacoco.xml'. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
00:02:00.106 [sonar:sonar] No report imported, no coverage information will be imported by JaCoCo XML Report Importer
00:02:00.106 [sonar:sonar] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=10ms
00:02:00.106 [sonar:sonar] Sensor ThymeLeaf template sensor [securityjavafrontend]
00:02:00.108 [sonar:sonar] Sensor ThymeLeaf template sensor [securityjavafrontend] (done) | time=3ms
00:02:00.108 [sonar:sonar] Sensor JavaSecuritySensor [security]
00:02:00.108 [sonar:sonar] Reading type hierarchy from: ####\.scannerwork\ucfg2\java
00:02:00.424 [sonar:sonar] Read 1072 type definitions
00:02:00.444 [sonar:sonar] Reading UCFGs from: ####\.scannerwork\ucfg2\java
00:02:01.787 [sonar:sonar] 16:04:26.79 Building Runtime Type propagation graph
00:02:01.957 [sonar:sonar] 16:04:26.964 Running Tarjan on 14203 nodes
00:02:01.996 [sonar:sonar] 16:04:26.998 Tarjan found 14171 components
00:02:02.046 [sonar:sonar] 16:04:27.053 Variable type analysis: done
00:02:02.049 [sonar:sonar] 16:04:27.056 Building Runtime Type propagation graph
00:02:02.199 [sonar:sonar] 16:04:27.206 Running Tarjan on 14203 nodes
00:02:02.221 [sonar:sonar] 16:04:27.228 Tarjan found 14171 components
00:02:02.266 [sonar:sonar] 16:04:27.273 Variable type analysis: done
00:02:02.274 [sonar:sonar] Analyzing 5425 ucfgs to detect vulnerabilities.
00:02:06.832 [sonar:sonar] All rules entrypoints : 0 Retained UCFGs : 0
00:02:06.835 [sonar:sonar] rule: S5131, entrypoints: 0
00:02:06.836 [sonar:sonar] rule: S5131 done
00:02:06.836 [sonar:sonar] rule: S3649, entrypoints: 0
00:02:06.836 [sonar:sonar] rule: S3649 done
00:02:06.836 [sonar:sonar] rule: S2076, entrypoints: 0
00:02:06.836 [sonar:sonar] rule: S2076 done
00:02:06.837 [sonar:sonar] rule: S2091, entrypoints: 0
00:02:06.837 [sonar:sonar] rule: S2091 done
00:02:06.837 [sonar:sonar] rule: S2078, entrypoints: 0
00:02:06.837 [sonar:sonar] rule: S2078 done
00:02:06.837 [sonar:sonar] rule: S2631, entrypoints: 0
00:02:06.838 [sonar:sonar] rule: S2631 done
00:02:06.838 [sonar:sonar] rule: S5135, entrypoints: 0
00:02:06.838 [sonar:sonar] rule: S5135 done
00:02:06.842 [sonar:sonar] rule: S2083, entrypoints: 0
00:02:06.842 [sonar:sonar] rule: S2083 done
00:02:06.842 [sonar:sonar] rule: S5167, entrypoints: 0
00:02:06.842 [sonar:sonar] rule: S5167 done
00:02:06.842 [sonar:sonar] rule: S5144, entrypoints: 0
00:02:06.843 [sonar:sonar] rule: S5144 done
00:02:06.843 [sonar:sonar] rule: S5145, entrypoints: 0
00:02:06.843 [sonar:sonar] rule: S5145 done
00:02:06.844 [sonar:sonar] rule: S5146, entrypoints: 0
00:02:06.844 [sonar:sonar] rule: S5146 done
00:02:06.844 [sonar:sonar] rule: S5334, entrypoints: 0
00:02:06.844 [sonar:sonar] rule: S5334 done
00:02:06.844 [sonar:sonar] Sensor JavaSecuritySensor [security] (done) | time=6728ms
00:02:06.844 [sonar:sonar] Sensor CSharpSecuritySensor [security]
00:02:06.844 [sonar:sonar] Reading type hierarchy from: ####\ucfg_cs2
00:02:06.844 [sonar:sonar] Read 0 type definitions
00:02:06.844 [sonar:sonar] Reading UCFGs from: ####\ucfg_cs2
00:02:06.844 [sonar:sonar] No UCFGs have been included for analysis.
00:02:06.844 [sonar:sonar] Sensor CSharpSecuritySensor [security] (done) | time=1ms
00:02:06.844 [sonar:sonar] Sensor PhpSecuritySensor [security]
00:02:06.844 [sonar:sonar] Reading type hierarchy from: ####\.scannerwork\ucfg2\php
00:02:06.844 [sonar:sonar] Read 0 type definitions
00:02:06.845 [sonar:sonar] Reading UCFGs from: ####\.scannerwork\ucfg2\php
00:02:06.845 [sonar:sonar] No UCFGs have been included for analysis.
00:02:06.845 [sonar:sonar] Sensor PhpSecuritySensor [security] (done) | time=1ms
00:02:06.845 [sonar:sonar] Sensor PythonSecuritySensor [security]
00:02:06.845 [sonar:sonar] Reading type hierarchy from: ####\.scannerwork\ucfg2\python
00:02:06.845 [sonar:sonar] Read 0 type definitions
00:02:06.845 [sonar:sonar] Reading UCFGs from: ####\.scannerwork\ucfg2\python
00:02:06.845 [sonar:sonar] No UCFGs have been included for analysis.
00:02:06.845 [sonar:sonar] Sensor PythonSecuritySensor [security] (done) | time=0ms
00:02:06.845 [sonar:sonar] Sensor JsSecuritySensor [security]
00:02:06.845 [sonar:sonar] Reading type hierarchy from: ####\.scannerwork\ucfg2\js
00:02:06.845 [sonar:sonar] Read 0 type definitions
00:02:06.845 [sonar:sonar] Reading UCFGs from: ####\.scannerwork\ucfg2\js
00:02:06.845 [sonar:sonar] No UCFGs have been included for analysis.
00:02:06.845 [sonar:sonar] Sensor JsSecuritySensor [security] (done) | time=1ms
00:02:06.846 [sonar:sonar] ------------- Run sensors on project
00:02:06.869 [sonar:sonar] Sensor Zero Coverage Sensor
00:02:07.209 [sonar:sonar] Sensor Zero Coverage Sensor (done) | time=340ms
00:02:07.209 [sonar:sonar] Sensor Java CPD Block Indexer
00:02:08.006 [sonar:sonar] Sensor Java CPD Block Indexer (done) | time=798ms
00:02:08.017 [sonar:sonar] SCM Publisher SCM provider for this project is: git
00:02:08.017 [sonar:sonar] SCM Publisher 899 source files to be analyzed
00:02:08.020 [sonar:sonar] Shallow clone detected, no blame information will be provided. You can convert to non-shallow with 'git fetch --unshallow'.
00:02:08.020 [sonar:sonar] SCM Publisher 0/899 source files have been analyzed (done) | time=3ms
00:02:08.021 [sonar:sonar] Missing blame information for the following files:
00:02:08.021 [sonar:sonar] * ####.java
00:02:08.021 [sonar:sonar] * ####.java
...
...
...
00:02:08.077 [sonar:sonar] * ####.java
00:02:08.077 [sonar:sonar] * ####.java
00:02:08.077 [sonar:sonar] * ####.java
00:02:08.077 [sonar:sonar] This may lead to missing/broken features in SonarQube
00:02:08.144 [sonar:sonar] CPD Executor 201 files had no CPD blocks
00:02:08.144 [sonar:sonar] CPD Executor Calculating CPD for 698 files
00:02:08.452 [sonar:sonar] CPD Executor CPD calculation finished (done) | time=307ms
00:02:08.468 [sonar:sonar] Load New Code definition
00:02:08.483 [sonar:sonar] Load New Code definition (done) | time=15ms
00:02:09.684 [sonar:sonar] Analysis report generated in 770ms, dir size=6 MB
00:02:11.855 [sonar:sonar] Analysis report compressed in 2170ms, zip size=2 MB
00:02:12.124 [sonar:sonar] Analysis report uploaded in 268ms
00:02:12.126 [sonar:sonar] ANALYSIS SUCCESSFUL, you can browse https://####
00:02:12.127 [sonar:sonar] Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
00:02:12.127 [sonar:sonar] More about the report processing at https://####
00:02:13.519 [sonar:sonar] Analysis total time: 2:07.084 sHi,
Thanks for the log, although I don’t see anything in it about metadata. And you’re analyzing nearly 1k files in just over 2 minutes. Not too shabby IMO. Going back in the thread, I see this in the OP:
Time to analyze any given file is going to depend at least partly on the rules enabled and the size & complexity of the file itself.
On the face of it, I’m not seeing a problem here. And I’ve tagged this with the language in question to draw the attention of the specialists who are better qualified to go further than this.
Ann