Hi, I ran into a problem where sonarscan is consuming too much memory in my gitlab CI job, as much as 5GB.
Here are a bit of info on deployment:
I noticed that sonarscan spends a lot of time checking different languages in the repo. Is there a way to limit the check to a subset of languages? Also, what parameter can I set to reduce and put a limit to the resource consumed?
Thanks!
Hi,
You can’t directly limit the resources consumed. For indirect limits, you’re on the right path with limiting what files are analyzed. You should look at either including the files of interest by file extension or excluding the files you don’t want analyzed - whichever way is easier to describe.
As a general rule, you don’t need both inclusions and exclusions together.
Â
HTH,
Ann
Hi thanks for the response. If I cannot directly limit the resource consumed, what is the purpose of the sonar scanner option parameter? I saw this in the doc:
SONAR_SCANNER_OPTS: this parameter combine the above 3 paramters to 1 (default value: “-Xmx2048m -XX:MaxPermSize=512m -XX:ReservedCodeCacheSize=128m”), you can set it alone without above parameters like this: “-Xmx3062m -XX:MaxPermSize=1024m -XX:ReservedCodeCacheSize=128m”
I was under the impression that this can control how much resource is allocated to sonar scan.
A follow-up question on that note: is there any other way to limit resource used? My company doesn’t use all of the programming languages in the default quality profile. Is it possible to disable a few quality profiles? Also, we created some new quality profile by extending the default rules because we want to receive automatic update when the default rule set is updated. However, I cannot disable a rule if it’s inherited from another rule and I cannot disable a rule in the default quality profile. Is there a way to disable a rule from the default profile while creating a new rule by extending it so that we can still receive the automatic update on the profile?
Thanks!
Hi,
Setting SONAR_SCANNER_OPTS allows you to allocate more memory to an analysis. I suppose you could use it in the opposite direction to restrict the amount of memory analysis uses, but it’s quite likely to result in the analysis erroring-out because too little memory was available.
Ehm… only the profiles corresponding to the languages found in your project will be applied during analysis.
Yes, sorry. That’s how inherited profiles work. You can, however, change the severity. Perhaps it would be enough to just set the severity of those unwanted rules to Info?
Note that we try to keep it to one topic per thread. Otherwise it can get messy, fast. So if you have followups, I reserve the right to ask you to create new threads for them. 
Â
HTH,
Ann
Hi is there a general agreed amount of memory an MR scan would usually take? Does an MR scan speed depend on just the amount of new codes in the MR or does the total lines of codes in the repo matter as well?
Some of the MR we have had little change but it took close to an hour to finish the sonarscan job. Why is that? If it helps, I can post the gitlab cicd job logs as reference.
Hi,
Analysis speed and memory requirements depend mostly on the size of the code base. Unfortunately, there’s not a simple rubric I can give you for determining the necessary resources. In general, start at the defaults, and scale up as necessary.
It would be interesting to see those analysis logs.
Â
Ann
Is there an option to dm you?
Hi,
Feel free to redact your logs as necessary.
Â
Ann
Here is the gitlab job log. Hope this helps:
[0KRunning with some-service-some-version (436955cb)e[0;m
e[0K on some-service-some-hash HZX3sC2_, system ID: some_ide[0;m
section_start:1684174469:resolve_secrets
e[0Ke[0Ke[36;1mResolving secretse[0;me[0;m
section_end:1684174469:resolve_secrets
e[0Ksection_start:1684174469:prepare_executor
e[0Ke[0Ke[36;1mPreparing the "kubernetes" executore[0;me[0;m
e[0KUsing Kubernetes namespace: some-namespacee[0;m
e[0KUsing Kubernetes executor with image sonarsource/sonar-scanner-cli:latest ...e[0;m
e[0KUsing attach strategy to execute scripts...e[0;m
section_end:1684174469:prepare_executor
e[0Ksection_start:1684174469:prepare_script
e[0Ke[0Ke[36;1mPreparing environmente[0;me[0;m
Waiting for pod some-pod to be running, status is Pending
Running on some-pod via some-service...
section_end:1684174473:prepare_script
e[0Ksection_start:1684174473:get_sources
e[0Ke[0Ke[36;1mGetting source from Git repositorye[0;me[0;m
e[32;1mFetching changes...e[0;m
Initialized empty Git repository in /some/folder/to/.git/
e[32;1mCreated fresh repository.e[0;m
e[32;1mChecking out 9815bb45 as detached HEAD (ref is refs/merge-requests/2886/head)...e[0;m
e[32;1mSkipping Git submodules setupe[0;m
section_end:1684174486:get_sources
e[0Ksection_start:1684174486:restore_cache
e[0Ke[0Ke[36;1mRestoring cachee[0;me[0;m
e[32;1mChecking cache for sonarscan-non_protected...e[0;m
Downloading cache.zip from https://some/url/for/sonarscan/job/cachee[0;m
e[32;1mSuccessfully extracted cachee[0;m
section_end:1684174502:restore_cache
e[0Ksection_start:1684174502:step_script
e[0Ke[0Ke[36;1mExecuting "step_script" stage of the job scripte[0;me[0;m
e[32;1m$ sonar-scanner -Dsonar.projectKey=${SONAR_PROJECT_KEY} -Dsonar.qualitygate.wait=truee[0;m
INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarScanner 4.8.0.2856
INFO: Java 11.0.19 Alpine (64-bit)
INFO: Linux 5.10.162+ amd64
INFO: User cache: /some/folder/some-service/.sonar/cache
INFO: Analyzing on SonarQube server 10.0.0.68432
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Load global settings
INFO: Load global settings (done) | time=222ms
INFO: Server id: some-server-id
INFO: User cache: /some/folder/some-service/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=108ms
INFO: Load/download plugins (done) | time=613ms
INFO: Loaded core extensions: developer-scanner
INFO: Process project properties
INFO: Process project properties (done) | time=1ms
INFO: Execute project builders
INFO: Execute project builders (done) | time=2ms
INFO: Project key: some_project_name_AYPIfQ0SYDwiMQkHpeub
INFO: Base dir: /some/base/dir
INFO: Working dir: /some/folder/some-service/.scannerwork
INFO: Load project settings for component key: 'some_project_name_AYPIfQ0SYDwiMQkHpeub'
INFO: Load project settings for component key: 'some_project_name_AYPIfQ0SYDwiMQkHpeub' (done) | time=31ms
INFO: Load project branches
INFO: Load project branches (done) | time=28ms
INFO: Load branch configuration
INFO: Detected branch/PR in 'GitLab'
INFO: Auto-configuring pull request '2886'
INFO: Load branch configuration (done) | time=4ms
INFO: Auto-configuring with CI 'Gitlab CI'
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=95ms
INFO: Load active rules
INFO: Load active rules (done) | time=2729ms
INFO: Load analysis cache
INFO: Load analysis cache | time=2071ms
INFO: Pull request 2886 for merge into master from some-git-branch-name
INFO: Load project repositories
INFO: Load project repositories (done) | time=89ms
INFO: SCM collecting changed files in the branch
INFO: Merge base sha1: some_hash
INFO: SCM collecting changed files in the branch (done) | time=526ms
INFO: Indexing files...
INFO: Project configuration:
INFO: Excluded sources: tests/**
INFO: 3760 files indexed
INFO: 0 files ignored because of inclusion/exclusion patterns
INFO: 1 file ignored because of scm ignore settings
INFO: Quality profile for json: Sonar way
INFO: Quality profile for php: Sonar way Extended
INFO: Quality profile for plsql: Sonar way
INFO: Quality profile for py: Sonar way
INFO: Quality profile for ruby: Sonar way
INFO: Quality profile for terraform: Sonar way
INFO: Quality profile for web: Sonar way
INFO: Quality profile for xml: Sonar way
INFO: Quality profile for yaml: Sonar way
INFO: ------------- Run sensors on module some_project_name_AYPIfQ0SYDwiMQkHpeub
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=75ms
INFO: Sensor IaC Terraform Sensor [iac]
INFO: Sensor IaC Terraform Sensor is restricted to changed files only
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC Terraform Sensor [iac] (done) | time=296ms
INFO: Sensor IaC CloudFormation Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC CloudFormation Sensor [iac] (done) | time=421ms
INFO: Sensor IaC Kubernetes Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC Kubernetes Sensor [iac] (done) | time=199ms
INFO: Sensor PL/SQL Sensor [plsql]
INFO: Sensor PL/SQL Sensor is restricted to changed files only
WARN: The Data Dictionary is not configured for the PLSQL analyzer, which prevents rule(s) S3641, S3921, S3618, S3651 from raising issues. See https://docs.sonarqube.org/latest/analysis/languages/plsql/
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor PL/SQL Sensor [plsql] (done) | time=822ms
INFO: Sensor C# Project Type Information [csharp]
INFO: Sensor C# Project Type Information [csharp] (done) | time=4ms
INFO: Sensor C# Analysis Log [csharp]
INFO: Sensor C# Analysis Log [csharp] (done) | time=50ms
INFO: Sensor C# Properties [csharp]
INFO: Sensor C# Properties [csharp] (done) | time=0ms
INFO: Sensor HTML [web]
INFO: Sensor HTML is restricted to changed files only
INFO: Sensor HTML [web] (done) | time=92ms
INFO: Sensor XML Sensor [xml]
INFO: Sensor XML Sensor is restricted to changed files only
INFO: Sensor XML Sensor [xml] (done) | time=4ms
INFO: Sensor TextAndSecretsSensor [text]
INFO: Sensor TextAndSecretsSensor is restricted to changed files only
INFO: 4 source files to be analyzed
INFO: 4/4 source files have been analyzed
INFO: Sensor TextAndSecretsSensor [text] (done) | time=106ms
INFO: Sensor VB.NET Project Type Information [vbnet]
INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=4ms
INFO: Sensor VB.NET Analysis Log [vbnet]
INFO: Sensor VB.NET Analysis Log [vbnet] (done) | time=20ms
INFO: Sensor VB.NET Properties [vbnet]
INFO: Sensor VB.NET Properties [vbnet] (done) | time=0ms
INFO: Sensor Python Sensor [python]
WARN: Your code is analyzed as compatible with python 2 and 3 by default. This will prevent the detection of issues specific to python 2 or python 3. You can get a more precise analysis by setting a python version in your configuration via the parameter "sonar.python.version"
INFO: Using cached data to retrieve global symbols.
INFO: Cached information of global symbols will be used for 2 out of 2 main files. Global symbols will be recomputed for the remaining files.
INFO: Fully optimized analysis can be performed for 2 out of 2 files.
INFO: Partially optimized analysis can be performed for 2 out of 2 files.
INFO: Starting global symbols computation
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Starting rules execution
INFO: 2 source files to be analyzed
INFO: 2/2 source files have been analyzed
INFO: The Python analyzer was able to leverage cached data from previous analyses for 2 out of 2 files. These files were not parsed.
INFO: Sensor Python Sensor [python] (done) | time=1876ms
INFO: Sensor Cobertura Sensor for Python coverage [python]
INFO: Sensor Cobertura Sensor for Python coverage [python] (done) | time=189ms
INFO: Sensor PythonXUnitSensor [python]
INFO: Sensor PythonXUnitSensor [python] (done) | time=99ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=3ms
INFO: Sensor JavaScript inside YAML analysis [javascript]
INFO: No input files found for analysis
INFO: Hit the cache for 0 out of 0
INFO: Miss the cache for 0 out of 0
INFO: Sensor JavaScript inside YAML analysis [javascript] (done) | time=93ms
INFO: Sensor JavaScript inside HTML analysis [javascript]
INFO: 20 source files to be analyzed
INFO: 20/20 source files have been analyzed
INFO: Hit the cache for 20 out of 20
INFO: Miss the cache for 0 out of 20
INFO: Sensor JavaScript inside HTML analysis [javascript] (done) | time=8128ms
INFO: Sensor CSS Rules [javascript]
INFO: Sensor CSS Rules is restricted to changed files only
INFO: 4 source files to be analyzed
INFO: 4/4 source files have been analyzed
INFO: Hit the cache for 0 out of 0
INFO: Miss the cache for 0 out of 0
INFO: Sensor CSS Rules [javascript] (done) | time=497ms
INFO: Sensor Ruby Sensor [ruby]
INFO: Sensor Ruby Sensor is restricted to changed files only
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor Ruby Sensor [ruby] (done) | time=5098ms
INFO: Sensor SimpleCov Sensor for Ruby coverage [ruby]
INFO: Sensor SimpleCov Sensor for Ruby coverage [ruby] (done) | time=1ms
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend]
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend] (done) | time=7ms
INFO: Sensor Python HTML templates processing [securitypythonfrontend]
INFO: Found no 'settings.py' files in the project. We will consider 'autoescape' project option enabled.
INFO: Sensor Python HTML templates processing [securitypythonfrontend] (done) | time=83ms
INFO: Sensor PHP sensor [php]
INFO: Starting PHP symbol indexer
INFO: 3303 source files to be analyzed
INFO: 3303/3303 source files have been analyzed
INFO: Cached information of global symbols will be used for 3254 out of 3303 files. Global symbols were recomputed for the remaining files.
INFO: Starting PHP rules
INFO: 3303 source files to be analyzed
INFO: 310/3303 files analyzed, current file: some/path/to/php/file.php
INFO: 984/3303 files analyzed, current file: some/path/to/php/file.php
INFO: 1465/3303 files analyzed, current file: some/path/to/php/file.php
INFO: 2728/3303 files analyzed, current file: some/path/to/php/file.php
INFO: 3303/3303 source files have been analyzed
INFO: The PHP analyzer was able to leverage cached data from previous analyses for 3252 out of 3303 files. These files were not parsed.
INFO: No PHPUnit tests reports provided (see 'sonar.php.tests.reportPath' property)
INFO: No PHPUnit coverage reports provided (see 'sonar.php.coverage.reportPaths' property)
INFO: Sensor PHP sensor [php] (done) | time=55464ms
INFO: Sensor Analyzer for "php.ini" files [php]
INFO: Sensor Analyzer for "php.ini" files [php] (done) | time=51ms
INFO: Sensor IaC Docker Sensor [iac]
INFO: Sensor IaC Docker Sensor is restricted to changed files only
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC Docker Sensor [iac] (done) | time=728ms
INFO: Sensor Serverless configuration file sensor [security]
INFO: 0 Serverless function entries were found in the project
INFO: 0 Serverless function handlers were kept as entrypoints
INFO: Sensor Serverless configuration file sensor [security] (done) | time=7ms
INFO: Sensor AWS SAM template file sensor [security]
INFO: Sensor AWS SAM template file sensor [security] (done) | time=58ms
INFO: Sensor AWS SAM Inline template file sensor [security]
INFO: Sensor AWS SAM Inline template file sensor [security] (done) | time=399ms
INFO: Sensor javabugs [dbd]
INFO: Reading IR files from: /some/folder/some-service/.scannerwork/ir/java
INFO: No IR files have been included for analysis.
INFO: Sensor javabugs [dbd] (done) | time=9ms
INFO: Sensor pythonbugs [dbd]
INFO: Reading IR files from: /some/folder/some-service/.scannerwork/ir/python
INFO: Analyzing 2 functions to detect bugs.
INFO: Sensor pythonbugs [dbd] (done) | time=226ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Reading type hierarchy from: /some/folder/some-service/.scannerwork/ucfg2/java
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor JavaSecuritySensor [security] (done) | time=4ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading type hierarchy from: /some/folder/some-service/ucfg_cs2
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor CSharpSecuritySensor [security] (done) | time=1ms
INFO: Sensor PhpSecuritySensor [security]
INFO: Reading type hierarchy from: /some/folder/some-service/.scannerwork/ucfg2/php
INFO: Read 3571 type definitions
INFO: Reading UCFGs from: /some/folder/some-service/.scannerwork/ucfg2/php
ERROR: isAlive was interrupted
java.lang.InterruptedException: null
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:385)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:541)
at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)
at org.sonar.plugins.javascript.eslint.EslintBridgeServerImpl.isAlive(EslintBridgeServerImpl.java:405)
at org.sonar.plugins.javascript.eslint.EslintBridgeServerImpl.heartbeat(EslintBridgeServerImpl.java:138)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
section_end:1684178069:step_script
e[0Ke[31;1mERROR: Job failed: execution took longer than 1h0m0s seconds
e[0;m
Hi,
I don’t see any single part of analysis taking an inordinate amount of time. The longest operation takes 55464ms, which is under a minute. Is there any way you can add timestamping to each line of the log? I’d like to see if the time is taken before analysis or in the back-and-forth with the server at the beginning of analysis.
Â
Ann
Hi this is the timestamp available on gitlab. I will look for the actual log stored in the gitlab server.
The step script takes the most amount of time, and the entire pipeline ended up timing out.
Just to clarify, are you saying that the speed of the sonar scanner job for a merge request depends on the size of the entire code base and not just the committed code size?
If sonar scan doesn’t have an option to do that I’m afraid I can’t. Is that the case?
GitLab doesn’t provide that information to us unfortunately.
Hi,
Oof!
My my searching tells me this has been a sore point with GitLab for years. 
Since we can’t get exact timing, I’ve gone back to look more closely at your log. IIRC, these “x/y files analyzed…” lines are printed every 20 (15?) seconds if the sensor takes long at all, so this only takes a couple minutes at most:
INFO: 310/3303 files analyzed, current file: some/path/to/php/file.php INFO: 984/3303 files analyzed, current file: some/path/to/php/file.php INFO: 1465/3303 files analyzed, current file: some/path/to/php/file.php INFO: 2728/3303 files analyzed, current file: some/path/to/php/file.php
Other than that, I just don’t see where the time can be going, although I do seem from your screenshot that it’s taken in the analysis step. I’ll be honest & say I don’t remember how much more timing data debug logging will get us, but it’s worth trying. Can you add -Dsonar.verbose=true to your analysis command & post those logs, redacted as necessary?
Â
Ann
Can I instead do sonar.log.level=TRACE? I read that this also logs at the debug level but includes the timing as well.
Hi,
Works for me!
Â
Ann
