OutOfMemory Error for an XL Project on Azure DevOps Pipeline

aozmez · June 18, 2021, 10:02am

versions used: SonarQube EE v8.9.0, Sonar Scanner on Azure DevOps (latest)
error observed:

2021-06-18T08:56:15.5579959Z INFO: ------------------------------------------------------------------------
2021-06-18T08:56:15.5581478Z INFO: Total time: 5:19:03.223s
2021-06-18T08:56:20.7113091Z INFO: Final Memory: 2103M/4096M
2021-06-18T08:56:20.7165106Z INFO: ------------------------------------------------------------------------
2021-06-18T08:56:20.7191688Z ##[error]ERROR: Error during SonarScanner execution
2021-06-18T08:56:20.7193310Z ERROR: Error during SonarScanner execution
2021-06-18T08:56:20.7194438Z ##[error]java.lang.OutOfMemoryError: Java heap space
2021-06-18T08:56:20.7195843Z java.lang.OutOfMemoryError: Java heap space
2021-06-18T08:56:20.7203892Z ##[error]at java.base/java.util.Comparator$$Lambda$1256/0x0000000800c85040.get$Lambda(Unknown Source)
	at java.base/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
	at java.base/java.lang.invoke.Invokers$Holder.linkToTargetMethod(Invokers$Holder)
	at java.base/java.util.Comparator.thenComparing(Comparator.java:215)
	at java.base/java.util.Comparator.thenComparingInt(Comparator.java:284)
	at com.sonar.A.A.A(na:2633)
	at com.sonar.security.analysis.D.J$_B.A(na:1022)
	at com.sonar.security.analysis.D.J$_B.<init>(na:1350)
	at com.sonar.security.analysis.D.J$_B.C(na:2253)
	at com.sonar.security.analysis.D.J$_B.A(na:3353)
	at com.sonar.security.analysis.D.C.A(na:1306)
	at com.sonar.security.analysis.D.J.A(na:2625)
2021-06-18T08:56:20.7211992Z 	at java.base/java.util.Comparator$$Lambda$1256/0x0000000800c85040.get$Lambda(Unknown Source)
2021-06-18T08:56:20.7212945Z 	at java.base/java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(DirectMethodHandle$Holder)
2021-06-18T08:56:20.7213754Z 	at java.base/java.lang.invoke.Invokers$Holder.linkToTargetMethod(Invokers$Holder)
2021-06-18T08:56:20.7218850Z 	at java.base/java.util.Comparator.thenComparing(Comparator.java:215)
2021-06-18T08:56:20.7219728Z 	at java.base/java.util.Comparator.thenComparingInt(Comparator.java:284)
2021-06-18T08:56:20.7220364Z 	at com.sonar.A.A.A(na:2633)
2021-06-18T08:56:20.7221152Z 	at com.sonar.security.analysis.D.J$_B.A(na:1022)
2021-06-18T08:56:20.7222248Z 	at com.sonar.security.analysis.D.J$_B.<init>(na:1350)
2021-06-18T08:56:20.7222896Z 	at com.sonar.security.analysis.D.J$_B.C(na:2253)
2021-06-18T08:56:20.7223593Z 	at com.sonar.security.analysis.D.J$_B.A(na:3353)
2021-06-18T08:56:20.7224279Z 	at com.sonar.security.analysis.D.C.A(na:1306)
2021-06-18T08:56:20.7224908Z 	at com.sonar.security.analysis.D.J.A(na:2625)
2021-06-18T08:56:20.7229439Z ##[error]at com.sonar.security.analysis.D.K.C(na:405)
	at com.sonar.security.analysis.D.K.A(na:2439)
	at com.sonar.security.analysis.D.A.B(na:3336)
	at com.sonar.security.analysis.D.A.C(na:978)
	at com.sonar.security.analysis.D.E.A(na:2957)
	at com.sonar.security.analysis.D.E.A(na:1060)
2021-06-18T08:56:20.7234172Z 	at com.sonar.security.analysis.D.K.C(na:405)
2021-06-18T08:56:20.7234916Z 	at com.sonar.security.analysis.D.K.A(na:2439)
2021-06-18T08:56:20.7235619Z 	at com.sonar.security.analysis.D.A.B(na:3336)
2021-06-18T08:56:20.7236284Z 	at com.sonar.security.analysis.D.A.C(na:978)
2021-06-18T08:56:20.7236980Z 	at com.sonar.security.analysis.D.E.A(na:2957)
2021-06-18T08:56:20.7237889Z 	at com.sonar.security.analysis.D.E.A(na:1060)
2021-06-18T08:56:20.7240499Z ##[error]at com.sonar.security.analysis.D.E.A(na:1075)
	at com.sonar.security.analysis.D.J$_B.A(na:3289)
2021-06-18T08:56:20.7242205Z 	at com.sonar.security.analysis.D.E.A(na:1075)
2021-06-18T08:56:20.7243384Z 	at com.sonar.security.analysis.D.J$_B.A(na:3289)
2021-06-18T08:56:20.7245352Z ##[error]at com.sonar.security.analysis.D.J$_B.C(na:2096)
	at com.sonar.security.analysis.D.J$_B.B(na:26)
	at com.sonar.security.analysis.D.J$_B.A(na:2328)
2021-06-18T08:56:20.7247372Z 	at com.sonar.security.analysis.D.J$_B.C(na:2096)
2021-06-18T08:56:20.7248133Z 	at com.sonar.security.analysis.D.J$_B.B(na:26)
2021-06-18T08:56:20.7248829Z 	at com.sonar.security.analysis.D.J$_B.A(na:2328)
2021-06-18T08:56:20.7250967Z ##[error]at com.sonar.security.analysis.D.C.A(na:2373)
	at com.sonar.security.analysis.D.J.A(na:2625)
	at com.sonar.security.analysis.D.K.C(na:405)
	at com.sonar.security.analysis.D.K.A(na:2439)
2021-06-18T08:56:20.7254177Z 	at com.sonar.security.analysis.D.C.A(na:2373)
2021-06-18T08:56:20.7254918Z 	at com.sonar.security.analysis.D.J.A(na:2625)
2021-06-18T08:56:20.7255774Z 	at com.sonar.security.analysis.D.K.C(na:405)
2021-06-18T08:56:20.7256617Z 	at com.sonar.security.analysis.D.K.A(na:2439)
2021-06-18T08:56:20.7259559Z ##[error]at com.sonar.security.analysis.D.A.B(na:3336)
	at com.sonar.security.analysis.D.A.C(na:978)
	at com.sonar.security.analysis.D.E.A(na:2957)
	at com.sonar.security.analysis.D.E.A(na:1060)
2021-06-18T08:56:20.7261621Z 	at com.sonar.security.analysis.D.A.B(na:3336)
2021-06-18T08:56:20.7262885Z 	at com.sonar.security.analysis.D.A.C(na:978)
2021-06-18T08:56:20.7263642Z 	at com.sonar.security.analysis.D.E.A(na:2957)
2021-06-18T08:56:20.7264508Z 	at com.sonar.security.analysis.D.E.A(na:1060)
2021-06-18T08:56:20.7265786Z ##[error]at com.sonar.security.analysis.D.E.A(na:1075)
ERROR:
2021-06-18T08:56:20.7268151Z 	at com.sonar.security.analysis.D.E.A(na:1075)

steps to reproduce:

Set the environment variable SONAR_SCANNER_OPTS to "-Xmx4G” in Azure DevOps Build Pipeline
Trigger the first time analysis for the XL Project (> 5.5M LoC)
Get the reported OutOfMemory error during the “Run Code Analysis” step

potential workaround:
Increase the SONAR_SCANNER_OPTS parameter value even further which may cause a resource bottleneck on the shared Build Machine

Monty_Dickerson · June 21, 2021, 9:42pm

Hi Ali Özmez,

Welcome to the SonarSource community.

Yes, 5.5M LoC is a large number. What languages, environments, and types of files are being analyzed in that repository?

Which files have the most number of lines? You could narrow the focus of your analysis to the code that is most relevant, and exclude files that are generated, constant, or irrelevant.

Have you tried increasing your VM, and heap to 8G of memory or larger, like SONAR_SCANNER_OPTS=-Xmx8192M -Xms8192M

—
Regards
Monty

aozmez · June 22, 2021, 7:40am

Hi Monty,

The project consists of the following languages: C#, CSS, JavaScript, HTML, and XML.
We will try increasing the memory limits to 8GB.

Regards,

Monty_Dickerson · June 22, 2021, 1:31pm

Hi Ali,

Which operating system and version of dotnet does the project run on?
Are you using our SonarScanner for .NET?

Monty

aozmez · June 22, 2021, 1:50pm

Hi Monty,
The project runs on Windows with .NET 4.8.
We are using SonarScanner for MSBuild 5.2.1.
Regards,

aozmez · June 23, 2021, 7:13am

Hello,
We managed to finish the analysis after increasing the memory limit to 8GB. We determined the OutOfMemory error was occuring during Tarjan Anlysis. This operation took around 3 hours by itself as you can see at the end of the following log info. This operation was not running with SonarQube Community Edition. Is it specific to Enterprise Edition and can we disable it on a project basis?

INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading type hierarchy from: E:\agent\_work\2\.sonarqube\out\ucfg_cs2
INFO: Read 27141 type definitions
INFO: Reading UCFGs from: E:\agent\_work\2\.sonarqube\out\ucfg_cs2
INFO: 07:22:03.5746302 Building Runtime Type propagation graph
INFO: 07:22:33.7205897 Running Tarjan on 3296067 nodes
INFO: 07:22:41.8903728 Tarjan found 3295114 components
INFO: 07:22:55.9295913 Variable type analysis: done
INFO: 07:22:55.9345063 Building Runtime Type propagation graph
INFO: 07:23:26.7503193 Running Tarjan on 3296343 nodes
INFO: 07:23:35.3845066 Tarjan found 3295390 components
INFO: 07:23:48.5407219 Variable type analysis: done
INFO: Analyzing 299535 ucfgs to detect vulnerabilities.
INFO: All rules entrypoints : 18368 Retained UCFGs : 121780
...
INFO: Sensor CSharpSecuritySensor [security] (done) | time=10302268ms

Monty_Dickerson · June 28, 2021, 8:18pm

Hi Ali,

Excellent. For a project of this size, 8GB is a very reasonable allocation.

I would not recommend disabling tarjan analysis. It is a very beneficial security feature of the commercial editions of SonarQube.

I understand that you are frustrated that it adds 3 hours to your analysis time. I am in communication with our product engineers who are working to improve its performance. And I will relay this concern to SonarQube product managers on your behalf as well.

—
Regards,
Monty

aozmez · July 5, 2021, 12:59pm

Hi Monty,

I noticed that SonarQube Major Release 9.0 just came up. Does it involve any improvements that may address this issue?

Regards,
Ali

Alexandre_Gigleux · July 5, 2021, 3:59pm

Hello,

It would be better to wait for SQ 9.1 because there is no significant performance improvement for C# security analysis with SQ 9.0.
You should track the progress of https://jira.sonarsource.com/browse/MMF-2432 which is scheduled for SQ 9.1 and should really help to get better performance on huge projects like yours.

Regards
Alex

system · July 12, 2021, 3:59pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.