Java: OutOfMemoryError: GC overhead limit exceeded in com.sonar.security.analysis

JayDi85 · February 10, 2019, 8:12am

Latest sonarcloud can’t process project (it’s takes much longer time than January version and raises out of memory error, -xmx4000M for maven command don’t help).

Maven -e logs:

    [INFO] ------------- Run sensors on module Mage Root
    [INFO] 1/1 source files have been analyzed
    [INFO] Sensor JavaXmlSensor [java]
    [INFO] 1 source files to be analyzed
    [INFO] Sensor JavaXmlSensor [java] (done) | time=5ms
    [INFO] Sensor HTML [web]
    [INFO] 1/1 source files have been analyzed
    [INFO] Sensor HTML [web] (done) | time=0ms
    [INFO] Sensor XML Sensor [xml]
    [INFO] 1 source files to be analyzed
    [INFO] Sensor XML Sensor [xml] (done) | time=6ms
    [INFO] Sensor JaCoCo XML Report Importer [jacoco]
    [INFO] 1/1 source files have been analyzed
    [INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=0ms
    [INFO] Sensor Zero Coverage Sensor
    [INFO] Sensor Zero Coverage Sensor (done) | time=2369ms
    [INFO] Sensor JavaSecuritySensor [security]
    [INFO] Reading UCFGs from: C:\mage\target\sonar\ucfg2\java
    [INFO] 11:41:16.113 Building Type propagation graph
    [INFO] 11:41:41.78 Running Tarjan on 1106348 nodes
    [INFO] 11:41:43.956 Tarjan found 1096687 components
    [INFO] Skipping component with 4902 elements and 12 types
    Exception in thread "LOG_FLUSHER" java.lang.OutOfMemoryError: GC overhead limit exceeded
    	at java.util.ArrayList.iterator(ArrayList.java:840)
    	at java.util.Collections$SynchronizedCollection.iterator(Collections.java:2031)
    	at com.persistit.Persistit.pollAlertMonitors(Persistit.java:2285)
    	at com.persistit.Persistit$LogFlusher.run(Persistit.java:192)
    ...
    ...
    [INFO] BUILD FAILURE
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 24:37 min
    [INFO] Finished at: 2019-02-10T11:45:16+04:00
    [INFO] Final Memory: 74M/2620M
    [INFO] ------------------------------------------------------------------------
    [ERROR] GC overhead limit exceeded -> [Help 1]
    java.lang.OutOfMemoryError: GC overhead limit exceeded
    	at java.util.HashMap.newNode(HashMap.java:1747)
    	at java.util.HashMap.putVal(HashMap.java:631)
    	at java.util.HashMap.put(HashMap.java:612)
    	at java.util.HashSet.add(HashSet.java:220)
    	at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
    	at com.sonar.security.analysis.A.A.B.A(Unknown Source)
    	at com.sonar.security.analysis.A.A.B$$Lambda$1710/1878453963.accept(Unknown Source)
    	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
    	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1553)
    	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
    	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
    	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
    	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
    	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
    	at com.sonar.security.analysis.A.A.B.<init>(Unknown Source)
    	at com.sonar.security.analysis.J.<init>(Unknown Source)
    	at com.sonar.security.F.execute(Unknown Source)
    	at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:48)
    	at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:85)
    	at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:62)
    	at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:82)
    	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
    	at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:122)
    	at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:408)
    	at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:403)
    	at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:360)
    	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)
    	at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:122)
    	at org.sonar.scanner.bootstrap.GlobalContainer.doAfterStart(GlobalContainer.java:126)
    	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:136)

Memory and cpu usage before error:

saberduck · February 11, 2019, 4:28pm

Hello @JayDi85,

the project you are analyzing is quite big. The first thing you might want to verify is that you are not analyzing accidentally some generated code, like parser, or state machine, as such code can make memory requirements explode (too many local variables to keep track of).

We are aware that analyzers became more memory demanding with the new taint analysis we are doing now, and we are looking into some ways how to optimize it.

Can you also try to allocate more memory for the analysis, to see if we are in some pathological state or just more memory is needed?

JayDi85 · February 12, 2019, 5:41pm

OK, will try to use 8000 MB memory for maven later (current sonarcloud services is down with connection reset error all day – sonarcloud.io/18.197.43.2).

saberduck · February 14, 2019, 10:49am

we are not aware of any outages recently, if you experience some please come back with more details and we will investigate them.

JayDi85 · February 16, 2019, 10:11am

It was my fault with dev proxy settings (services availability post).

Well, I’d tested with more memory but that’s not helps at all – sonar security checks creates too many string data/hashmaps. January version of sonar works fine, fast and without memory consumption: January takes 5-10 mins for full project analyse, February takes 20 minutes until crush.

JayDi85 · February 17, 2019, 6:09pm

8GB memory and -XX:-UseGCOverheadLimit is not helps… Sonar is out of memory.

I found that you changed default java rules set. It’s can be the reason – one of new changes is broken. But I can’t find new rules to disable it and test again.

Another dev bring current source code to the end (e.g. old sonar from local server works fine – I think it’s a new rules set on your last updates).

saberduck · February 17, 2019, 9:30pm

hello @JayDi85,

thanks for further investigation. Indeed the behavior is not normal, the failure happens during security analysis. Now, to be able to investigate further I would need to have a look at the code which is being analyzed. During the analysis we create intermediary representation of the code in the ucfg2/java directory (there will be thousands of files ending with .proto). It would be great if you could share the files with me, privately if you want (the files are transformed code and reveal some information like method names and control flow, so it’s up to your consideration if you can share them).

Otherwise if you just want your analysis to pass, you can disable security related rules S3649, S2076, S2091, S2078, S2631, S2083 and analysis should pass without any extra memory.

saberduck · February 17, 2019, 9:50pm

In fact, I’ve just noticed that your project is actually public on github, I will try to run analysis myself tomorrow

JayDi85 · February 17, 2019, 10:12pm

Anyway I’d uploaded proto files here (80 MB, 108k files): https://yadi.sk/d/ha1H7thkzvA-Wg

JayDi85 · February 17, 2019, 11:25pm

I can confirm: disables security rules can help – now project works fine and fast, without memory consumption.

[INFO] Sensor JavaSecuritySensor [security]
[INFO] Reading UCFGs from: C:\mage\target\sonar\ucfg2\java
[INFO] 03:13:54.04 Building Type propagation graph
[INFO] 03:13:54.044 Running Tarjan on 0 nodes
[INFO] 03:13:54.046 Tarjan found 0 components
[INFO] 03:13:54.048 Variable type analysis: done
[INFO] UCFGs: 0, excluded: 0, source entrypoints: 0
[INFO] No UCFGs have been included for analysis.
[INFO] Sensor JavaSecuritySensor [security] (done) | time=20ms

saberduck · February 21, 2019, 4:42pm

Thanks for the feedback. I created ticket to handle this issue, however this is a private project, so the ticket is not visible.

Nicolas_Peru · March 6, 2019, 9:43am

Hi,

Just to let you know that we worked on this issue and we managed to make the analysis pass without memory issue (2min30s and 2.5GB of memory consumed). This fix should come in the next release of sonarqube (and so with its deployment to SonarCloud).

Thanks a lot for your feedback. It has been very helpful.