Java OutOfMemoryError after upgrading to Sonar Scanner 6.0.0.4432

sodul · June 28, 2024, 4:47am

After upgrading our scanner from 5.0.1.3006 to 6.0.0.4432 we run into an OutOfMemoryError.

This is odd as we monitor our RAM consumption through cgroups while our pipeline is running in kubernetes and we never seem to reach full memory usage.

Here are our stats from cgroups:

Memory Limit: 7247757312 bytes (6.75 GiB)
Initial usage: 2082988032 bytes (1.94 GiB) 28.7%
Initial kernel: 20041728 bytes (19.11 MiB) 0.3%
Initial cache: 2048065536 bytes (1.91 GiB) 28.3%
Initial rss: 13787136 bytes (13.15 MiB) 0.2%

Max: 04:27:04.288
  usage: 5412970496 5.04 GiB 74.7%
  kernel: 118190080 112.71 MiB 1.6%
  cache: 2832175104 2.64 GiB 39.1%
  rss: 2459787264 2.29 GiB 33.9%

Max kernel: 04:26:40.684
  usage: 5396283392 5.03 GiB 74.5%
  kernel: 118272000 112.79 MiB 1.6%
  cache: 2831364096 2.64 GiB 39.1%
  rss: 2443702272 2.28 GiB 33.7%

Max cache: 04:26:55.759
  usage: 5412048896 5.04 GiB 74.7%
  kernel: 118210560 112.73 MiB 1.6%
  cache: 2832175104 2.64 GiB 39.1%
  rss: 2458570752 2.29 GiB 33.9%

Max rss: 04:27:08.049
  usage: 5404954624 5.03 GiB 74.6%
  kernel: 117604352 112.16 MiB 1.6%
  cache: 2825146368 2.63 GiB 39.0%
  rss: 2460463104 2.29 GiB 33.9%

Which shows we never exceeded 75% of the total memory for the container and our over process (rss) memory never exceeded 2.3GB.

We have these cli options to tune the heap:

SONAR_SCANNER_OPTS='-XX:MinRAMPercentage=60 -XX:MaxRAMPercentage=60 -XX:+UseCompressedClassPointers -XshowSettings:vm

And the logs show these settings put the heap at about 4GB:

VM settings:
    Max. Heap Size (Estimated): 4.05G
    Using VM: OpenJDK 64-Bit Server VM

04:25:18.928 INFO  Scanner configuration file: /usr/local/bin/sonar-scanner-6.0.0/conf/sonar-scanner.properties
04:25:18.930 INFO  Project root configuration file: /home/jenkins/agent/workspace/[..edited..]/sonar-project.properties
04:25:18.949 INFO  SonarScanner CLI 6.0.0.4432
04:25:18.950 INFO  Java 17.0.11 Eclipse Adoptium (64-bit)
04:25:18.951 INFO  Linux 5.10.219-208.866.amzn2.x86_64 amd64
04:25:18.952 INFO  SONAR_SCANNER_OPTS=-XX:MinRAMPercentage=60 -XX:MaxRAMPercentage=60 -XX:+UseCompressedClassPointers -XshowSettings:vm
04:25:18.973 INFO  User cache: /root/.sonar/cache
04:25:19.352 INFO  JRE provisioning: os[linux], arch[x86_64]
04:25:24.667 INFO  Communicating with SonarCloud
04:25:24.931 INFO  Starting SonarScanner Engine...
04:25:24.931 INFO  Java 17.0.11 Eclipse Adoptium (64-bit)
04:25:25.468 INFO  Load global settings

So the heap clearly never reached 4GB.

Log of the failure:

04:26:49.714 INFO  Load coverage report from '/home/jenkins/agent/workspace/[..edited..]-coverage.out'
04:27:07.914 ERROR [stderr] Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
04:27:07.914 ERROR [stderr] 	at java.base/java.lang.StringLatin1.newString(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.lang.String.substring(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.Scanner.hasNextLine(Unknown Source)
04:27:07.914 ERROR [stderr] 	at org.sonar.go.coverage.GoCoverSensor.parse(GoCoverSensor.java:201)
04:27:07.914 ERROR [stderr] 	at org.sonar.go.coverage.GoCoverSensor.lambda$execute$1(GoCoverSensor.java:81)
04:27:07.914 ERROR [stderr] 	at org.sonar.go.coverage.GoCoverSensor$$Lambda$2651/0x00007f463080f1c0.accept(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.stream.ReferencePipeline$7$1.accept(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
04:27:07.914 ERROR [stderr] 	at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source)
04:27:07.914 ERROR [stderr] 	at org.sonar.go.coverage.GoCoverSensor.execute(GoCoverSensor.java:81)
04:27:07.914 ERROR [stderr] 	at org.sonar.go.coverage.GoCoverSensor.execute(GoCoverSensor.java:75)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:62)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:75)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.sensor.ModuleSensorsExecutor.lambda$execute$1(ModuleSensorsExecutor.java:48)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.sensor.ModuleSensorsExecutor$$Lambda$720/0x00007f4630586498.run(Unknown Source)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.sensor.ModuleSensorsExecutor.withModuleStrategy(ModuleSensorsExecutor.java:66)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:48)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:64)
04:27:07.914 ERROR [stderr] 	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:123)
04:27:07.914 ERROR [stderr] 	at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:109)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:192)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:188)
04:27:07.914 ERROR [stderr] 	at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:159)
04:27:07.914 ERROR [stderr] 	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:123)
04:27:08.387 INFO  EXECUTION FAILURE
04:27:08.388 INFO  Total time: 1:49.486s

Colin · June 28, 2024, 5:00am

Hey there.

This needs to be changed to SONAR_SCANNER_JAVA_OPTS.

We’re currently discussing internally:

If we could have done a better job with backwards compatability
Where some Upgrade Notes for the scanner belong

sodul · June 28, 2024, 6:16pm

I can easily set both but I’m finding it really odd that this might be the issue if I do see this in the logs, as it means that the java process is taking the 60% settings into account as 6.75G*0.6=4.05G:

I’ll rebuild our CI container with the latest scanner version released yesterday and will set both variables.

sodul · June 28, 2024, 9:34pm

Sorry for the delay, we had to deal with the GitHub outage today.

I can confirm that our pipelines are working with scanner 6.1.0.4477 (released yesterday) when using SONAR_SCANNER_JAVA_OPTS. We set the old SONAR_SCANNER_OPTS for temporary backward compatibility for now.