CLI Scanner Runs Out of Memory

SonarQube Server / Community Build
1

Must-share information (formatted with Markdown):

  • which versions are you using: SonarQube 10.3 build 82913, SonarScanner 5.0.1.3006
  • how is SonarQube deployed: Docker
  • what are you trying to achieve: Trying to analyze a front-end codebase which has over 800 javascript and typescript files
  • what have you tried so far to achieve this: I’ve attempted to run the CLI analyzer to scan the codebase. It eats up a lot of memory and fails before it can complete scanning.

I have given it 40 gigabytes of RAM and it ran for an hour and a half and only analyzed less than 20% of the code before it ran out of memory and crashed. The logs below are from a run when I only gave it 16 GB of RAM, but I have in the past given it up to 40.

INFO: 153/811 files analyzed, current file: C:/FrontEnd/src/components/search/nightsRange.js
ERROR: The analysis will stop due to the Node.js process running out of memory (heap size limit 16048 MB)
ERROR: You can see how Node.js heap usage evolves during analysis with "sonar.javascript.node.debugMemory=true"
ERROR: Try setting "sonar.javascript.node.maxspace" to a higher value to increase Node.js heap size limit
ERROR: If the problem persists, please report the issue at https://community.sonarsource.com
ERROR: Failure during analysis
java.lang.IllegalStateException: The bridge server is unresponsive
        at org.sonar.plugins.javascript.bridge.BridgeServerImpl.request(BridgeServerImpl.java:393)
        at org.sonar.plugins.javascript.bridge.BridgeServerImpl.analyzeJavaScript(BridgeServerImpl.java:351)
        at org.sonar.plugins.javascript.bridge.AnalysisWithWatchProgram.analyze(AnalysisWithWatchProgram.java:146)
        at org.sonar.plugins.javascript.bridge.AnalysisWithWatchProgram.analyzeTsConfig(AnalysisWithWatchProgram.java:121)
        at org.sonar.plugins.javascript.bridge.AnalysisWithWatchProgram.analyzeFiles(AnalysisWithWatchProgram.java:78)
        at org.sonar.plugins.javascript.bridge.JsTsSensor.analyzeFiles(JsTsSensor.java:125)
        at org.sonar.plugins.javascript.bridge.AbstractBridgeSensor.execute(AbstractBridgeSensor.java:73)
        at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:64)
        at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:88)
        at org.sonar.scanner.sensor.ModuleSensorsExecutor.lambda$execute$1(ModuleSensorsExecutor.java:61)
        at org.sonar.scanner.sensor.ModuleSensorsExecutor.withModuleStrategy(ModuleSensorsExecutor.java:79)
        at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:61)
        at org.sonar.scanner.scan.SpringModuleScanContainer.doAfterStart(SpringModuleScanContainer.java:82)
        at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:201)
        at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:180)
        at org.sonar.scanner.scan.SpringProjectScanContainer.scan(SpringProjectScanContainer.java:398)
        at org.sonar.scanner.scan.SpringProjectScanContainer.scanRecursively(SpringProjectScanContainer.java:394)
        at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:363)
        at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:201)
        at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:180)
        at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:139)
        at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:201)
        at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:180)
        at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:71)
        at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:65)
        at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
        at jdk.proxy1/jdk.proxy1.$Proxy0.execute(Unknown Source)
        at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
        at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
        at org.sonarsource.scanner.cli.Main.execute(Main.java:126)
        at org.sonarsource.scanner.cli.Main.execute(Main.java:81)
        at org.sonarsource.scanner.cli.Main.main(Main.java:62)
Caused by: java.io.IOException: HTTP/1.1 header parser received no bytes
        at java.net.http/jdk.internal.net.http.HttpClientImpl.send(Unknown Source)
        at java.net.http/jdk.internal.net.http.HttpClientFacade.send(Unknown Source)
        at org.sonar.plugins.javascript.bridge.BridgeServerImpl.request(BridgeServerImpl.java:388)
        ... 36 common frames omitted
Caused by: java.io.IOException: HTTP/1.1 header parser received no bytes
        at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Unknown Source)
        at java.net.http/jdk.internal.net.http.Http1Response$HeadersReader.onReadError(Unknown Source)
        at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Unknown Source)
        at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Unknown Source)
        at java.net.http/jdk.internal.net.http.common.SequentialScheduler$LockingRestartableTask.run(Unknown Source)
        at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(Unknown Source)
        at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException: EOF reached while reading
        at java.net.http/jdk.internal.net.http.Http1AsyncReceiver$Http1TubeSubscriber.onComplete(Unknown Source)
        at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadSubscription.signalCompletion(Unknown Source)
        at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.read(Unknown Source)
        at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowTask.run(Unknown Source)
        at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(Unknown Source)
        at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(Unknown Source)
        at java.net.http/jdk.internal.net.http.common.SequentialScheduler.runOrSchedule(Unknown Source)
        at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$InternalReadSubscription.signalReadable(Unknown Source)
        at java.net.http/jdk.internal.net.http.SocketTube$InternalReadPublisher$ReadEvent.signalEvent(Unknown Source)
        at java.net.http/jdk.internal.net.http.SocketTube$SocketFlowEvent.handle(Unknown Source)
        at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.handleEvent(Unknown Source)
        at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.lambda$run$3(Unknown Source)
        at java.base/java.util.ArrayList.forEach(Unknown Source)
        at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(Unknown Source)
2

Hi,

Welcome to the community!

Would it be feasible to analyze your project as multiple sub-sets? I.e. instead of 1 project of 800 files, maybe… 10 projects of 80 (or something)?

 
Ann

3

Hi Ann,

The server that will run the scan only has 8 GB of RAM, and I’ve noticed that the CLI scanner eats through that after analyzing about 40-50 JS files. I’m not sure if there’s a logical way of breaking up the solution into chunks of 40 files. Does that affect duplication scanning?

That said, I found a workaround: the npm scanner seems to be able to scan the entire solution in under 5 minutes using only 4 GB of RAM. I’m not sure why the CLI scanner is so much slower and requires so much memory.

1 Like
4

Hi,

I’m glad you worked through this.

To answer your question, yes splitting it up into multiple projects would affect duplication detection.

 
Ann

6

Hi @bepm

I understand that

I have given it 40 gigabytes of RAM

But I just want to make sure you did that by setting sonar.javascript.node.maxspace to very high value (i.e. 40000). Did you use this parameter?

We suggest in our docs to set it to 8192 for big projects.

9

Hi Łukasz,

Yes, I set sonar.javascript.node.maxspace=40000. I had tried 8192, but it would crash after analyzing about 40 files. Setting it to 40000 allowed me to get almost 200 files scanned before it ran out of memory and crashed.

Thanks,
Ben

1 Like
13

Hi @bepm

I would like to investigate this particular statement, in order to determine if there is something to improve on our side.

The Scanner for NPM is a wrapper around the Scanner CLI, so I am very surprised that it leads to different memory consumption. The key factor is likely the number of files that are processed during the analysis. Can you double-check that you have the same configuration for input files in both cases (same value for sonar.sources, sonar.tests, sonar.inclusions, sonar.exclusions, …)?

One typical case where analysis goes crazy on JS projects is when the node_modules folder is included in the analysis.

That would be super helpful if you could share (even privately) the logs of the two analyses (one with Scanner CLI and one with Scanner for NPM) with verbose output enabled.