DevOps Sonar Analysis stuck at Running symbolic analysis for 'JS'

  • ALM used: Azure DevOps
  • CI system used: Azure DevOps
  • Languages of the repository: JS/TS (Angular 12)
  • Error observed
INFO: 21:08:31.001678 Tarjan found 151826 components

INFO: 21:08:31.166449 Variable type analysis: done

INFO: Analyzing 30789 ucfgs to detect vulnerabilities.

INFO: Taint analysis starting. Entrypoints: 2708

INFO: Running symbolic analysis for 'JS'

##[error]Exception in thread "HttpClient-1-Worker-7"

##[error]java.lang.OutOfMemoryError: Java heap space

at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2087)

at java.net.http/jdk.internal.net.http.PlainHttpConnection$ConnectEvent.lambda$handle$1(PlainHttpConnection.java:137)

at java.net.http/jdk.internal.net.http.PlainHttpConnection$ConnectEvent$$Lambda$1310/0x0000000100bef040.run(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

java.lang.OutOfMemoryError: Java heap space

at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2087)

at java.net.http/jdk.internal.net.http.PlainHttpConnection$ConnectEvent.lambda$handle$1(PlainHttpConnection.java:137)

at java.net.http/jdk.internal.net.http.PlainHttpConnection$ConnectEvent$$Lambda$1310/0x0000000100bef040.run(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

##[error]ERROR: isAlive was interrupted

java.lang.InterruptedException: null

at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:385)

at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)

at java.net.http/jdk.internal.net.http.HttpClientImpl.send(HttpClientImpl.java:541)

at java.net.http/jdk.internal.net.http.HttpClientFacade.send(HttpClientFacade.java:119)

at org.sonar.plugins.javascript.eslint.EslintBridgeServerImpl.isAlive(EslintBridgeServerImpl.java:331)

at org.sonar.plugins.javascript.eslint.EslintBridgeServerImpl.heartbeat(EslintBridgeServerImpl.java:121)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

The pipeline/Sonar Analysis stopped working as of 22nd November 2022, we haven’t made any changes to the configuration.

I’ve looked at similar topics and added the configuration for increasing memory but that did not help. Looking at the last successful run, there is enough spare memory, so not sure what’s going wrong.

1. INFO: Analysis total time: 12:34.332 s

INFO: ------------------------------------------------------------------------

INFO: EXECUTION SUCCESS

INFO: ------------------------------------------------------------------------

INFO: Total time: 12:45.166s

INFO: Final Memory: 1006M/1736M

INFO: ------------------------------------------------------------------------

Finishing: Run Code Analysis

Running analysis in verbose mode shows this error.

10:36:04.512 DEBUG: Adding stub module wreck

10:36:04.512 DEBUG: Adding stub module wrench

10:36:04.513 DEBUG: Adding stub module write

10:36:04.513 DEBUG: Merged with previously registered stub module 'write'

10:36:04.514 DEBUG: Adding stub module xar

10:36:04.516 DEBUG: Adding stub module xml-escape

10:36:04.516 DEBUG: Adding stub module x-ray

10:36:04.516 DEBUG: Adding stub module yesql

10:36:04.516 DEBUG: Adding stub module zdf

10:36:06.345 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with argunsubscribe .

10:36:06.346 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with argunsubscribe .

10:36:06.596 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with argpadEnd .

10:36:06.596 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with argpadEnd .

10:36:07.921 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with argtest .

10:36:07.921 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with argtest .

10:36:11.198 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M.

10:36:11.207 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with arglength .

10:36:11.207 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with arglength .

10:36:11.208 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with arglength .

10:36:11.208 DEBUG: Did not expect to visit symbol class com.sonar.security.E.D.B.M with arglength .

Update:

Increased SONAR_SCANNER_OPTS to -Xmx4048m (wanted this to be 4G but just replaced from 2048) and the analysis was completed successfully but took a really long time to complete.

Past runs would take around 12-13 min, this time it took more than 26 min.

See below for previous and current run comparison

Before:

INFO: Sensor JsSecuritySensor [security] (done) | time=558942ms
.
.
.
INFO: Analysis total time: 11:32.345 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 11:42.393s
INFO: Final Memory: 1006M/1736M
INFO: ------------------------------------------------------------------------
Finishing: Run Code Analysis

After

INFO: Sensor JsSecuritySensor [security] (done) | time=1303568ms
.
.
.
INFO: Analysis total time: 26:53.355 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 27:06.658s
INFO: Final Memory: 2736M/4048M
INFO: ------------------------------------------------------------------------
Finishing: Run Code Analysis

After 22nd Nov 2022, it seems like it’s taking awful long time to complete JsSecuritySensor.

Hey @Amit_Kamble,

thank you for the detailed logs and investigation.

With the previous release, we added new built-in support in the security analyzer for over 200 JS libraries. Depending on what libraries your code is using, it might be that the analysis is much more detailed now, and needs more resources.

How big is your project? how many lines of JS/TS code does it have?

Best,
karim.

Hi Karim,

Thanks for explaining. It’s an Angular project with 270,458 lines of code. We are only scanning the src/app folder.

Is there a way we can disable libraries that we don’t need to use? or will that have to be per rule?
Having support for all libraries is great, but that adds a significant amount of time to the pipeline builds.

Thanks
Regards

Hello @Amit_Kamble,

Thanks for the information. 270k lines of code is not small, but the more than doubling of analysis time would be interesting to understand to see if there is a valid reason behind it or not.

Understandable. Unfortunately, there is no way of disabling support for specific libraries. Only complete rules can be activated/deactivated.

Is test code also correctly identified as such by setting the correct analysis parameters? see Narrowing the Focus | SonarQube Docs

Is the project by any chance open-source?

Thanks again Karim,

Yeah, 270k is not small and this will keep growing. But considering that 200+ rules were introduced in the background and overnight the analysis time has doubled. Is there a way we can check what rules are causing these?

We have added spec files to the source exclusion list but can add them to test exclusions too and check.

Considering the build times in the pipeline, we have currently paused analysis and 40+ minutes is not ideal. Would really appreciate it if you could help us with analyzing why it’s taking so long.

Thanks
Regards

Hi Karim,

Was looking at raw logs and noticed that the scanner is going through node_modules but we have explicitly set the analyzer folder to src/app ~ cliSources: src/app

Hi Karim,

Just checking so if we can get some assistance here. We’ve paused analysis for quite some time now.

Cheers

Hi @Karim_El_Ouerghemmi ,

Just checking so if we can get some assistance here. We’ve paused analysis for quite some time now.

Cheers

Hey @Amit_Kamble ,

thank you for the additional information.

Those debug log messages do not indicate that the files are also scanned. To be sure, you can also add **/node_modules/** to the exclusions list.

Since this seems to be happening during the security analysis, a possibility is to try out the analysis with different rules out of this list deactivated in the used profile (when you deactivate in the JavaScript profile, you should also deactivate in the TypeScript profile to be sure).

If this doesn’t help, to further investigate if the time increase is due to a regression, I’ll need a way to reproduce this. If your project is not open-source, would you be willing to share some intermediate representation files created by the scanner during analysis in its working directory? (we can do that in a private thread).