We suddenly started getting out of memory exceptions when scanning our repo. Nothing major changed from our end. See error below.
Tried to increase heap by setting environment variable SONAR_SCANNER_OPTS: -Xmx16G
using dotnet scanner version 5.8 but tried different ones.
ALM used - GitHub
CI system used - Github Actions
Scanner command used when applicable (private details masked): ..sonar\scanner\dotnet-sonarscanner begin
Languages of the repository - c#, html, javascript
Error observed (wrap logs/code around with triple quotes ``` for proper formatting)
INFO: Sensor CSS Rules [javascript]
INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
INFO: Sensor CSS Rules [javascript] (done) | time=0ms
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend]
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend] (done) | time=0ms
INFO: Sensor Serverless configuration file sensor [security]
INFO: 0 Serverless function entries were found in the project
INFO: 0 Serverless function handlers were kept as entrypoints
INFO: Sensor Serverless configuration file sensor [security] (done) | time=0ms
INFO: Sensor AWS SAM template file sensor [security]
INFO: Sensor AWS SAM template file sensor [security] (done) | time=0ms
INFO: Sensor AWS SAM Inline template file sensor [security]
INFO: Sensor AWS SAM Inline template file sensor [security] (done) | time=0ms
INFO: Sensor javabugs [dbd]
INFO: Reading IR files from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\.sonar\ir\java
INFO: No IR files have been included for analysis.
INFO: Sensor javabugs [dbd] (done) | time=0ms
INFO: Sensor pythonbugs [dbd]
INFO: Reading IR files from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\.sonar\ir\python
INFO: No IR files have been included for analysis.
INFO: Sensor pythonbugs [dbd] (done) | time=15ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Reading type hierarchy from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\.sonar\ucfg2\java
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor JavaSecuritySensor [security] (done) | time=6ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading type hierarchy from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\ucfg_cs2
INFO: Read 177 type definitions
INFO: Reading UCFGs from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\ucfg_cs2
INFO: 20:55:43.1743835 Building Runtime Type propagation graph
INFO: 20:55:43.4375671 Running Tarjan on 25153 nodes
INFO: 20:55:43.5000678 Tarjan found 25054 components
INFO: 20:55:43.5938218 Variable type analysis: done
INFO: 20:55:43.5938218 Building Runtime Type propagation graph
INFO: 20:55:43.7031986 Running Tarjan on 25153 nodes
INFO: 20:55:43.7344551 Tarjan found 25054 components
INFO: 20:55:43.7819019 Variable type analysis: done
INFO: Analyzing 2598 ucfgs to detect vulnerabilities.
INFO: All rules entrypoints : 24
INFO: Retained UCFGs : 1384
INFO: Taint analysis starting. Entrypoints: 24
INFO: Running symbolic analysis for 'CSHARP'
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000004f8400000, 5209325568, 0) failed; error='The paging file is too small for this operation to complete' (DOS error/errno=1455)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 5209325568 bytes for Failed to commit area from 0x00000004f8400000 to 0x000000062ec00000 of length 5209325568.
# An error report file with more information is saved as:
# D:\a\mvcwebcore\mvcwebcore\hs_err_pid1672.log
The SonarScanner did not complete successfully
20:55:55.202 Post-processing failed. Exit code: 1
Error: Process completed with exit code 1.
Potential workaround - no work arround - suddenly started happening
This error message was a new one for me, so I searched and landed (predictably?) at SO:
The message above means that youâre running so many programs on your PC that there is no memory left to run one more. This isnât a Java problem and no Java option is going to change this.
This is running on a machine that is ONLY running sonar scanning. There is nothing else on it and its on a github action runner - so literally nothing else open.
Itâs possible, but not that simple in github actions yet. They have a feature in beta mode that lets you do that, but you have to apply for it, etc.
As I mentioned though nothing changed on our end. Also, these runners have 7 GB of ram and its only running this. Are we saying its expected that sonar needs more than 7 GB to run? This repo has 40k lines of code. I doubt it really needs more than 8 GB.
Thanks for reporting this.
Weâve recently identified an issue with the memory consumption of the security analysis. Weâre implementing a fix and will deploy it to SonarCloud as soon as we can.
In the meantime, would you be willing to share with us the content of the mvcwebcore\.sonarqube\out\ucfg_cs2 generated during the analysis?
That way we could either ensure that the fix will resolve your performance problems, or investigate further the issue if necessary.
Thanks for your follow up. Iâm working on this now. We had disabled things to avoid the errors so iâm getting it back up and running. Please let me know when there is a fix to test. Thank you.
Some memory optimization has been deployed on SonarCloud, you should try again to see if it improves the situation on your side.
If it does not change, we would still be interested in the content of the mvcwebcore\.sonarqube\out\ucfg_cs2 folder.
Please do not post them here, I will reach out to you privately.
It appears to be the same thing. Is there anything i have to do from my end to update to your fix?
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=0ms
INFO: Sensor CSS Rules [javascript]
INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
INFO: Sensor CSS Rules [javascript] (done) | time=0ms
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend]
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend] (done) | time=0ms
INFO: Sensor Serverless configuration file sensor [security]
INFO: 0 Serverless function entries were found in the project
INFO: 0 Serverless function handlers were kept as entrypoints
INFO: Sensor Serverless configuration file sensor [security] (done) | time=0ms
INFO: Sensor AWS SAM template file sensor [security]
INFO: Sensor AWS SAM template file sensor [security] (done) | time=0ms
INFO: Sensor AWS SAM Inline template file sensor [security]
INFO: Sensor AWS SAM Inline template file sensor [security] (done) | time=0ms
INFO: Sensor javabugs [dbd]
INFO: Reading IR files from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\.sonar\ir\java
INFO: No IR files have been included for analysis.
INFO: Sensor javabugs [dbd] (done) | time=0ms
INFO: Sensor pythonbugs [dbd]
INFO: Reading IR files from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\.sonar\ir\python
INFO: No IR files have been included for analysis.
INFO: Sensor pythonbugs [dbd] (done) | time=0ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Reading type hierarchy from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\.sonar\ucfg2\java
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor JavaSecuritySensor [security] (done) | time=0ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading type hierarchy from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\ucfg_cs2
INFO: Read 177 type definitions
INFO: Reading UCFGs from: D:\a\mvcwebcore\mvcwebcore\.sonarqube\out\ucfg_cs2
INFO: 18:15:30.7573343 Building Runtime Type propagation graph
INFO: 18:15:30.8979669 Running Tarjan on 25153 nodes
INFO: 18:15:30.9604676 Tarjan found 25054 components
INFO: 18:15:31.0385952 Variable type analysis: done
INFO: 18:15:31.0385952 Building Runtime Type propagation graph
INFO: 18:15:31.163599 Running Tarjan on 25153 nodes
INFO: 18:15:31.179226 Tarjan found 25054 components
INFO: 18:15:31.2104765 Variable type analysis: done
INFO: Analyzing 2598 ucfgs to detect vulnerabilities.
INFO: All rules entrypoints : 24
INFO: Retained UCFGs : 1384
INFO: Taint analysis starting. Entrypoints: 24
INFO: Running symbolic analysis for 'CSHARP'
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
INFO: Total time: 4:27.082s
ERROR: Error during SonarScanner execution
INFO: Final Memory: 104M/350M
java.lang.OutOfMemoryError: Java heap space
INFO: ------------------------------------------------------------------------
at java.base/java.util.HashMap.resize(HashMap.java:700)
at java.base/java.util.HashMap.putVal(HashMap.java:625)
at java.base/java.util.HashMap.put(HashMap.java:608)
at com.sonar.security.E.D.A.S.A(na:2877)
at com.sonar.security.E.D.A.P.C(na:850)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:2252)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:2252)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:1851)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:2252)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:2252)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:2252)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:1851)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:2252)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:1851)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:2252)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:2252)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:2252)
at com.sonar.security.E.D.A.H.A(na:590)
at com.sonar.security.E.D.A.D.C(na:1851)
at com.sonar.security.E.D.A.H.A(na:590)
ERROR:
The SonarScanner did not complete successfully
18:16:58.792 Post-processing failed. Exit code: 1
Error: Process completed with exit code 1.
We are looking into it, we did not manage to identify the problem yet, it turns out to be tricky.
In the meantime, if your analysis is failing, you can disable the rules related to the Java Security analyzer as a temporary workaround (these are the rules: S2076, S2078, S2083, S2091, S2631, S3649, S5131, S5135, S5144, S5145, S5146, S5147, S5167, S5334, S5883, S6096, S6287, S6350, S6384, S6173, S6390, S6398, S6399). This is definitely far from ideal, but it should at least let you run the analysis for all the other rules in the meantime.
Indeed, you should disable the same set of rules, but for C# of course! (Some of them may not be available for C#, you donât have to do anything with them then).
I come back to you with good news: we managed to identify and fix a memory bottleneck in the analyzer. I tested with the data (âucfgsâ) you provided, and the analysis goes through smoothly.
The change is already deployed on SonarCloud, I encourage you to try again (with all rules mentioned previously re-enabled).
Thanks for your patience and collaboration. Let us know how it goes.
Best,
Quentin
We are still getting this issue on SonarCloud, building a dotnet 6 application.
We are using Azure Hosted agents and the settings below:
Version: SonarScanner for MSBuild 5.13
ALM used - Azure DevOps
CI system used - Azure DevOps
Azure Hosted Agents: windows-latest
Scanner command used when applicable (private details masked):
task: SonarCloudPrepare@1
displayName: âPrepare analysis on SonarCloudâ
inputs: masked*
env:
SONAR_DOTNET_ENABLE_CONCURRENT_EXECUTION: true
SONAR_SCANNER_OPTS: -Xmx8G
In log: D:\a_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.35.0\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe begin masked params**
Languages of the repository - c#
Error observed:
Exception of type âSystem.OutOfMemoryExceptionâ was thrown.
----> System.OutOfMemoryException : Exception of type âSystem.OutOfMemoryExceptionâ was thrown.
It is unfortunate that you still face the issue.
Without more information, it is difficult to do anything on our side.
Would you consider sharing (privately) the .sonarqube\out\ucfg_cs2 folder generated after the analysis? It is a great source of information to understand what is happening.
If you have more details to share about your problem, please open a new topic.