which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
Sonarqube version 6.7.1
Sonar scanner 3.2.0.1227-linux
what are you trying to achieve
I am trying to analyse 4500 Cobol programs
what have you tried so far to achieve this
I have successfully analysed 500programs but when I increase the threshold to 4500 then it fails the analysis
Saying ‘Killed’
I am exporting SONAR_SCANNER_OPTS="-Xmx4096m"
Sonarqube container memory is
-Xmx2048m
How can I resolve this?
Hello G Ann,
Hope you are fine.
The situation we have is java heap space out of memory error in sonarqube. We are getting this error in background tasks.
I want to understand the difference between SONAR scanner opts , ceopts, webopts, esopts and how do they consume the RAM. We have a license of 5million lines of code.
We host 220 sonar instances. We want to have good customer experience and hence require the info on how can we manage the memory in sonarqube.
I’m going to start by restating what I understand from this thread so far:
You’re analyzing Cobol
A job that analyzed 500 files (where each file is a program) succeeds
When you pulled a larger file set from the DB, 4500, the processing of the background task failed with a message that started with “Killed”
If I’m correct on all points…
It would probably be helpful for you to share the full stacktrace if what’s below doesn’t get you where you need to be.
The SONAR_SCANNER_OPTS was a good guess, but they only affect the analysis side, and have no impact on background task processing, which takes place on the SonarQube server.
The other “opts” settings you mention do affect the SonarQube server. If the problem is that background task processing is running out of memory, then adjusting the sonar.ce.javaOpts should help. ce here stands for Compute Engine, which handles background task processing.
Hello G Ann,
Thank you so much for the above explanation.
I would like to clarify few things here.
We are trying to analyse 4500 Cobol programs via Jenkins.
We resolved the killed issue which I could see in Jenkins console logs. It was because the user was using a wrong docker image template to run the analysis.
We now have the issue wherein we are not able to upload a huge report of 2gb actual size and a compressed size of 724mb.
We are using postgresql database.
Can you let us know in which particular config file and which parameter in need to change in postgresql configuration so that the report gets uploaded to sonarqube successfully?
After googling few website I could see max_packet_allowed parameter for mysql database which can resolve the problem but I am not aware about the above equivalent in postgresql.
Can you try allocating more memory to the CE and trying again, please? BTW, you’ll make that change to sonar.ce.javaOpts in $SONARQUBE_HOME/conf/sonar.properties.
Hello G Ann,
I tried with modifying the sonar.ce.javaOpts parameters to the following:
sonar.ce.javaOpts=-Xmx2048m -Xms1024m -XX:+HeapDumpOnOutOfMemoryError
and restarted sonarqube.
Now after analysing i am getting the following error:
java.lang.IllegalStateException: Fail to extract report AWoS_EQegSCCR3S3bWEV from database
at org.sonar.server.computation.task.projectanalysis.step.ExtractReportStep.execute(ExtractReportStep.java:66)
at org.sonar.server.computation.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:64)
at org.sonar.server.computation.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:52)
at org.sonar.server.computation.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:73)
at org.sonar.ce.taskprocessor.CeWorkerImpl.executeTask(CeWorkerImpl.java:134)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:97)
at org.sonar.ce.taskprocessor.CeWorkerImpl.withCustomizedThreadName(CeWorkerImpl.java:81)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:73)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:43)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at org.sonar.api.internal.apachecommons.io.IOUtils.copyLarge(IOUtils.java:1793)
at org.sonar.api.internal.apachecommons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.sonar.api.internal.apachecommons.io.IOUtils.copy(IOUtils.java:1744)
at org.sonar.api.utils.ZipUtils.copy(ZipUtils.java:169)
at org.sonar.api.utils.ZipUtils.unzipEntry(ZipUtils.java:109)
at org.sonar.api.utils.ZipUtils.unzip(ZipUtils.java:92)
at org.sonar.api.utils.ZipUtils.unzip(ZipUtils.java:61)
at org.sonar.server.computation.task.projectanalysis.step.ExtractReportStep.execute(ExtractReportStep.java:64)
I am running sonarqube within a container.
Can you let me know what do i need to do next?
Hello Ann,
Thanks for your response but a 2gb report is something which is unacceptable.
I would like to know what is the default report size which we can suggest our users to upload to sonarqube?
Is is possible that a user can analyse the project in two chunks for example 500 programs are getting analysed easily without any changes in the ce opts params.
So is there a way where 4500 programs can be divided and analysed easily?
If you find a report that size unacceptable, then yes you need to divide and conquer. My experience setting up Cobol analysis at my previous company tells me that each program/file is pretty much independent, and so groupings are at least somewhat arbitrary. (Yes, I understand that there are probably sets of programs that interact.)
So by all means, divide your 4500 programs into subsets and analyze them independently. You’ll just want to make sure you give each subset a unique sonar.projectKey value.