As said by you i have created new topic and assume you already have the log so i am not attaching the log here.
History from previous Topic
I have a java projects and it has few subprojects in it and scanning this project used to take nearly 8 hours but from few days it has increased to 12-13 hours. Please let me know how we can find sudden increase in the scan time? How can identify where exactly the scan time got increased?
Our Sonar server version is 7.4 and we use Sonar Maven plugin for scanning our java project.
Note: We are already in process of upgrading our sonar to LTS version.
Also please let us know the inputs for decreasing my scan timing.
We don’t generally have assignees for people/threads/posts, so it’s best to
not invoke (@) people who haven’t already interacted in your thread
provide or link to all the context in each thread
Without your logs, it’s difficult for me to comment on your situation, but what I can say is that 7.4 is too old to expect effective help. If this problem persists once you’ve upgraded please do come back to us. In that case, enabling debug logging during analysis will help identify which steps are taking more time than they should.
Apologies again for unlisting the other thread, let’s continue here now. I’m still reviewing your logs (its huge, just like you mentioned).
Can you please answer these questions for me in the meantime?
Are you using SonarQube 7.4 Community Edition or Developer Edition or Enterprise Edition?
Is there any reason or obstacle that is blocking you from being to upgrade to SonarQube 7.9 LTS? We have a lot of bug fixes and optimization improvements. Can you try a testing your build locally on SonarQube 7.9 and see if that changes your scan time? Please post the logs here or send me a link to it with debug/verbose mode.
When you get a chance, I highly suggest upgrading your maven build tool to 3.6.3. Can you try upgrade maven and reattempt as well?
Will respond again once I review your initial 12hr log.
Thank you for the answers. Yes, please download SonarQube 7.9 locally and run your maven build and sonar:sonar command against it. This way you do not need to upgrade your public SonarQube.
At around 25-Oct-2020 20:07:20, it takes about <5hr to process 25 files. This part alone takes about 4hr 48min. The bulk of that time occurs in the build 25-Oct-2020 22:26:31 [DEBUG] 22:26:31.436 Process file src/com/vitechinc/v3/enrollment/view/components/CoverageInfoComponent.java. Is this file large in terms of lines of code or complexity?
I still want to emphasize that you continue your test on your local environment with SonarQube 7.9, upgrade the plugins within SonarQube 7.9, and upgrade Maven. After you run this test, please link or show the debug logs from your scanner.
Joe
EDIT: For your reading enjoyment, I also highly suggest you read through the Release Upgrade Notes to see any regressions or improvements that you should be aware of.
I dont have enough resources to run the build on my local but i have another sonar server with version Community Edition Version 8.0 (build 29455) there are also we can see that time of build is same version 7.4. Do you think will it makes difference if we run scan with 7.9?
No, I don’t think that will make much of a difference, but let’s try a few things to get an idea of what to do next:
Please check my observations from earlier and see if you agree with them.
FindBugs plugin sensor doesn’t look correct. Please remove or upgrqde it.
In this file src/com/vitechinc/v3/enrollment/view/components/CoverageInfoComponent.java, is this file complex or has lots of lines of code? Can you compare your logs with earlier scans that were quicker and see if something changed in this file?
Try excluding that one CoverageInfoComponent.java file from analysis and running a scan and see how much faster it is, just for testing purposes. Then if it is much faster, then we can focus on why this specific file causes an issue.
Try breaking out your massive, multi-module Maven project into separate project scans.
Please upgrade all your plugins in the Marketplace in Sonarqube 8.0 (or 7.4) and compare.
Finally, please try upgrading to 8.5.1 (our latest version) and see if the results change. There have been many improvements that can help. Here are our Upgrade Notes and Release Notes that you should read before doing the upgrade.
A combination of the above suggestions should help us hone in on the problem. I’ll continue to review the logs. If you choose one thing to do, I suggest doing the upgrade to 8.5.1 (plugins upgraded), running the scan, and get debug logs to show.
I cannot remove findbugs plugin since our DEV team is interested in scanning using findbugs
This file src/com/vitechinc/v3/enrollment/view/components/CoverageInfoComponent.java has only 2300 lines of code, I dont have logs from our CI server to compare but i have verified this file and i dont see lot changes happened to this file recently.
Noted i will exclude this file from scan on the upgraded test region but it takes some time.
I will upgrade our sonar server and will scan the project and let you know the timings but it will take few more days.
I will update you with debug logs as soon as we are done with scanning the project on upgraded version. Thanks once again for your help.
No worries, I have as much time as you need to help you. I hope I can be as quick and speedy as you in replying.
I suggest upgrading the Findbugs plugin or filing an issue here if you continue to get errors after upgrading the plugin.
Thank you for checking that CoverageInfoComponent.java file. Hopefully the upgrade will prove we won’t need to inspect this file any deeper.
Great! I hope the exclusion works to prove that is the culprit (on your old SonarQube) and that your upgrade to the new SonarQube 8.5.1 (with the file included now) fixes it.
Sorry for the delayed response, I have upgraded our SonarQA server to SonarQube 8.5.1 and the time was not reduced even after the upgrade but it got reduced from 12 hours to 5 hours if we run scan on machine which good amount of resources…
Thank you for providing an update. That is good news that it was reduced! Are scans consistently around 5 hours? Did you exclude src/com/vitechinc/v3/enrollment/view/components/CoverageInfoComponent.java file? Try excluding it and running it again.
By the way, you should go ahead and try upgrading to SonarQube 8.6, which was released just last Friday. Please try our latest version and exclude that file again and see what happens. Please post your maven debug logs (-X) also or send it to me privately.
If you have some time, can you try upgrading and testing your project on it? I also suggest removing/disabling FindBugs plugin temporarily so you can see what your analysis looks like “naked”.
Thanks for providing the blog, I have already started my upgrade on our staging region and facing upgrade issues.
currently we are on Community Edition Version 7.4 since we cannot upgrade directly to 8.9, We are proceeding with below approach.
upgrade Community Edition Version 7.4 to Community Edition Version 7.9.5
Once above is completed then upgrade 7.9.5 to 8.9.0
But unfortunately when we are trying to upgrade from 7.4 to 7.9.5 we are not able to bring up the sonar after upgrade. Seeing below issue in the logs.
2021.05.06 23:31:05 WARN web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.IllegalArgumentException: Unable to create shared memory :
at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:103)
at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.<init>(DefaultProcessCommands.java:34)
at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.secondary(DefaultProcessCommands.java:52)
at org.sonar.server.app.WebServer.isOperational(WebServer.java:69)
at org.sonar.server.app.WebServer.getStatus(WebServer.java:61)
at org.sonar.process.ProcessEntryPoint.waitForStatus(ProcessEntryPoint.java:121)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:104)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:81)
at org.sonar.server.app.WebServer.main(WebServer.java:99)
Caused by: java.io.FileNotFoundException: /u01/sonarqube-7.9.5/temp/sharedmemory (Too many open files in system)
at java.base/java.io.RandomAccessFile.open0(Native Method)
at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:100)
... 8 common frames omitted
2021.05.06 23:31:05 INFO web[][o.s.p.ProcessEntryPoint] Hard stopping process
I have also attached full logs for your reference, can you please help me in moving forward with this upgrade.
Note: I have tried all the solutions removing /u01/sonarqube-7.9.5/temp/sharedmemory followed by restart but still see the same issue.
Our DB version is Postgres 9.6.1 and which is compatible with 7.9.5 and 8.9
You will have to work with your network/Linux admin to tune your settings. I think you may have to increase those settings more than what may be suggested or what you have currently. See also similar issue here.
