My project scanning time got increased suddenly with huge time difference

Get help SonarQube
#1

Hi @Joe

As said by you i have created new topic and assume you already have the log so i am not attaching the log here.

History from previous Topic

I have a java projects and it has few subprojects in it and scanning this project used to take nearly 8 hours but from few days it has increased to 12-13 hours. Please let me know how we can find sudden increase in the scan time? How can identify where exactly the scan time got increased?

Our Sonar server version is 7.4 and we use Sonar Maven plugin for scanning our java project.

Note: We are already in process of upgrading our sonar to LTS version.

Also please let us know the inputs for decreasing my scan timing.

Thanks,
Sunil Palugula

1 Like
#2

Hi Sunil,

We don’t generally have assignees for people/threads/posts, so it’s best to

  • not invoke (@) people who haven’t already interacted in your thread
  • provide or link to all the context in each thread

Without your logs, it’s difficult for me to comment on your situation, but what I can say is that 7.4 is too old to expect effective help. If this problem persists once you’ve upgraded please do come back to us. In that case, enabling debug logging during analysis will help identify which steps are taking more time than they should.

 
Ann

#5

Hi @Palugula_Sunil!

Apologies again for unlisting the other thread, let’s continue here now. I’m still reviewing your logs (its huge, just like you mentioned).

Can you please answer these questions for me in the meantime?

  1. Are you using SonarQube 7.4 Community Edition or Developer Edition or Enterprise Edition?
  2. Is there any reason or obstacle that is blocking you from being to upgrade to SonarQube 7.9 LTS? We have a lot of bug fixes and optimization improvements. Can you try a testing your build locally on SonarQube 7.9 and see if that changes your scan time? Please post the logs here or send me a link to it with debug/verbose mode.
  3. When you get a chance, I highly suggest upgrading your maven build tool to 3.6.3. Can you try upgrade maven and reattempt as well?

Will respond again once I review your initial 12hr log.

Joe

#6

I apologize for specially tagging people directly but as instructed by Joe in other unlisted thread i have directly tagged him.

But still noted your instructions below.

  • not invoke (@) people who haven’t already interacted in your thread
  • provide or link to all the context in each thread

Thanks,
Sunil Palugula

2 Likes
#7

Hi @Joe

See below for my answers.

  1. This is my sonar version - Community Edition Version 7.4 (build 18908)
  2. I am planning to upgrade but it will take some time, Can you please confirm ig you want me to install sonar 7.9 in my local and then do the scan?
  3. I will upgrade maven with our current sonar and let you know

Thanks,
Sunil Palugula

#8

Thank you for the answers. Yes, please download SonarQube 7.9 locally and run your maven build and sonar:sonar command against it. This way you do not need to upgrade your public SonarQube.

#9

Hi @Palugula_Sunil,

Small update on my analysis your logs for the ~9hr run. Here are my observations:

  • I noticed that you are using the latest SonarScanner for Maven 3.7.0.1746, which is good.

  • Total mvn build and scan to completion is <12hr (25-Oct-2020 13:33 to 26-Oct-2020 1:22ish).

  • V3 product module alone takes ~9hr!

  • The actual sonar analysis scan starts around 25-Oct-2020 16:17:06, so that’s about 2hr 45min from the start of your mvn command.

  • Possibly need to fix your node installation to prevent this error:

    • build 25-Oct-2020 18:57:00 [ERROR] 18:57:00.408 Failure during analysis, Node.js command to start eslint-bridge server was not built yet.

  • Over 1hr 10min of nothing happening while it runs the FindBugs sensor, very strange, you may want to remove this plugin or upgrade it and check again:

    build	25-Oct-2020 18:57:00	[INFO] 18:57:00.408 Sensor ESLint-based SonarJS [javascript] (done) | time=886ms
build	25-Oct-2020 18:57:00	[INFO] 18:57:00.408 Sensor FindBugs Sensor [findbugs]
build	25-Oct-2020 18:57:21	[INFO] 18:57:21.550 Loading findbugs plugin: /u01/BambooRMAgent/xml-data/build-dir/V3SONAR-V310UC-JOB1/benefits/target/sonar/findbugs/findsecbugs-plugin.jar
build	25-Oct-2020 18:57:21	[INFO] 18:57:21.550 Findbugs output report: /u01/BambooRMAgent/xml-data/build-dir/V3SONAR-V310UC-JOB1/benefits/target/sonar/findbugs-result.xml
error	25-Oct-2020 20:07:17	The following classes needed for analysis were missing:
error	25-Oct-2020 20:07:17	  accept
error	25-Oct-2020 20:07:17	  apply
error	25-Oct-2020 20:07:17	  doInTransaction
error	25-Oct-2020 20:07:17	  execute
error	25-Oct-2020 20:07:17	  test
error	25-Oct-2020 20:07:17	  init
error	25-Oct-2020 20:07:17	  evaluate
error	25-Oct-2020 20:07:17	  compare
error	25-Oct-2020 20:07:17	  isSelected
error	25-Oct-2020 20:07:17	  getId
error	25-Oct-2020 20:07:17	  transform
error	25-Oct-2020 20:07:17	  doInHibernate
error	25-Oct-2020 20:07:17	  applyAsLong
error	25-Oct-2020 20:07:17	  getLabel
error	25-Oct-2020 20:07:17	  get
error	25-Oct-2020 20:07:17	  callback
error	25-Oct-2020 20:07:17	  doInPreparedStatement
error	25-Oct-2020 20:07:17	  run
error	25-Oct-2020 20:07:17	  getDatasheetOwnerEntity
build	25-Oct-2020 20:07:18	[INFO] 20:07:18.001 Sensor FindBugs Sensor [findbugs] (done) | time=4217593ms

  • At around 25-Oct-2020 20:07:20, it takes about <5hr to process 25 files. This part alone takes about 4hr 48min. The bulk of that time occurs in the build 25-Oct-2020 22:26:31 [DEBUG] 22:26:31.436 Process file src/com/vitechinc/v3/enrollment/view/components/CoverageInfoComponent.java. Is this file large in terms of lines of code or complexity?

    build	25-Oct-2020 20:07:19	[DEBUG] 20:07:19.505 Working directory: /u01/BambooRMAgent/xml-data/build-dir/V3SONAR-V310UC-JOB1/benefits
build	25-Oct-2020 20:07:19	[DEBUG] 20:07:19.506 Process file src/com/vitechinc/v3/agreement/model/BillingCycle.java
build	25-Oct-2020 20:07:20	[DEBUG] 20:07:20.315 Process file src/com/vitechinc/v3/enrollment/model/ContinuationCompCalcs.java
build	25-Oct-2020 20:07:20	[DEBUG] 20:07:20.384 Process file src/com/vitechinc/v3/enrollment/model/CoverageContinuationDefinition.java
build	25-Oct-2020 20:07:20	[DEBUG] 20:07:20.424 Process file src/com/vitechinc/v3/enrollment/services/AjaxEnrollmentService.java
build	25-Oct-2020 20:07:20	[DEBUG] 20:07:20.691 Process file src/com/vitechinc/v3/enrollment/services/EnrollmentServiceImpl.java
build	25-Oct-2020 20:07:29	[INFO] 20:07:29.506 4/29 files analyzed
...
build	26-Oct-2020 00:55:10	[INFO] 00:55:10.867 29/29 files analyzed

I still want to emphasize that you continue your test on your local environment with SonarQube 7.9, upgrade the plugins within SonarQube 7.9, and upgrade Maven. After you run this test, please link or show the debug logs from your scanner.

Joe

EDIT: For your reading enjoyment, I also highly suggest you read through the Release Upgrade Notes to see any regressions or improvements that you should be aware of.

#10

Hi @Joe

I dont have enough resources to run the build on my local but i have another sonar server with version Community Edition Version 8.0 (build 29455) there are also we can see that time of build is same version 7.4. Do you think will it makes difference if we run scan with 7.9?

Thanks,
Sunil Palugula

#11

Hey @Palugula_Sunil,

No, I don’t think that will make much of a difference, but let’s try a few things to get an idea of what to do next:

  • Please check my observations from earlier and see if you agree with them.
  • FindBugs plugin sensor doesn’t look correct. Please remove or upgrqde it.
  • In this file src/com/vitechinc/v3/enrollment/view/components/CoverageInfoComponent.java, is this file complex or has lots of lines of code? Can you compare your logs with earlier scans that were quicker and see if something changed in this file?
    • Try excluding that one CoverageInfoComponent.java file from analysis and running a scan and see how much faster it is, just for testing purposes. Then if it is much faster, then we can focus on why this specific file causes an issue.
  • Try breaking out your massive, multi-module Maven project into separate project scans.
  • Please upgrade all your plugins in the Marketplace in Sonarqube 8.0 (or 7.4) and compare.
  • Finally, please try upgrading to 8.5.1 (our latest version) and see if the results change. There have been many improvements that can help. Here are our Upgrade Notes and Release Notes that you should read before doing the upgrade.

A combination of the above suggestions should help us hone in on the problem. I’ll continue to review the logs. If you choose one thing to do, I suggest doing the upgrade to 8.5.1 (plugins upgraded), running the scan, and get debug logs to show.

Joe

#12

Hi @Joe

Sorry for the delayed response.

  • I cannot remove findbugs plugin since our DEV team is interested in scanning using findbugs
  • This file src/com/vitechinc/v3/enrollment/view/components/CoverageInfoComponent.java has only 2300 lines of code, I dont have logs from our CI server to compare but i have verified this file and i dont see lot changes happened to this file recently.
  • Noted i will exclude this file from scan on the upgraded test region but it takes some time.
  • I will upgrade our sonar server and will scan the project and let you know the timings but it will take few more days.

I will update you with debug logs as soon as we are done with scanning the project on upgraded version. Thanks once again for your help.

Thanks,
Sunil Palugula

1 Like
#13

Hi @Palugula_Sunil!

No worries, I have as much time as you need to help you. I hope I can be as quick and speedy as you in replying.

  • I suggest upgrading the Findbugs plugin or filing an issue here if you continue to get errors after upgrading the plugin.
  • Thank you for checking that CoverageInfoComponent.java file. Hopefully the upgrade will prove we won’t need to inspect this file any deeper.
  • Great! I hope the exclusion works to prove that is the culprit (on your old SonarQube) and that your upgrade to the new SonarQube 8.5.1 (with the file included now) fixes it.

I will await the logs (if any!).

Joe

#14

Hi @Joe

Sorry for the delayed response, I have upgraded our SonarQA server to SonarQube 8.5.1 and the time was not reduced even after the upgrade but it got reduced from 12 hours to 5 hours if we run scan on machine which good amount of resources…

Thanks,
Sunil Palugula

#15

Hi @Palugula_Sunil!

Thank you for providing an update. That is good news that it was reduced! Are scans consistently around 5 hours? Did you exclude src/com/vitechinc/v3/enrollment/view/components/CoverageInfoComponent.java file? Try excluding it and running it again.

By the way, you should go ahead and try upgrading to SonarQube 8.6, which was released just last Friday. Please try our latest version and exclude that file again and see what happens. Please post your maven debug logs (-X) also or send it to me privately.

Joe

#16

Hi @Joe

I did not exclude any file, After using the machine with high resource time got reduced to 5 hours consistently.

Sure i will upgrade our sonar to 8.6 and let you know the scan time but need some time for upgrade

Thanks,
Sunil Palugula

1 Like
#17

Hi @Palugula_Sunil,

SonarSource has just released SonarQube 8.9 and a nice blog post on how to upgrade/prepare for it.

If you have some time, can you try upgrading and testing your project on it? I also suggest removing/disabling FindBugs plugin temporarily so you can see what your analysis looks like “naked”.

Joe

#18

Thanks for providing the blog, I have already started my upgrade on our staging region and facing upgrade issues.

currently we are on Community Edition Version 7.4 since we cannot upgrade directly to 8.9, We are proceeding with below approach.

  1. upgrade Community Edition Version 7.4 to Community Edition Version 7.9.5
  2. Once above is completed then upgrade 7.9.5 to 8.9.0

But unfortunately when we are trying to upgrade from 7.4 to 7.9.5 we are not able to bring up the sonar after upgrade. Seeing below issue in the logs.

    2021.05.06 23:31:05 WARN  web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.IllegalArgumentException: Unable to create shared memory : 
	at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:103)
	at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.<init>(DefaultProcessCommands.java:34)
	at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.secondary(DefaultProcessCommands.java:52)
	at org.sonar.server.app.WebServer.isOperational(WebServer.java:69)
	at org.sonar.server.app.WebServer.getStatus(WebServer.java:61)
	at org.sonar.process.ProcessEntryPoint.waitForStatus(ProcessEntryPoint.java:121)
	at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:104)
	at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:81)
	at org.sonar.server.app.WebServer.main(WebServer.java:99)
Caused by: java.io.FileNotFoundException: /u01/sonarqube-7.9.5/temp/sharedmemory (Too many open files in system)
	at java.base/java.io.RandomAccessFile.open0(Native Method)
	at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
	at org.sonar.process.sharedmemoryfile.AllProcessesCommands.<init>(AllProcessesCommands.java:100)
	... 8 common frames omitted
2021.05.06 23:31:05 INFO  web[][o.s.p.ProcessEntryPoint] Hard stopping process

I have also attached full logs for your reference, can you please help me in moving forward with this upgrade.

Note: I have tried all the solutions removing /u01/sonarqube-7.9.5/temp/sharedmemory followed by restart but still see the same issue.

Our DB version is Postgres 9.6.1 and which is compatible with 7.9.5 and 8.9

Thanks,
Sunil Palugulaes.log (3.8 KB) sonar.log (2.6 KB) web.log (6.4 KB)

#19

Hi @Palugula_Sunil ,

Please check these 4 parameters as listed here: Prerequisites and Overview | SonarQube Docs

If you’re running on Linux, you must ensure that:

  • vm.max_map_count is greater or equals to 262144
  • fs.file-max is greater or equals to 65536
  • the user running SonarQube can open at least 65536 file descriptors
  • the user running SonarQube can open at least 4096 threads
#20

Hi @Joe

Before upgrade i have once again verified all the prerequisites from the below link, See below screenshot.

https://docs.sonarqube.org/latest/requirements/requirements/

image

Thanks,
Sunil Palugula

#21

Hi @Palugula_Sunil ,

You will have to work with your network/Linux admin to tune your settings. I think you may have to increase those settings more than what may be suggested or what you have currently. See also similar issue here.

#22

Hi Joe,

I have reach our Linux team and have set the values to below but still seeing same issue. please let me know the next steps for me to proceed further?

sysctl -w vm.max_map_count=786432
sysctl -w fs.file-max=262144
ulimit -n 262144
ulimit -u 12288

Thanks,
Sunil Palugula