SonarQube 6.7.x is taking 31 hrs to scan 658k lines of code using sonar scanner 3.4. However with sonar qube 5.6.x it use to take only 30 mins


(Harshal Bhavsar) #1

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    Sonarqube 7.4, Sonarqube scanner 3.4
  • what are you trying to achieve
    Analyze the code for sonar issues. Jococo Test reports are excluded
  • what have you tried so far to achieve this
    Exported sonarqube 5.6 ruleset and imported to custom profile in 7.4 but no improvement(Even tried SQ 6.6 and 7.5 version but same result)

I tries sonar scanner versions from 2.4 till 3.4 but same results. If I change to previous sonarqube version i.e., 5.6 its faster in 30 mins

Sonar configurations as below

sonar.sources=project/source, project/source
sonar.sourceEncoding=UTF-8
sonar.java.source=8
sonar.java.coveragePlugin=jacoco
sonar.exclusions=**/*.js,**/*.css,**/*.xml,**/*.html,**/*.xsd
sonar.java.binaries=**/classes/**

Please suggest if any settings to be modified in SonarQube to improve performance of scan. We are upgrading to sonarqube 6.6 or above version as it has better issue tracking system (graph like when it was introduced under activity tab of sonarqube for tracking purpose)

After enabling logs I see below exceptions some times but I guess its normal

org.sonar.java.se.ExplodedGraphWalker$MaximumStepsReachedException: reached limit of 16000 steps for method updateSpecialStatusHistory#212 in class MemberActivityManager at org.sonar.java.se.ExplodedGraphWalker.throwMaxSteps(ExplodedGraphWalker.java:281)

Thanks


(Adam Gabryś) #3

Please add information about installed plugins on both servers (7.4 and 5.6).


(Harshal Bhavsar) #4

Hi Adam,

There are no plugins added to these servers and they are out of box


(G Ann Campbell) #5

Hi,

You mentioned that you’re using a custom profile. What happens if you use the Sonar way profile? And just to verify, you don’t have the FindBugs plugin loaded, do you?

 
Ann


(Harshal Bhavsar) #6

Hi Ann, There is no FindBugs plugin loaded to sonarqube server.

Thanks
Harshal


(G Ann Campbell) #7

HI Harshal,

Thanks for the verification. I notice this in your initial post:

sonar.sources=project/source, project/source

Why do you have the same thing listed twice?

It may also help to add a duplications exclusion (project Administration > General Settings > Analysis Scope).

Also, it would help if you could turn on debug logging in your analysis so we could see which steps were taking the most time.

 
Ann


(Harshal Bhavsar) #8

Hi Ann,

Thanks for quick response! Due to confidentiality I named it like this. There are two different code sources.
In dubug logs its not clear which step is taking time. All I am interested is only Java file scanning and exclude other files which I did through sonar properties using exclusion

But as per my analysis, its duplicate detection of blocks or code is taking a very long time instead of applying rules during scan

Duplication in general settings flag is false by default.

I did not configure anything for Duplication exclusion under analysis scope. If I set **/*.java then I assume no duplication issues will be calculated which user doesnt want

Thanks
Harshal


(G Ann Campbell) #9

Hi Harshal,

It’s not quite clear to me whether you’ve worked through this or still have questions, so to be on the safe side…

Yes, setting your duplication exclusion to **/*.java should exclude those files from duplication detection. Given your other exclusions, Java files should be the only ones that are relevant here, but it might be simpler to set the duplication exclusion to **/*.*.

And to be clear, the duplication detection’s primary task is just finding duplications so they can be marked in the interface and metrics can be generated on them. Whether or not issues are raised is almost secondary (and dependent on your quality profile).

Hopefully turning off duplication detection via exclusions has solved your problem. If not, feel free to come back with more details.

 
Ann


(Harshal Bhavsar) #10

Hi Ann,

Thanks for quick response :slight_smile: Appreciate your continuous help. Unfortunately issue still stands unresolved :frowning: even if I add duplication exclusion **/*.java. It still takes same time. I may try **/*.* also but as source points to only java files it wont make a difference in my opinion.

Please find some debug logs attachedconsoleText1.txt (1.8 MB)
before I added above exclusion. Please let me know if you feel something suspicious

Thanks
Harshal


(G Ann Campbell) #11

Hi Harshal,

This isn’t the full debug log. Did the log roll, by chance? Are there additional files numbered 2, 3, …?

Also, what’s your version of SonarJava (found in Administration > Marketplace)? And if it’s not 5.10.1, could you upgrade and provide fresh logs? I know this is a commitment on your part, since analysis takes so long, but it would be best to know up front whether the problem has already been solved in the latest version.

 
Ann


(Harshal Bhavsar) #12

Hi Ann,

Entire log file is too huge to upload hence I uploaded only one part of it. Let me know if you need entire log file. Also I have been testing it with multiple sonarqube versions like 5.6.1, 5.6.7, 6.6.7, 7.5 etc
I found 5.6.1 a bit quicker in analysis

About sonarJava I dont find marketplace option under administration for 5.6.1 hence I checked 6.6.7 version.It was 4.15 version. I upgraded it just now to 5.9.2 I will start scan again but if takes time usually 30+ hrs
then I will provide some part of the logs shortly

Thanks for your efforts in helping me. I am sure now it will be fixed sooner or later with your help :slight_smile:


(G Ann Campbell) #13

Hi,

I’m going to set this aside until we have the results of your new analysis.

 
Ann


(Harshal Bhavsar) #14

Hi Ann,

Analysis is taking similar time. Its still running since last 2 to 3 hrs. Please find attached logsscanlogs.txt (1019.8 KB)

Thanks
Harshal


(G Ann Campbell) #15

Hi Harshal,

Please post the whole thing once analysis is done.

 
Ann


(Harshal Bhavsar) #16

sure

Thnx
Harshal


(Harshal Bhavsar) #17

Hi Ann,

Estimated time is close to 30 to 40 hrs. Do we have anything else to check meanwhile?

Thanks
Harshal


(Harshal Bhavsar) #18

Hi Ann,

Good Morning! Please find attached log of complete scan for analysis. It took 55 hrs this timeconsoleText3.txt (3.8 MB)
consoleText2.txt (3.8 MB)


(Harshal Bhavsar) #19

Please find attached 1st part consoleText1.txt (3.8 MB)


(Harshal Bhavsar) #21

Hi Ann,

Did you get a chance to look at logs?

Thanks
Harshal


(Michael Gumowski) #22

Hello @harshal_bhavsar,

I had a look at the logs and it seems that analysis of individual java files is taking way too much time that expected. From the total analysis time, JavaSquidSensor (from SonarJava plugin) is taking 3311min 15s (99%). This is huge and unexpected. It’s not obvious to me why it would take so much time. Still, there is several things that we can try. For all the following, please use the latest version of SonarJava (5.10.1).

  1. Did you try to run the analysis with the maven scanner? (your manual configuration might be incomplete) Longer? Shorter?

  2. Did you try allowing more resources to the analysis? For such large projects, we do expect to have the adequate resources. What is your configuration?

  3. Without changing anything to your configuration. From what I can see, scanning only the 1st files is already taking a huge amount of time. Let’s be radical and only keep about 50 first files for the analysis. Please perform an analysis with this very reduced set of file, and provide the logs. If you can see in the logs that the analysis still hangs on the same files, then use the very same files for the next steps as well.

  4. Can you try to disable all the SonarJava rules ? We will try here to only compute the metrics and check that parsing is not taking already too much time.

  5. There is 17 SonarJava rules which are using what we call the “Symbolic Execution Engine”. This engine might be extremely demanding in term of resources. While these rules are normally sharing resources, it can still be very costly. Could you try to disable the rules which rely on this engine?
    Here is the list of rule to disable [from SonarWay default profile]:

    • squid:S2589
    • squid:S2583
    • squid:S3518
    • squid:S3516
    • squid:S2222
    • squid:S3824
    • squid:S3065
    • squid:S2637
    • squid:S2189
    • squid:S2259
    • squid:S2689
    • squid:S3655
    • squid:S4449
    • squid:S4165
    • squid:S3959
    • squid:S3958
    • squid:S2095

    If you are not using the default profile, check that this rule is not enabled as well:

    • squid:S3546 (it’s a template rule, so you would need to also disable any rule extending it)
  6. Finally, when using sonar-scanner to perform the analysis (not maven), can you try to NOT provide anything for the following properties (it may be required by the analysis, if so, provide a path to an empty folder, or with a single class file):

    • sonar.java.binaries
    • sonar.java.libraries
    • sonar.java.test.binaries
    • sonar.java.test.libraries

    Note that not providing these properties will have a significant impact on the analysis quality. Only the most basic rules are going to be triggered, and you are losing a huge majority of high-value rules. But without these properties, the semantic model won’t be built for each file. Which could take some time depending the complexity of your project.

Please let me know the results of the investigation.

Michael