Incremental Analysis

scanner
sonarqube

(Christoph Forster) #1

I was wondering if SonarQube Scanner provides a “incremental” Analysis.

My project is growing permanently and we currently have nearly 90 Modules with over 160k LOC (in one project). The analysis takes over 25 Minutes (using SonarQube Scanner).

Regarding the analysis of Git Feature Branches we are thinking about the Developer Edition but such a long running Analysis is a NoGo for our CI Jobs, especially when only a few files have been modified.

I already tried to keep the “.scannerwork” directory with no time improvement and googled for an incremental configuration but only found out that sonar.analysis.mode=incremental is deprecated for a long time now.

Any ideas?

Thanks


Why feed "sonar.java.libraries"?
(Colin Mueller) #2

We do not currently support incremental analysis as it can result in less accurate scanning results. This has come up quite regularly, especially in the context of branches and pull requests, and internally there is a lot of discussion around this, but I don’t think we have anything to share at this time.

That said, 25 minutes on 160k LOC seems a little long (although that’s not exactly scientific).

  1. Which side of the analysis is taking so long – the actual scanning or the background task on the SonarQube Server’s side? (If the former, how long is the background task taking to process?)
  2. What version of SonarQube and relevant plugins are you running?
  3. Are you using any plugins that run other analysers (PMD, Checkstyle, Findbugs, etc.)? These are often the cause of lengthened analysis tasks and is rather out of our control.

If you have logs to share from analysis, preferably with debug mode enabled, we might be able to do a sanity check. :slight_smile:

Colin


(Christoph Forster) #3

Hi Colin,

thanks for the fast reply.
The scanning takes the 25 Minutes.
Background Task on SonarQube is finished in less than 2 minutes, so this is OK.
We’re using SonarQube 7.3 and FindBugs Plugin 3.8.0.

I will check if we’re currently using Rules from the FindBugs Plugin so that I can remove it and retest if it’s faster then.

How can I provide the logs to you if the deactivation does not have the desired effect?

Thanks,
Christoph


(Christoph Forster) #4

OK since our Profile uses most of it’s rules from Findbugs I can not deactivate it that easy …


(Colin Mueller) #5

Christoph,

You should be able to spot in your logs how long FindBugs is taking. If indeed Findbugs is the cause of long-running analysis, that is something to take up with the plugin developer (if not Findbugs itself, as I think the plugin mostly just wraps Findbugs).

That said, SonarJava is excellent (527 rules and counting), in my opinion.

Perhaps create a second project and assign it the default Sonar Way quality profile and scan your code using that new project key? It would give you an idea of what performance in SonarQube is expected to be like by our standards. You might even find it suitable to use instead of your existing QP. :slight_smile:

Edit: Had to strikethrough, see below

In the event you still wish to share logs, we accept them zipped up and attached to this topic.

Colin


(Christoph Forster) #8

OK thanks.
By the way: We are recently working on a QP Switch but it takes some time :slight_smile:


(Colin Mueller) #10

Christoph.

An update. My colleague @ganncamp informed me that when the Findbugs plugin is installed on an instance, it will always run and evaluate all rules, before parsing out the ones enabled in your Quality Profile, so testing performance might not be so simple. :frowning: It looks like an existing issue exists on the sonar-findbugs repo without much traction: https://github.com/spotbugs/sonar-findbugs/issues/72

I also forgot to mention that SonarJava does support the import of Findbugs reports (see: MMF-1300) so perhaps it would be possible to stop using the Findbugs plugin, pull out the Findbugs rules from your QP, and run/configure Findbugs seperately and use our native features to import the reports. Just a thought!

Cheers,

Colin


(Christoph Forster) #11

Hi Colin,

as feedback: I’ve summed up the different “Sensor XXX [xxx] (done) | time=…ms” Log Outputs.

Even if the calculated Total (~36 Minutes) does not equal to the Runtime (~23 Minutes) I have a good Overview what takes the most time:

Java Main Files AST scan (done): 2.8 Minutes
Java Test Files AST scan (done): 0.9 Minutes
JavaClasspath initialization (done): 5.5 Minutes
JavaTestClasspath initialization (done): 5.5 Minutes
Sensor FindBugs Sensor [findbugs] (done): 6 Minutes
Sensor JavaSquidSensor [java] (done): 15 Minutes

br,
Christoph


(Colin Mueller) #14

Thanks Christoph. Perhaps I should eat my words now since SonarJava appears to be the hotspot here.

Let’s rewind a little bit.

What version of SonarJava are you using?

Debug-level logs (-X) might also reveal more information about what’s taking up the analysis time within SonarJava.

Colin


(Christoph Forster) #16

I’m using SonarJava 5.9.1 (build 16423)

Activating the Debug Log I found out that there are tons of jar files analyzied for the classpath.

I managed it to decrease the time dramatically (7 Minutes!) by removing the libraries configuration:

sonar.java.libraries=../**/*.jar
sonar.java.test.libraries=../**/*.jar

Maybe I need to think about a more precise inclusion of dependencies.

Anyway: Thanks for the help - with 7 Minutes (or less) the Incremental Analysis is not that important anymore.


(Colin Mueller) #17

:tada: That’s great news (and this surely means you will buy our Developer Edition, right? No pressure. :stuck_out_tongue: )

Cheers,

Colin