SonarQube Java scanner slow on 675k loc

SonarQube Server / Community Build
,
1

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)

    • SonarQube Developer 8.4.1.35646
    • Java Code Quality and Security 6.5.1 (build 22586)
    • Vulnerability Rules for Java 8.4 (build 2759)

  • what are you trying to achieve
    Analyze legacy Java/C++ project with SonarQube. Here restricted to Java part only.
    Using CLI scanner to integrate in existing, working build.

  • what have you tried so far to achieve this

      - task: sonarsource.sonarqube.15B84CA1-B62F-4A2A-A403-89B77A063157.SonarQubePrepare@4
        displayName: 'Prepare analysis on SonarQube'
        continueOnError: true
        condition: and(succeeded(), eq(variables['AnalyseJava'], true))
        inputs:
          SonarQube: Sonarqube
          scannerMode: CLI
          extraProperties: |
            sonar.projectKey=key
            sonar.projectVersion='$(Build.BuildNumber)'
            sonar.sources=XXX1,XXX2
            sonar.tests=XXXTest
            sonar.projectBaseDir=$(SrcDir)
            sonar.java.binaries=$(JavaClassesPath)
Project Lines of Code
XXX1 652.620
XXX2 22191
Task time 8.2 time 8.4.1
Build jar archives from source 1m 39s 1m 20s
Complete the SonarQube analysis 10m 49s 11m 18s
Publish Quality Gate Result 10m 22s 5m 21s
2

Hi,

Would you mind including here some debug analysis logs so we can all see where the time is going?

Edit To make sure, is “Complete the SonarQube analysis” happening analysis-side or server-side? If the latter, then this is likely relevant:

 
Ann

4

Hello @ganncamp

These are Sonarsource tasks for Azure DevOps Server from marketplace:

      - task: sonarsource.sonarqube.6D01813A-9589-4B15-8491-8164AEB38055.SonarQubeAnalyze@4
        displayName: 'Complete the SonarQube analysis'
        condition: and(succeededOrFailed(), eq(variables['AnalyseJava'], true))

      - task: sonarsource.sonarqube.291ed61f-1ee4-45d3-b1b0-bf822d9095ef.SonarQubePublish@4
        displayName: 'Publish Quality Gate Result'
        condition: and(succeededOrFailed(), eq(variables['AnalyseJava'], true))

For the database maintanance I will ask our IT department, as I personally don’t have access to the MS SQL server, only to the Sonarqube server. Thus it would be welcome if in the administrative pages there would be an option to execute Refreshing your database's statistics and rebuilding. As 8.4.1 was installed in an empty directory, ElasticSearch indexes should have been rebuild automatically.

Which logs would be needed? As long as there is no source revealed I can attach them here, otherwise I can send them to you confidentially by mail.

Thanks,
Volker

6

Hi Volker,

That depends on where the slowness is, which is not clear to me from your initial report, “Complete the SonarQube analysis”. Does this mean after the analysis report is submitted to the server? If so, then no logs (currently) needed and I’ll be very interested in the stats after DB maintenance.

 
:smiley:
Ann

7

Hi Ann,

I digged a bit in the logs of<our task Run Code Analysis 4.11.0. For me it looks like the sensor for the Java files on the build server is cosuming the time. An excerpt of the log:

2020-07-25T12:04:32.8318464Z [command] ... run the `sonar-scanner.bat`
2020-07-25T12:04:33.0656676Z INFO: SonarScanner 4.4.0.2170
2020-07-25T12:04:33.0657136Z INFO: Java 1.8.0_152 Oracle Corporation (64-bit)
2020-07-25T12:04:33.8922709Z WARN: SonarScanner will require Java 11 to run starting in SonarQube 8.x
2020-07-25T12:04:38.9882986Z INFO: ------------- Run sensors on module <projectkey>
2020-07-25T12:04:51.8171829Z INFO: 34/3986 files analyzed, current file: ...
2020-07-25T12:13:54.4429174Z INFO: 3986/3986 source files have been analyzed
2020-07-25T12:14:17.8820959Z INFO: ------------- Run sensors on project
2020-07-25T12:15:46.9035403Z INFO: Analysis report uploaded in 449ms
2020-07-25T12:15:46.9062869Z INFO: ANALYSIS SUCCESSFUL, you can browse http://...
2020-07-25T12:15:46.9063412Z INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
2020-07-25T12:15:51.0019326Z INFO: Analysis total time: 11:16.660 s
2020-07-25T12:15:51.0102186Z INFO: ------------------------------------------------------------------------
2020-07-25T12:15:51.0103720Z INFO: EXECUTION SUCCESS
2020-07-25T12:15:51.0103856Z INFO: ------------------------------------------------------------------------
2020-07-25T12:15:51.0134289Z INFO: Total time: 11:18.003s
2020-07-25T12:15:51.3537735Z INFO: Final Memory: 81M/1084M
2020-07-25T12:15:51.3538188Z INFO: ------------------------------------------------------------------------
2020-07-25T12:15:51.5161263Z ##[section]Finishing: Complete the SonarQube analysis
8

Hi Volker,

Just over 9 minutes to analyze almost 4k files doesn’t seem unreasonable to me…?

By my math, that’s ~7.2 files per second.

 
Ann

9

Hi Ann,

I have to argue with my co-workers in total pipeline runtime: build jar files from 3986 java files need 1m20s plus additional 16m40s for SonarQube. And here I get into problems explaining that it is worth increasing pipeline runtime by 12.5 times build-time. As unit testing in the pipeline will also need time, the fraction of SonarQube in the pipeline will reduce – but remain determining.

For the C++ job I can argue that as soon as it is in cache SonarQube overhead is about build time of the solution (2.9k files, sonar.cfamily.threads=2). And with unit tests it will drop further.

Volker