Sonar Scanner - Slow scan of java project

Dennis_DECA · January 25, 2023, 11:43am

SonarQube: Community edition 8.9.3 LTS
Sonar Scanner: GitHub Action sonarqube-scan-action@master (v1.2.0)
Java 8 project: About 3 million lines
Custom plugins: Yes, we created 2 custom plugins/rules

Problem:
Sonar Scan takes about 80 minutes on a new fast machine, usually 180 minutes on an older machine.
SonarQube analysis takes about 10min which is fine.

How can we speed up the sonar scan?
What settings should we tweak?
Will upgrading to a developer or enterprise edition with PR scanning speed up this process considerably?

Log:

2023-01-11T18:07:42.2177757Z ##[command]/usr/bin/docker run --name b0279f2016ed73f7c458089453f192bc79ece_3a25c5 --label 0b0279 --workdir /github/workspace --rm -e "JAVA_HOME" -e "JAVA_HOME_8_X64" -e "SONAR_TOKEN" -e "SONAR_HOST_URL" -e "INPUT_PROJECTBASEDIR" -e "INPUT_ARGS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true --entrypoint "/entrypoint.sh" -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/data/actions-runner/_work/_temp/_github_home":"/github/home" -v "/data/actions-runner/_work/_temp/_github_workflow":"/github/workflow" -v "/data/actions-runner/_work/_temp/_runner_file_commands":"/github/file_commands" -v "/data/actions-runner/_work/miles-core/miles-core":"/github/workspace" 0b0279:f2016ed73f7c458089453f192bc79ece
2023-01-11T18:07:42.9098751Z INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
2023-01-11T18:07:42.9103208Z INFO: Project root configuration file: /github/workspace/core/sonar-project.properties
2023-01-11T18:07:42.9423794Z INFO: SonarScanner 4.8.0.2856
2023-01-11T18:07:42.9424474Z INFO: Java 11.0.17 Alpine (64-bit)
2023-01-11T18:07:42.9425021Z INFO: Linux 4.18.0-348.el8.x86_64 amd64
2023-01-11T18:07:43.1889907Z INFO: User cache: /opt/sonar-scanner/.sonar/cache
2023-01-11T18:07:43.8734805Z INFO: Analyzing on SonarQube server 8.9.3
2023-01-11T18:07:43.8737997Z INFO: Default locale: "en_US", source code encoding: "UTF-8"
2023-01-11T18:07:44.1944026Z INFO: Load global settings
2023-01-11T18:07:44.5357267Z INFO: Load global settings (done) | time=343ms
2023-01-11T18:07:44.5383391Z INFO: Server id: XXXXXXX
2023-01-11T18:07:44.5407421Z INFO: User cache: /opt/sonar-scanner/.sonar/cache
2023-01-11T18:07:44.5428500Z INFO: Load/download plugins
2023-01-11T18:07:44.5428874Z INFO: Load plugins index
2023-01-11T18:07:44.8905473Z INFO: Load plugins index (done) | time=348ms
2023-01-11T18:07:51.7212874Z INFO: Load/download plugins (done) | time=7178ms
2023-01-11T18:07:52.8646610Z INFO: Process project properties
2023-01-11T18:07:52.8723264Z INFO: Process project properties (done) | time=7ms
2023-01-11T18:07:52.8723712Z INFO: Execute project builders
2023-01-11T18:07:52.8732561Z INFO: Execute project builders (done) | time=1ms
2023-01-11T18:07:52.8766247Z INFO: Project key: miles:core
2023-01-11T18:07:52.8766754Z INFO: Base dir: /github/workspace/core
2023-01-11T18:07:52.8767146Z INFO: Working dir: /github/workspace/core/MyBuild/sonar
2023-01-11T18:07:52.9653136Z INFO: Load project settings for component key: 'miles:core'
2023-01-11T18:07:53.2123251Z INFO: Auto-configuring with CI 'Github Actions'
2023-01-11T18:07:53.2131001Z INFO: Load quality profiles
2023-01-11T18:07:53.6608752Z INFO: Load quality profiles (done) | time=448ms
2023-01-11T18:07:53.6637588Z INFO: Auto-configuring with CI 'Github Actions'
2023-01-11T18:07:53.6669518Z INFO: Load active rules
2023-01-11T18:07:57.9935767Z INFO: Load active rules (done) | time=4326ms
2023-01-11T18:07:58.0729224Z INFO: Indexing files...
2023-01-11T18:07:58.0732494Z INFO: Project configuration:
2023-01-11T18:07:58.0737642Z INFO:   Excluded sources: **/*Stub.java, **/*.jar, **/*Test.java, **/*XXXX.xml
2023-01-11T18:08:01.3376604Z INFO: 20283 files indexed
2023-01-11T18:08:01.3380571Z INFO: 131 files ignored because of inclusion/exclusion patterns
2023-01-11T18:08:01.3386835Z INFO: Quality profile for java: Sofico Web
2023-01-11T18:08:01.3387153Z INFO: Quality profile for xml: Sonar way
2023-01-11T18:08:01.3387802Z INFO: ------------- Run sensors on module MilesCore
2023-01-11T18:08:01.5011639Z INFO: Load metrics repository
2023-01-11T18:08:01.7238154Z INFO: Load metrics repository (done) | time=223ms
2023-01-11T18:08:02.3577689Z INFO: Sensor JavaSquidSensor [java]
2023-01-11T18:08:02.5372009Z INFO: Configured Java source version (sonar.java.source): 8
2023-01-11T18:08:02.5447737Z INFO: JavaClasspath initialization
2023-01-11T18:08:02.5494684Z INFO: JavaClasspath initialization (done) | time=5ms
2023-01-11T18:08:02.5495314Z INFO: JavaTestClasspath initialization
2023-01-11T18:08:02.5511930Z INFO: JavaTestClasspath initialization (done) | time=1ms
2023-01-11T18:08:02.5570488Z INFO: Java Main Files AST scan
2023-01-11T18:08:02.5735578Z INFO: 19655 source files to be analyzed
2023-01-11T18:08:02.5914809Z INFO: Load project repositories
2023-01-11T18:08:02.7966411Z INFO: Load project repositories (done) | time=204ms
*****
2023-01-11T18:43:40.4090320Z INFO: 19655/19655 source files have been analyzed
2023-01-11T18:43:40.4092260Z INFO: Slowest analyzed files:
2023-01-11T18:43:40.4092947Z     _java/source/XXXXXXX.java (14856ms, 30349B)
2023-01-11T18:43:40.4098314Z     _java/source/XXXXXXX.java (11742ms, 931157B)
2023-01-11T18:43:40.4098797Z     _java/source/XXXXXXX.java (11168ms, 1059005B)
2023-01-11T18:43:40.4099221Z     _java/source/XXXXXXX.java (10774ms, 870479B)
2023-01-11T18:43:40.4100166Z     _java/source/XXXXXXX.java (8691ms, 1004916B)
2023-01-11T18:43:40.4100576Z     _java/source/XXXXXXX.java (8408ms, 676694B)
2023-01-11T18:43:40.4101021Z     _java/source/XXXXXXX.java (7931ms, 604244B)
2023-01-11T18:43:40.4101479Z     _java/source/XXXXXXX.java (7679ms, 421705B)
2023-01-11T18:43:40.4101892Z     _java/source/XXXXXXX.java (7450ms, 24501B)
2023-01-11T18:43:40.4106692Z     _java/source/XXXXXXX.java (7080ms, 447210B)
2023-01-11T18:43:40.4107331Z WARN: Unresolved imports/types have been detected during analysis. Enable DEBUG mode to see them.
2023-01-11T18:43:40.4152328Z INFO: Java Main Files AST scan (done) | time=2137859ms
2023-01-11T18:43:40.4152650Z INFO: Java Test Files AST scan
2023-01-11T18:43:40.4206428Z INFO: 488 source files to be analyzed
2023-01-11T18:43:41.6415230Z WARN: Invalid character encountered in file /github/workspace/core/XXXXXXX.java at line 1292 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
2023-01-11T18:43:42.1986279Z WARN: Invalid character encountered in file /github/workspace/core/XXXXXXX.java at line 528 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
2023-01-11T18:43:50.4217133Z INFO: 337/488 files analyzed, current file: _java/test/be/sofico/basecamp/ctrl/fleetvehiclesale/policy/SofBillVehicleSaleUnitTest.java
2023-01-11T18:43:55.1026371Z INFO: 488/488 source files have been analyzed
2023-01-11T18:43:55.1027169Z WARN: Unresolved imports/types have been detected during analysis. Enable DEBUG mode to see them.
2023-01-11T18:43:55.1032145Z INFO: Java Test Files AST scan (done) | time=14687ms
2023-01-11T18:43:55.1032470Z INFO: Java Generated Files AST scan
2023-01-11T18:43:55.1038434Z INFO: 0 source files to be analyzed
2023-01-11T18:43:55.1038726Z INFO: 0/0 source files have been analyzed
2023-01-11T18:43:55.1039010Z INFO: Java Generated Files AST scan (done) | time=0ms
2023-01-11T18:43:55.1040164Z INFO: Sensor JavaSquidSensor [java] (done) | time=2152747ms
2023-01-11T18:43:55.1040479Z INFO: Sensor CSS Rules [cssfamily]
2023-01-11T18:43:55.1210063Z INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
2023-01-11T18:43:55.1210724Z INFO: Sensor CSS Rules [cssfamily] (done) | time=17ms
2023-01-11T18:43:55.1211207Z INFO: Sensor PmdSensor [pmd]
2023-01-11T18:43:55.1229505Z INFO: Execute PMD 6.30.0
2023-01-11T18:43:55.1331015Z INFO: Java version: 1.8
****
2023-01-11T18:44:57.0562551Z INFO: PMD configuration: /github/workspace/core/MyBuild/sonar/pmd-unit-tests.xml
2023-01-11T18:44:57.0715872Z INFO: Execute PMD 6.30.0 (done) | time=61949ms
2023-01-11T18:44:57.4255038Z INFO: Sensor PmdSensor [pmd] (done) | time=62305ms
2023-01-11T18:44:57.4255426Z INFO: Sensor JaCoCo XML Report Importer [jacoco]
2023-01-11T18:44:57.4716564Z INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
2023-01-11T18:44:57.4724713Z INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
2023-01-11T18:44:57.4725133Z INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=47ms
2023-01-11T18:44:57.4725483Z INFO: Sensor C# Project Type Information [csharp]
2023-01-11T18:44:57.4787116Z INFO: Sensor C# Project Type Information [csharp] (done) | time=6ms
2023-01-11T18:44:57.4787453Z INFO: Sensor C# Properties [csharp]
2023-01-11T18:44:57.4796618Z INFO: Sensor C# Properties [csharp] (done) | time=1ms
2023-01-11T18:44:57.4797021Z INFO: Sensor FindBugs Sensor [findbugs]
2023-01-11T18:45:08.3428968Z INFO: Loading findbugs plugin: /github/workspace/core/MyBuild/sonar/findbugs/sb-contrib.jar
2023-01-11T18:45:08.4753878Z INFO: Loading findbugs plugin: /github/workspace/core/MyBuild/sonar/findbugs/findsecbugs-plugin.jar
2023-01-11T18:45:08.5397493Z INFO: Findbugs output report: /github/workspace/core/MyBuild/sonar/findbugs-result.xml
2023-01-11T19:13:32.5150752Z The following errors occurred during analysis:
2023-01-11T19:13:32.5152034Z   Exception analyzing
****
2023-01-11T19:13:47.5020036Z INFO: Sensor FindBugs Sensor [findbugs] (done) | time=1730022ms
2023-01-11T19:13:47.5020711Z INFO: Sensor SurefireSensor [java]
2023-01-11T19:13:47.5045493Z INFO: parsing [/github/workspace/core/target/surefire-reports]
2023-01-11T19:13:47.5055584Z INFO: Sensor SurefireSensor [java] (done) | time=4ms
2023-01-11T19:13:47.5056172Z INFO: Sensor Removed properties sensor [java]
2023-01-11T19:13:47.5060909Z WARN: Property 'sonar.jacoco.reportPath' is no longer supported. Use JaCoCo's xml report and sonar-jacoco plugin.
2023-01-11T19:13:47.5061545Z WARN: Property 'sonar.jacoco.itReportPath' is no longer supported. Use JaCoCo's xml report and sonar-jacoco plugin.
2023-01-11T19:13:47.5061967Z INFO: Sensor Removed properties sensor [java] (done) | time=0ms
2023-01-11T19:13:47.5062257Z INFO: Sensor JavaXmlSensor [java]
2023-01-11T19:13:47.6097258Z INFO: 63 source files to be analyzed
2023-01-11T19:13:48.4403819Z INFO: 63/63 source files have been analyzed
2023-01-11T19:13:48.4404489Z INFO: Sensor JavaXmlSensor [java] (done) | time=934ms
2023-01-11T19:13:48.4405003Z INFO: Sensor HTML [web]
2023-01-11T19:13:48.4809715Z INFO: Sensor HTML [web] (done) | time=41ms
2023-01-11T19:13:48.4810213Z INFO: Sensor XML Sensor [xml]
2023-01-11T19:13:48.4876153Z INFO: 3 source files to be analyzed
2023-01-11T19:13:49.0181071Z INFO: 3/3 source files have been analyzed
2023-01-11T19:13:49.0181622Z INFO: Sensor XML Sensor [xml] (done) | time=537ms
2023-01-11T19:13:49.0182131Z INFO: Sensor CheckstyleSensor [checkstyle]
2023-01-11T19:13:49.0393791Z INFO: Checkstyle output report: /github/workspace/core/MyBuild/sonar/checkstyle-result.xml
2023-01-11T19:13:49.0737795Z INFO: Checkstyle configuration: /github/workspace/core/MyBuild/sonar/checkstyle.xml
2023-01-11T19:13:49.1457372Z INFO: Checkstyle charset: UTF-8
2023-01-11T19:16:55.9617849Z INFO: Sensor CheckstyleSensor [checkstyle] (done) | time=186943ms
2023-01-11T19:16:55.9618605Z INFO: Sensor VB.NET Project Type Information [vbnet]
2023-01-11T19:16:55.9689657Z INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=7ms
2023-01-11T19:16:55.9691158Z INFO: Sensor VB.NET Properties [vbnet]
2023-01-11T19:16:55.9702452Z INFO: Sensor VB.NET Properties [vbnet] (done) | time=1ms
2023-01-11T19:16:55.9762105Z INFO: ------------- Run sensors on project
2023-01-11T19:16:56.0120594Z INFO: Sensor Dependency-Check [dependencycheck]
2023-01-11T19:16:56.0121258Z INFO: Process Dependency-Check report
2023-01-11T19:16:56.0158589Z INFO: Using JSON-Reportparser
2023-01-11T19:16:56.0182470Z INFO: Dependency-Check JSON report does not exists. Please check property sonar.dependencyCheck.jsonReportPath:/github/workspace/scripts/build/dependency-check/dependency-check-report.json
2023-01-11T19:16:56.0183578Z INFO: JSON-Analysis skipped/aborted due to missing report file
2023-01-11T19:16:56.0184165Z INFO: Using XML-Reportparser
2023-01-11T19:16:56.0189577Z INFO: Dependency-Check XML report does not exists. Please check property sonar.dependencyCheck.xmlReportPath:/github/workspace/core/${WORKSPACE}/dependency-check-report.xml
2023-01-11T19:16:56.0190190Z INFO: XML-Analysis skipped/aborted due to missing report file
2023-01-11T19:16:56.0191723Z INFO: Dependency-Check HTML report does not exists. Please check property sonar.dependencyCheck.htmlReportPath:/github/workspace/scripts/build/dependency-check/dependency-check-report.html
2023-01-11T19:16:56.0192298Z INFO: HTML-Dependency-Check report does not exist.
2023-01-11T19:16:56.0192722Z INFO: Process Dependency-Check report (done) | time=7ms
2023-01-11T19:16:56.0193137Z INFO: Sensor Dependency-Check [dependencycheck] (done) | time=7ms
2023-01-11T19:16:56.0193441Z INFO: Sensor Zero Coverage Sensor
2023-01-11T19:16:59.1697256Z INFO: Sensor Zero Coverage Sensor (done) | time=3150ms
2023-01-11T19:16:59.1697940Z INFO: Sensor Java CPD Block Indexer
2023-01-11T19:17:26.9061192Z INFO: Sensor Java CPD Block Indexer (done) | time=27736ms
2023-01-11T19:17:26.9075252Z INFO: SCM Publisher is disabled
2023-01-11T19:17:28.8976293Z INFO: CPD Executor 3464 files had no CPD blocks
2023-01-11T19:17:28.8976855Z INFO: CPD Executor Calculating CPD for 16191 files
******
2023-01-11T19:17:58.7719376Z INFO: CPD Executor CPD calculation finished (done) | time=29873ms
2023-01-11T19:18:05.4939139Z INFO: Analysis report generated in 5385ms, dir size=377 MB
2023-01-11T19:23:39.8288079Z INFO: Analysis report compressed in 334332ms, zip size=129 MB
2023-01-11T19:23:44.9919600Z INFO: Analysis report uploaded in 5163ms
2023-01-11T19:23:44.9955679Z INFO: ANALYSIS SUCCESSFUL, you can browse XXXXXXX
2023-01-11T19:23:44.9957161Z INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
2023-01-11T19:23:44.9958133Z INFO: More about the report processing at XXXXXXX
2023-01-11T19:23:48.9861030Z INFO: Analysis total time: 1:15:56.864 s
2023-01-11T19:23:48.9972589Z INFO: ------------------------------------------------------------------------
2023-01-11T19:23:48.9973183Z INFO: EXECUTION SUCCESS
2023-01-11T19:23:48.9973843Z INFO: ------------------------------------------------------------------------
2023-01-11T19:23:48.9974389Z INFO: Total time: 1:16:06.089s
2023-01-11T19:23:49.8516260Z INFO: Final Memory: 2893M/10040M
2023-01-11T19:23:49.8517391Z INFO: ------------------------------------------------------------------------

ganncamp · January 26, 2023, 5:08pm

Hi,

Welcome to the community!

Before we go any further with this, can you try disabling your custom rules & re-running analysis to see if it’s still slow?

Thx,
Ann

Dennis_DECA · January 27, 2023, 1:00pm

Thank you for your reply.

Installed plugins:

Checkstyle 9.0.1
Dependency-Check 2.0.8
Findbugs 4.0.3
Custom rule we created ourselves
PMD 3.3.1

Uninstalled the plugins:

Delete the plugin from the <SONARQUBE_HOME>/extensions/plugins folder.
Reboot SonarQube
Run scan again

Went down from about 75 minutes to about 40 minutes. 2x faster, but still slow, no? I do not have any references.

Log:

2023-01-27T12:03:03.4796406Z ##[group]Run sonarsource/sonarqube-scan-action@master
2023-01-27T12:03:03.4796741Z with:
2023-01-27T12:03:03.4796959Z   projectBaseDir: core
2023-01-27T12:03:03.4797190Z env:
2023-01-27T12:03:03.4797489Z   JAVA_HOME: /data/actions-runner/_work/_tool/Java_Temurin-Hotspot_jdk/8.0.352-8/x64
2023-01-27T12:03:03.4797886Z   JAVA_HOME_8_X64: /data/actions-runner/_work/_tool/Java_Temurin-Hotspot_jdk/8.0.352-8/x64
2023-01-27T12:03:03.4798304Z   SONAR_TOKEN: ***
2023-01-27T12:03:03.4798574Z   SONAR_HOST_URL: XXXXX
2023-01-27T12:03:03.4798834Z ##[endgroup]
2023-01-27T12:03:03.5024469Z ##[command]/usr/bin/docker run --name b0279015a7dd823314fb7a4d4536992d498af_7c5f22 --label 0b0279 --workdir /github/workspace --rm -e "JAVA_HOME" -e "JAVA_HOME_8_X64" -e "SONAR_TOKEN" -e "SONAR_HOST_URL" -e "INPUT_PROJECTBASEDIR" -e "INPUT_ARGS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true --entrypoint "/entrypoint.sh" -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/data/actions-runner/_work/_temp/_github_home":"/github/home" -v "/data/actions-runner/_work/_temp/_github_workflow":"/github/workflow" -v "/data/actions-runner/_work/_temp/_runner_file_commands":"/github/file_commands" -v "/data/actions-runner/_work/miles-core/miles-core":"/github/workspace" 0b0279:015a7dd823314fb7a4d4536992d498af
2023-01-27T12:03:04.3031210Z INFO: Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
2023-01-27T12:03:04.3049570Z INFO: Project root configuration file: /github/workspace/core/sonar-project.properties
2023-01-27T12:03:04.3323266Z INFO: SonarScanner 4.8.0.2856
2023-01-27T12:03:04.3324234Z INFO: Java 11.0.17 Alpine (64-bit)
2023-01-27T12:03:04.3324780Z INFO: Linux 4.18.0-348.el8.x86_64 amd64
2023-01-27T12:03:04.5795453Z INFO: User cache: /opt/sonar-scanner/.sonar/cache
2023-01-27T12:03:05.0299648Z INFO: Analyzing on SonarQube server 8.9.3
2023-01-27T12:03:05.0302281Z INFO: Default locale: "en_US", source code encoding: "UTF-8"
2023-01-27T12:03:05.3362537Z INFO: Load global settings
2023-01-27T12:03:05.6773267Z INFO: Load global settings (done) | time=342ms
2023-01-27T12:03:05.6799472Z INFO: Server id: XXXXXXXXXX
2023-01-27T12:03:05.6820140Z INFO: User cache: /opt/sonar-scanner/.sonar/cache
2023-01-27T12:03:05.6841748Z INFO: Load/download plugins
2023-01-27T12:03:05.6842304Z INFO: Load plugins index
2023-01-27T12:03:05.9472935Z INFO: Load plugins index (done) | time=263ms
2023-01-27T12:03:11.0213542Z INFO: Load/download plugins (done) | time=5337ms
2023-01-27T12:03:11.3647491Z INFO: Process project properties
2023-01-27T12:03:11.3710694Z INFO: Process project properties (done) | time=6ms
2023-01-27T12:03:11.3711683Z INFO: Execute project builders
2023-01-27T12:03:11.3723316Z INFO: Execute project builders (done) | time=1ms
2023-01-27T12:03:11.3747978Z INFO: Project key: miles:core
2023-01-27T12:03:11.3748572Z INFO: Base dir: /github/workspace/core
2023-01-27T12:03:11.3749205Z INFO: Working dir: /github/workspace/core/MyBuild/sonar
2023-01-27T12:03:11.4589572Z INFO: Load project settings for component key: 'miles:core'
2023-01-27T12:03:11.6800319Z INFO: Load project settings for component key: 'miles:core' (done) | time=221ms
2023-01-27T12:03:11.7094747Z INFO: Auto-configuring with CI 'Github Actions'
2023-01-27T12:03:11.7102125Z INFO: Load quality profiles
2023-01-27T12:03:11.9696395Z INFO: Load quality profiles (done) | time=260ms
2023-01-27T12:03:11.9720083Z INFO: Auto-configuring with CI 'Github Actions'
2023-01-27T12:03:11.9745830Z INFO: Load active rules
2023-01-27T12:03:16.5319553Z INFO: Load active rules (done) | time=4557ms
2023-01-27T12:03:16.5715686Z INFO: Indexing files...
2023-01-27T12:03:16.5720354Z INFO: Project configuration:
2023-01-27T12:03:16.5725550Z INFO:   Excluded sources: **/*Stub.java, **/*.jar, **/*Test.java, **/*XXXX.xml, ...
2023-01-27T12:03:19.9413865Z INFO: 20315 files indexed
2023-01-27T12:03:19.9414526Z INFO: 131 files ignored because of inclusion/exclusion patterns
2023-01-27T12:03:19.9420758Z INFO: Quality profile for java: Sofico Web
2023-01-27T12:03:19.9421282Z INFO: Quality profile for xml: Sonar way
2023-01-27T12:03:19.9422193Z INFO: ------------- Run sensors on module MilesCore
2023-01-27T12:03:20.0226986Z INFO: Load metrics repository
2023-01-27T12:03:20.3192277Z INFO: Load metrics repository (done) | time=296ms
2023-01-27T12:03:20.9153766Z INFO: Sensor JavaSquidSensor [java]
2023-01-27T12:03:21.0882910Z INFO: Configured Java source version (sonar.java.source): 8
2023-01-27T12:03:21.0955584Z INFO: JavaClasspath initialization
2023-01-27T12:03:21.1051802Z INFO: JavaClasspath initialization (done) | time=9ms
2023-01-27T12:03:21.1052389Z INFO: JavaTestClasspath initialization
2023-01-27T12:03:21.1065530Z INFO: JavaTestClasspath initialization (done) | time=2ms
2023-01-27T12:03:21.1119093Z INFO: Java Main Files AST scan
2023-01-27T12:03:21.1286083Z INFO: 19682 source files to be analyzed
2023-01-27T12:03:21.1451664Z INFO: Load project repositories
2023-01-27T12:03:22.1782178Z INFO: Load project repositories (done) | time=1033ms
2023-01-27T12:03:27.7557797Z WARN: A multicriteria issue exclusion uses the rule key 'squid:CommentedOutCodeLine' that has been changed. The pattern should be updated to 'java:S125'
2023-01-27T12:03:31.1290749Z INFO: 75/19682 files analyzed, current file:
2023-01-27T12:03:41.1294136Z INFO: 166/19682 files analyzed, current file:
******
2023-01-27T12:38:21.2166036Z INFO: 19636/19682 files analyzed, current file:
2023-01-27T12:38:25.2860327Z INFO: 19682/19682 source files have been analyzed
2023-01-27T12:38:25.2870932Z INFO: Slowest analyzed files:
2023-01-27T12:38:25.2871490Z      XXXXXX.java (15187ms, 30349B)
2023-01-27T12:38:25.2876655Z      XXXXXX.java (11678ms, 940865B)
2023-01-27T12:38:25.2877210Z      XXXXXX.java (10045ms, 1062022B)
2023-01-27T12:38:25.2877705Z      XXXXXX.java (9945ms, 872232B)
2023-01-27T12:38:25.2878268Z      XXXXXX.java (8141ms, 421705B)
2023-01-27T12:38:25.2878856Z      XXXXXX.java (8103ms, 1004916B)
2023-01-27T12:38:25.2879490Z      XXXXXX.java (7877ms, 676694B)
2023-01-27T12:38:25.2881886Z      XXXXXX.java (7799ms, 604244B)
2023-01-27T12:38:25.2882512Z      XXXXXX.java (7190ms, 447210B)
2023-01-27T12:38:25.2883005Z      XXXXXX.java (7056ms, 25373B)
2023-01-27T12:38:25.2884013Z WARN: Unresolved imports/types have been detected during analysis. Enable DEBUG mode to see them.
2023-01-27T12:38:25.2934008Z INFO: Java Main Files AST scan (done) | time=2104181ms
2023-01-27T12:38:25.2934426Z INFO: Java Test Files AST scan
2023-01-27T12:38:25.2990448Z INFO: 493 source files to be analyzed
2023-01-27T12:38:26.4968536Z WARN: Invalid character encountered in file  XXXXXX.java at line 1292 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
2023-01-27T12:38:27.0988221Z WARN: Invalid character encountered in file  XXXXXX.java at line 528 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
2023-01-27T12:38:35.2996810Z INFO: 326/493 files analyzed, current file:  XXXXXX.java
2023-01-27T12:38:40.5677156Z INFO: 493/493 source files have been analyzed
2023-01-27T12:38:40.5677861Z WARN: Unresolved imports/types have been detected during analysis. Enable DEBUG mode to see them.
2023-01-27T12:38:40.5684720Z INFO: Java Test Files AST scan (done) | time=15275ms
2023-01-27T12:38:40.5685034Z INFO: Java Generated Files AST scan
2023-01-27T12:38:40.5689535Z INFO: 0 source files to be analyzed
2023-01-27T12:38:40.5690025Z INFO: 0/0 source files have been analyzed
2023-01-27T12:38:40.5690390Z INFO: Java Generated Files AST scan (done) | time=0ms
2023-01-27T12:38:40.5690736Z INFO: Sensor JavaSquidSensor [java] (done) | time=2119654ms
2023-01-27T12:38:40.5691059Z INFO: Sensor CSS Rules [cssfamily]
2023-01-27T12:38:40.5836672Z INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
2023-01-27T12:38:40.5837230Z INFO: Sensor CSS Rules [cssfamily] (done) | time=15ms
2023-01-27T12:38:40.5837570Z INFO: Sensor JaCoCo XML Report Importer [jacoco]
2023-01-27T12:38:40.6175529Z INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
2023-01-27T12:38:40.6178628Z INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
2023-01-27T12:38:40.6179117Z INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=34ms
2023-01-27T12:38:40.6179467Z INFO: Sensor C# Project Type Information [csharp]
2023-01-27T12:38:40.6236458Z INFO: Sensor C# Project Type Information [csharp] (done) | time=6ms
2023-01-27T12:38:40.6237240Z INFO: Sensor C# Properties [csharp]
2023-01-27T12:38:40.6241886Z INFO: Sensor C# Properties [csharp] (done) | time=0ms
2023-01-27T12:38:40.6242282Z INFO: Sensor SurefireSensor [java]
2023-01-27T12:38:40.6248768Z INFO: parsing [/github/workspace/core/target/surefire-reports]
2023-01-27T12:38:40.6252603Z INFO: Sensor SurefireSensor [java] (done) | time=1ms
2023-01-27T12:38:40.6253018Z INFO: Sensor Removed properties sensor [java]
2023-01-27T12:38:40.6255965Z WARN: Property 'sonar.jacoco.reportPath' is no longer supported. Use JaCoCo's xml report and sonar-jacoco plugin.
2023-01-27T12:38:40.6256680Z WARN: Property 'sonar.jacoco.itReportPath' is no longer supported. Use JaCoCo's xml report and sonar-jacoco plugin.
2023-01-27T12:38:40.6257114Z INFO: Sensor Removed properties sensor [java] (done) | time=0ms
2023-01-27T12:38:40.6257429Z INFO: Sensor JavaXmlSensor [java]
2023-01-27T12:38:40.7269075Z INFO: 63 source files to be analyzed
2023-01-27T12:38:41.3591017Z INFO: 63/63 source files have been analyzed
2023-01-27T12:38:41.3591701Z INFO: Sensor JavaXmlSensor [java] (done) | time=733ms
2023-01-27T12:38:41.3592250Z INFO: Sensor HTML [web]
2023-01-27T12:38:41.3887935Z INFO: Sensor HTML [web] (done) | time=30ms
2023-01-27T12:38:41.3888444Z INFO: Sensor XML Sensor [xml]
2023-01-27T12:38:41.3941671Z INFO: 3 source files to be analyzed
2023-01-27T12:38:41.7976419Z INFO: 3/3 source files have been analyzed
2023-01-27T12:38:41.7977189Z INFO: Sensor XML Sensor [xml] (done) | time=409ms
2023-01-27T12:38:41.7977739Z INFO: Sensor VB.NET Project Type Information [vbnet]
2023-01-27T12:38:41.8042713Z INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=6ms
2023-01-27T12:38:41.8049504Z INFO: Sensor VB.NET Properties [vbnet]
2023-01-27T12:38:41.8050229Z INFO: Sensor VB.NET Properties [vbnet] (done) | time=1ms
2023-01-27T12:38:41.8081791Z INFO: ------------- Run sensors on project
2023-01-27T12:38:41.8270519Z INFO: Sensor Zero Coverage Sensor
2023-01-27T12:38:45.1500036Z INFO: Sensor Zero Coverage Sensor (done) | time=3323ms
2023-01-27T12:38:45.1500722Z INFO: Sensor Java CPD Block Indexer
2023-01-27T12:39:10.4265474Z INFO: Sensor Java CPD Block Indexer (done) | time=25276ms
2023-01-27T12:39:10.4274704Z INFO: SCM Publisher is disabled
2023-01-27T12:39:12.4704744Z INFO: CPD Executor 3465 files had no CPD blocks
2023-01-27T12:39:12.4705371Z INFO: CPD Executor Calculating CPD for 16217 files
2023-01-27T12:39:12.5378744Z WARN: Too many duplication references on file XXXXXX.java for block at line 61. Keep only the first 100 references.
2023-01-27T12:39:12.5383534Z WARN: Too many duplication references on file XXXXXX.java for block at line 61. Keep only the first 100 references.
2023-01-27T12:39:12.5389204Z WARN: Too many duplication references on file XXXXXX.java for block at line 66. Keep only the first 100 references.
******
WARN Line 341 -> 14106 
******
2023-01-27T12:39:39.6404752Z INFO: CPD Executor CPD calculation finished (done) | time=27169ms
2023-01-27T12:39:46.1077498Z INFO: Analysis report generated in 5233ms, dir size=374 MB
2023-01-27T12:43:41.9268233Z INFO: Analysis report compressed in 235817ms, zip size=128 MB
2023-01-27T12:43:46.7710281Z INFO: Analysis report uploaded in 4845ms
2023-01-27T12:43:46.7726962Z INFO: ANALYSIS SUCCESSFUL, you can browse XXXXXX
2023-01-27T12:43:46.7727478Z INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
2023-01-27T12:43:46.7727994Z INFO: More about the report processing at XXXXXX
2023-01-27T12:43:50.7677450Z INFO: Analysis total time: 40:39.665 s
2023-01-27T12:43:50.7731435Z INFO: ------------------------------------------------------------------------
2023-01-27T12:43:50.7732302Z INFO: EXECUTION SUCCESS
2023-01-27T12:43:50.7733332Z INFO: ------------------------------------------------------------------------
2023-01-27T12:43:50.7733968Z INFO: Total time: 40:46.471s
2023-01-27T12:43:51.1629440Z INFO: Final Memory: 14M/736M
2023-01-27T12:43:51.1630260Z INFO: ------------------------------------------------------------------------

ganncamp · January 27, 2023, 1:45pm

Hi,

Ehm… 3million LOC is a lot, and a I’m not shocked that it takes a while.

In terms of further improvement, here’s what I can tell you:

My experience of the FindBugs plugin is that it’s just slow in general. You can probably restore Checkstyle, PMD and Dependency-Check without a signification slow down. No clue about your custom rules. You should probably add plugins back one at a time tho.
I believe that if you turn on debug logging you’ll get more detail about where the time is going. That could be helpful
We’ve done a lot of work on speed in the 9-series, with a significant improvement to basic Java analysis in 9.4. (Obvs, you shouldn’t upgrade to 9.4, but go straight to the current: 9.8).
We’ve also sped up PR analysis (Developer Edition($)+) in the 9-series by only analyzing the files changed in the PR. This may help day-to-day

And just in case you’re an LTS-only shop, SonarQube 9.9 LTS is due on 7 Feb, so you either way I guess there’s an upgrade in your near future.

Ann

Dennis_DECA · January 30, 2023, 7:58am

Hi Ann,

Ok, just getting realistic expectations. My hope was that there was still some secret magic left

Upgrading to 9.9 LTS is on my radar already.
After the upgrade I will check again if the timings improved.
I might also request a trail license for DE after upgrading to 9.9.X to checkout the PR analysis speed.

Would you like me to do another run with debug logging on and provide the details here?

ganncamp · January 30, 2023, 12:23pm

Hi,

The debug logging would tell you specific rules the time was going into. If you were interested, you could evaluate turning them off. But I don’t need the logging since you’re on 8.9 and we already know you’ll get a bit speed boost with an upgrade.

Ann