devk_s
October 21, 2021, 12:05pm
1
SonarQube ID information
When we scan our maven build we encounterd a bottelneck while the surefire-reports are beeing parsed by the maven-sonarqube-plugin.
pom.xml
<properties>
<java.version>11</java.version>
<sonar.jacoco.myReportPath>${project.basedir}/target/jacoco.exec</sonar.jacoco.myReportPath>
<!-- <surefire.version>2.22.2</surefire.version> -->
<surefire.version>3.0.0-M5</surefire.version>
[...]
</properties>
[...]
<plugin>
<groupId>org.sonarsource.scanner.maven</groupId>
<artifactId>sonar-maven-plugin</artifactId>
<version>3.6.0.1398</version>
</plugin>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.5</version>
<configuration>
<append>true</append>
</configuration>
<executions>
<execution>
<id>prepare-agent</id>
<goals>
<goal>prepare-agent</goal>
</goals>
</execution>
<execution>
<id>report</id>
<goals>
<goal>report</goal>
</goals>
</execution>
</executions>
</plugin>
[...]
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire.version}</version>
<configuration>
<groups>unit</groups>
<excludedGroups>integration</excludedGroups>
<parallel>classesAndMethods</parallel>
<useUnlimitedThreads>true</useUnlimitedThreads>
<!-- <reportFormat>plain</reportFormat> -->
<!-- <skip.surefire.tests>true</skip.surefire.tests> -->
<!-- <skipTests>true</skipTests> -->
<argLine>-Xmx2G -XX:MaxPermSize=1G -XX:-UseSplitVerifier</argLine>
</configuration>
</plugin>
So the sonar scan itself only takes 1 minute. But after everything is done the step “parsing surefire-reports…” takes around 10 minutes.
The amount of reports is still relatively small right now and we are planing to add more tests. But we are not sure if want to use the sonar-maven-plugin if it takes this much time.
Why is it taking so long for a vanilla maven build?
Greetz
devk_s
October 21, 2021, 1:22pm
3
same result with the new sonar-maven-plugin
<plugin>
<groupId>org.sonarsource.scanner.maven</groupId>
<artifactId>sonar-maven-plugin</artifactId>
<!-- <version>3.6.0.1398</version> -->
<version>3.9.0.2155</version>
</plugin>
[INFO] parsing [/home/selfservice-api/target/surefire-reports] 10 minutes
[INFO] ------------------< dige.selfservice:selfservice-api >------------------
[INFO] Building selfservice-api 0.0.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- sonar-maven-plugin:3.9.0.2155:sonar (default-cli) @ selfservice-api ---
[INFO] User cache: /home/xxx/.sonar/cache
[INFO] SonarQube version: 8.9.2
[INFO] Default locale: "en_US", source code encoding: "UTF-8"
[INFO] Load global settings
[INFO] Load global settings (done) | time=338ms
[INFO] Server id: XXXX
[INFO] User cache: /home/xxx/.sonar/cache
[INFO] Load/download plugins
[INFO] Load plugins index
[INFO] Load plugins index (done) | time=79ms
[INFO] Load/download plugins (done) | time=217ms
[INFO] Loaded core extensions: developer-scanner
[INFO] JavaScript/TypeScript frontend is enabled
[INFO] Process project properties
[INFO] Process project properties (done) | time=12ms
[INFO] Execute project builders
[INFO] Execute project builders (done) | time=2ms
[INFO] Project key: alm-deploy-cloud-testi-dige
[INFO] Base dir: /api
[INFO] Working dir: /api/target/sonar
[INFO] Load project settings for component key: 'alm-deploy-cloud-testi-dige'
[INFO] Load project settings for component key: 'alm-deploy-cloud-testi-dige' (done) | time=112ms
[INFO] Load project branches
[INFO] Load project branches (done) | time=58ms
[INFO] Load project pull requests
[INFO] Load project pull requests (done) | time=73ms
[INFO] Load branch configuration
[INFO] Load branch configuration (done) | time=6ms
[INFO] Load quality profiles
[INFO] Load quality profiles (done) | time=147ms
[INFO] Load active rules
[INFO] Load active rules (done) | time=4483ms
[INFO] Indexing files...
[INFO] Project configuration:
[INFO] Excluded sources: **/test/**
[INFO] Excluded sources for coverage: **/dto/**, **/types/**, **/thirdParty/**, **/exeptions/**, **/type/**, **/impl/**, **/connector/**, **/controller/**, **/data/**, **/filter/**, **Provider.java
[INFO] 285 files indexed
[INFO] 0 files ignored because of inclusion/exclusion patterns
[INFO] 0 files ignored because of scm ignore settings
[INFO] Quality profile for java: Sonar way
[INFO] Quality profile for xml: Sonar way
[INFO] ------------- Run sensors on module alm-deploy/cloud/testi/dige
[INFO] JavaScript/TypeScript frontend is enabled
[INFO] Load metrics repository
[INFO] Load metrics repository (done) | time=41ms
[INFO] Sensor JavaSquidSensor [java]
[INFO] Configured Java source version (sonar.java.source): 11
[INFO] JavaClasspath initialization
[INFO] JavaClasspath initialization (done) | time=5ms
[INFO] JavaTestClasspath initialization
[INFO] JavaTestClasspath initialization (done) | time=1ms
[INFO] Java Main Files AST scan
[INFO] 174 source files to be analyzed
[INFO] Load project repositories
[INFO] Load project repositories (done) | time=114ms
Status ERROR: org.eclipse.jdt.core code=4 Could not retrieve declared methods java.lang.NullPointerException
Status ERROR: org.eclipse.jdt.core code=4 Could not retrieve declared methods java.lang.NullPointerException
Status ERROR: org.eclipse.jdt.core code=4 Could not retrieve declared methods java.lang.NullPointerException
Status ERROR: org.eclipse.jdt.core code=4 Could not retrieve declared methods java.lang.NullPointerException
Status ERROR: org.eclipse.jdt.core code=4 Could not retrieve declared methods java.lang.NullPointerException
[INFO] 60/174 files analyzed, current file: src/main/java/dige/selfservice/selfserviceapi/connector/paris/ParisCustomerCheck.java
Status ERROR: org.eclipse.jdt.core code=4 Could not retrieve declared methods java.lang.NullPointerException
[INFO] 154/174 files analyzed, current file: src/main/java/dige/selfservice/selfserviceapi/data/DataSourceConfig.java
Status ERROR: org.eclipse.jdt.core code=4 Could not retrieve declared methods java.lang.NullPointerException
Status ERROR: org.eclipse.jdt.core code=4 Could not retrieve declared methods java.lang.NullPointerException
[INFO] 174/174 source files have been analyzed
[WARNING] Unresolved imports/types have been detected during analysis. Enable DEBUG mode to see them.
[INFO] Java Main Files AST scan (done) | time=21391ms
[INFO] Java Test Files AST scan
[INFO] 104 source files to be analyzed
[INFO] 104/104 source files have been analyzed
[WARNING] Unresolved imports/types have been detected during analysis. Enable DEBUG mode to see them.
[INFO] Java Test Files AST scan (done) | time=9170ms
[INFO] Java Generated Files AST scan
[INFO] 0 source files to be analyzed
[INFO] 0/0 source files have been analyzed
[INFO] Java Generated Files AST scan (done) | time=1ms
[INFO] Sensor JavaSquidSensor [java] (done) | time=30862ms
[INFO] Sensor CoberturaSensor [cobertura]
[WARNING] Cobertura report not found at /api/target/site/cobertura/coverage.xml
[INFO] Sensor CoberturaSensor [cobertura] (done) | time=1ms
[INFO] Sensor CSS Rules [cssfamily]
[INFO] No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
[INFO] Sensor CSS Rules [cssfamily] (done) | time=2ms
[INFO] Sensor C# Project Type Information [csharp]
[INFO] Sensor C# Project Type Information [csharp] (done) | time=2ms
[INFO] Sensor C# Properties [csharp]
[INFO] Sensor C# Properties [csharp] (done) | time=1ms
[INFO] Sensor SurefireSensor [java]
[INFO] parsing [/api/target/surefire-reports]
[INFO] Sensor SurefireSensor [java] (done) | time=651841ms
[INFO] Sensor JavaXmlSensor [java]
[INFO] 1 source file to be analyzed
[INFO] 1/1 source file has been analyzed
[INFO] Sensor JavaXmlSensor [java] (done) | time=161ms
[INFO] Sensor HTML [web]
[INFO] Sensor HTML [web] (done) | time=4ms
[INFO] Sensor XML Sensor [xml]
[INFO] Sensor XML Sensor [xml] (done) | time=1ms
[INFO] Sensor CheckstyleSensor [checkstyle]
[INFO] Checkstyle output report: /api/target/sonar/checkstyle-result.xml
[INFO] Checkstyle configuration: /api/target/sonar/checkstyle.xml
[INFO] Checkstyle charset: UTF-8
[INFO] Sensor CheckstyleSensor [checkstyle] (done) | time=981ms
[INFO] Sensor VB.NET Project Type Information [vbnet]
[INFO] Sensor VB.NET Project Type Information [vbnet] (done) | time=1ms
[INFO] Sensor VB.NET Properties [vbnet]
[INFO] Sensor VB.NET Properties [vbnet] (done) | time=3ms
[INFO] Sensor SmellMeasuresSensor [smells]
[INFO] Sensor SmellMeasuresSensor [smells] (done) | time=119ms
[INFO] Sensor JaCoCo XML Report Importer [jacoco]
[INFO] 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
[INFO] Importing 1 report(s). Turn your logs in debug mode in order to see the exhaustive list.
[INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=277ms
[INFO] Sensor ThymeLeaf template sensor [securityjavafrontend]
[INFO] Sensor ThymeLeaf template sensor [securityjavafrontend] (done) | time=5ms
[INFO] Sensor FindBugs Sensor [findbugs]
[INFO] Loading findbugs plugin: /api/target/sonar/findbugs/findsecbugs-plugin.jar
[INFO] Findbugs output report: /api/target/sonar/findbugs-result.xml
The following classes needed for analysis were missing:
makeConcatWithConstants
accept
test
apply
get
writeTo
[INFO] Sensor FindBugs Sensor [findbugs] (done) | time=23676ms
[INFO] Sensor JavaSecuritySensor [security]
[INFO] Reading type hierarchy from: /api/target/sonar/ucfg2/java
[INFO] Read 616 type definitions
[INFO] Reading UCFGs from: /api/target/sonar/ucfg2/java
[INFO] 15:17:52.006612 Building Runtime Type propagation graph
[INFO] 15:17:52.095491 Running Tarjan on 3874 nodes
[INFO] 15:17:52.109794 Tarjan found 3841 components
[INFO] 15:17:52.140651 Variable type analysis: done
[INFO] 15:17:52.147871 Building Runtime Type propagation graph
[INFO] 15:17:52.211179 Running Tarjan on 3874 nodes
[INFO] 15:17:52.21887 Tarjan found 3841 components
[INFO] 15:17:52.233447 Variable type analysis: done
[INFO] Analyzing 541 ucfgs to detect vulnerabilities.
[INFO] All rules entrypoints : 3 Retained UCFGs : 81
[INFO] rule: S5131, entrypoints: 0
[INFO] rule: S5131 done
[INFO] rule: S3649, entrypoints: 0
[INFO] rule: S3649 done
[INFO] rule: S2076, entrypoints: 0
[INFO] rule: S2076 done
[INFO] rule: S2091, entrypoints: 0
[INFO] rule: S2091 done
[INFO] rule: S2078, entrypoints: 0
[INFO] rule: S2078 done
[INFO] rule: S2631, entrypoints: 3
[INFO] Running symbolic analysis
[INFO] rule: S2631 done
[INFO] rule: S5135, entrypoints: 0
[INFO] rule: S5135 done
[INFO] rule: S2083, entrypoints: 0
[INFO] rule: S2083 done
[INFO] rule: S5167, entrypoints: 0
[INFO] rule: S5167 done
[INFO] rule: S5144, entrypoints: 0
[INFO] rule: S5144 done
[INFO] rule: S5145, entrypoints: 0
[INFO] rule: S5145 done
[INFO] rule: S5146, entrypoints: 0
[INFO] rule: S5146 done
[INFO] rule: S5334, entrypoints: 0
[INFO] rule: S5334 done
[INFO] rule: S6096, entrypoints: 0
[INFO] rule: S6096 done
[INFO] Sensor JavaSecuritySensor [security] (done) | time=3485ms
[INFO] Sensor CSharpSecuritySensor [security]
[INFO] Reading type hierarchy from: /api/target/ucfg_cs2
[INFO] Read 0 type definitions
[INFO] Reading UCFGs from: /api/target/ucfg_cs2
[INFO] No UCFGs have been included for analysis.
[INFO] Sensor CSharpSecuritySensor [security] (done) | time=1ms
[INFO] Sensor PhpSecuritySensor [security]
[INFO] Reading type hierarchy from: /api/target/sonar/ucfg2/php
[INFO] Read 0 type definitions
[INFO] Reading UCFGs from: /api/target/sonar/ucfg2/php
[INFO] No UCFGs have been included for analysis.
[INFO] Sensor PhpSecuritySensor [security] (done) | time=1ms
[INFO] Sensor PythonSecuritySensor [security]
[INFO] Reading type hierarchy from: /api/target/sonar/ucfg2/python
[INFO] Read 0 type definitions
[INFO] Reading UCFGs from: /api/target/sonar/ucfg2/python
[INFO] No UCFGs have been included for analysis.
[INFO] Sensor PythonSecuritySensor [security] (done) | time=2ms
[INFO] Sensor JsSecuritySensor [security]
[INFO] Reading type hierarchy from: /api/target/sonar/ucfg2/js
[INFO] Read 0 type definitions
[INFO] Reading UCFGs from: /api/target/sonar/ucfg2/js
[INFO] No UCFGs have been included for analysis.
[INFO] Sensor JsSecuritySensor [security] (done) | time=1ms
[INFO] ------------- Run sensors on project
[INFO] Sensor Zero Coverage Sensor
[INFO] Sensor Zero Coverage Sensor (done) | time=0ms
[INFO] Sensor Java CPD Block Indexer
[INFO] Sensor Java CPD Block Indexer (done) | time=381ms
[INFO] CPD Executor 58 files had no CPD blocks
[INFO] CPD Executor Calculating CPD for 116 files
[INFO] CPD Executor CPD calculation finished (done) | time=181ms
[INFO] Load New Code definition
[INFO] Load New Code definition (done) | time=296ms
[INFO] Analysis report generated in 507ms, dir size=1 MB
[INFO] Analysis report compressed in 723ms, zip size=817 KB
[INFO] Analysis report uploaded in 1614ms
[INFO] ------------- Check Quality Gate status
[INFO] Waiting for the analysis report to be processed (max 600s)
[INFO] QUALITY GATE STATUS: PASSED - View details on https://sonar.prod.internal.xxx.de/dashboard?id=alm-deploy-cloud-testi-dige
[INFO] Analysis total time: 13:33.919 s
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15:43 min
[INFO] Finished at: 2021-10-21T15:19:29+02:00
[INFO] ------------------------------------------------------------------------
ganncamp
October 25, 2021, 1:25pm
4
Hi,
Welcome to the community!
Your pom snippet indicates you’re passing a JaCoCo .exec file. You should be passing an XML report. Can you try that?
bump… looks like i have the same issue, takes a long time when parsing surefire report (initially it took about 5 minutes, but now we’re exceeding the 30mins. We have multiple projects based on the same blueprints on the structure like pipeline and poms. Project A has 1335 surefire report files, and project B 400 files. Project A runs 34 minutes, and project B is done within 3 minutes. What can be the cause of this?
