We are using SonarQube Developer’s Edition (v.8.4.2.36762). We have a large Java project that we are able to scan with the Sonar Scanner CLI. But when we tried to scan the same project with Maven plug-in, the process hangs at the following step. I tried to remove the rule S5131, but the process would hang at the next rule.
[INFO] Findbugs output report: C:\Users\caojq\IdeaProjects\eomis\WICSwar\target\sonar\findbugs-result.xml
The following classes needed for analysis were missing:
accept
apply
compare
onCreate
processRow
run
test
mapRow
runMetadataTask
get
getStringValue
applyAsInt
handle
call
[INFO] Sensor FindBugs Sensor [findbugs] (done) | time=697493ms
[INFO] ------------- Run sensors on module WICSear
[INFO] Sensor JavaXmlSensor [java]
[INFO] 1 source files to be analyzed
[INFO] Sensor JavaXmlSensor [java] (done) | time=237ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor HTML [web]
[INFO] Sensor HTML [web] (done) | time=5ms
[INFO] Sensor XML Sensor [xml]
[INFO] 1 source files to be analyzed
[INFO] Sensor XML Sensor [xml] (done) | time=109ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor JaCoCo XML Report Importer [jacoco]
[INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=7ms
[INFO] ------------- Run sensors on module WICS
[INFO] Sensor JavaXmlSensor [java]
[INFO] 1 source files to be analyzed
[INFO] Sensor JavaXmlSensor [java] (done) | time=37ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor HTML [web]
[INFO] Sensor HTML [web] (done) | time=8ms
[INFO] Sensor XML Sensor [xml]
[INFO] 1 source files to be analyzed
[INFO] Sensor XML Sensor [xml] (done) | time=522ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor JaCoCo XML Report Importer [jacoco]
[INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=4ms
[INFO] Sensor JavaSecuritySensor [security]
[INFO] Reading type hierarchy from: C:\Users\caojq\IdeaProjects\eomis\target\sonar\ucfg2\java
[INFO] Read 2873 type definitions
[INFO] Reading UCFGs from: C:\Users\caojq\IdeaProjects\eomis\target\sonar\ucfg2\java
[INFO] 16:04:24.592 Building Type propagation graph
[INFO] 16:04:28.735 Running Tarjan on 526838 nodes
[INFO] 16:04:29.435 Tarjan found 525317 components
[INFO] 16:04:31.147 Variable type analysis: done
[INFO] 16:04:31.154 Building Type propagation graph
[INFO] 16:04:35.293 Running Tarjan on 526320 nodes
[INFO] 16:04:36.198 Tarjan found 524800 components
[INFO] 16:04:37.877 Variable type analysis: done
[INFO] Analyzing 56774 ucfgs to detect vulnerabilities.
[INFO] All rules entrypoints : 4381 Retained UCFGs : 34547
[INFO] rule: S5131, entrypoints: 3978
thank you for sharing. Could you define more precisely what you mean by “the process hangs”? The rule in question is a taint analysis rule, which requires a complex analysis. For a very large project, it is therefore normal that this analysis takes some time. How long did you wait until you canceled the process? Also, how much memory did you allocate for the analysis? (i.e., what is your -Xmx, if any).
Finally, does the analysis finish normally when you disable all of the taint analysis rules: S5131, S3649, S2076, S2091, S2078, S2631, S5135, S2083, S5167, S5144, S5145, S5146, and S5334 ?
Hi Malte,
The longest time we waited before killing the process was overnight (>10 hours). We are using the MAVEN_OPTS -Xms1024m -Xmx2048m.
Scanning using CLI: sonar-scanner -Dsonar.login=**** finished in about an hour. Here’s the output of the scan relating to the same rules:
INFO: Sensor FindBugs Sensor [findbugs]
INFO: Loading findbugs plugin: C:\Users\caojq\IdeaProjects\eomis.scannerwork\findbugs\findsecbugs-plugin.jar
INFO: Findbugs output report: C:\Users\caojq\IdeaProjects\eomis.scannerwork\findbugs-result.xml
INFO: Sensor FindBugs Sensor [findbugs] (done) | time=880132ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Reading type hierarchy from: C:\Users\caojq\IdeaProjects\eomis.scannerwork\ucfg2\java
INFO: Read 1197 type definitions
INFO: Reading UCFGs from: C:\Users\caojq\IdeaProjects\eomis.scannerwork\ucfg2\java
INFO: 11:53:16.599456 Building Type propagation graph
INFO: 11:53:17.6661218 Running Tarjan on 123713 nodes
INFO: 11:53:17.9011116 Tarjan found 123397 components
INFO: 11:53:18.308711 Variable type analysis: done
INFO: 11:53:18.3127057 Building Type propagation graph
INFO: 11:53:19.0927609 Running Tarjan on 123694 nodes
INFO: 11:53:19.2947542 Tarjan found 123378 components
INFO: 11:53:19.8414609 Variable type analysis: done
INFO: Analyzing 42643 ucfgs to detect vulnerabilities.
INFO: All rules entrypoints : 0 Retained UCFGs : 0
INFO: rule: S3649, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S3649 done
INFO: rule: S2076, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S2076 done
INFO: rule: S2091, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S2091 done
INFO: rule: S2078, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S2078 done
INFO: rule: S2631, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S2631 done
INFO: rule: S5135, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S5135 done
INFO: rule: S2083, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S2083 done
INFO: rule: S5167, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S5167 done
INFO: rule: S5144, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S5145 done
INFO: rule: S5146, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S5146 done
INFO: rule: S5334, entrypoints: 0
INFO: Visited 0 ucfgs in 0 ms, 0 steps
INFO: rule: S5334 done
INFO: Sensor JavaSecuritySensor [security] (done) | time=41330ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading type hierarchy from: C:\Users\caojq\IdeaProjects\eomis\ucfg_cs2
INFO: Read 0 type definitions
INFO: Reading UCFGs from: C:\Users\caojq\IdeaProjects\eomis\ucfg_cs2
INFO: No UCFGs have been included for analysis.
What’s the differences between the CLI and the Maven plug-in. Does this mean the CLI skipped those rules automatically? I will try to skip all rules in the Maven scan and let you know the result later.
when you use the CLI, then SonarQube will not be able to process the Java dependencies and will only scan your application without knowing any of those dependencies. This means that the analysis performed will be rather superficial and complex rules such as the taint analysis rules thus remain essentially unprocessed. SonarQube does attempt to run the rules, but it doesn’t really have enough information for a sensible analysis in this case. As you can see in the log, it tells you about “0 entrypoints” and “0 ucfgs” everywhere. That shouldn’t be the case - just compare that to the “4381 entrypoints” and “56774 ucfgs” in the log with the Maven scanner.
For Java projects, you should always use the Maven or Gradle scanners (or if your project is really really oldschool, the Ant scanner). The CLI scanner is more geared towards languages that do not have a build system (or at least when there is no specific scanner for that build system), in particular interpreted languages such as PHP, Python, or JavaScript for example.
Yes, please let me know if the scan runs through when you deactivate the rules mentioned above.
I have a suspicion that you are not giving the scan enough memory for these rules. The taint analysis rules do not only invoke a complex analysis, that analysis also requires quite a bit of memory. For a large project, the amount of memory given (-Xmx2048m) is rather scarce and the JVM may hence be busy 99% of the time just trying to free up some memory. In fact, I am surprised you did not run into an OOM. Could you try to give the scan significantly more memory (say, 12 GB or so) to see if that runs through? Note that the scan will probably still need some time, but it should be in the ballpark of minutes or tens of minutes (at most) and not 10 hours. If that helps, we will have a better feeling if that is indeed the problem, how long the scan takes when given enough memory, and we can try to pinpoint how much memory is really needed for the scan to finish in a reasonable amount of time.
Hi Malte,
I tried to set the -Xmx12g and it was still running very long (~10hours). As I mentioned that we were using v8.4, so we did the upgrade to v8.9. After the upgrade, the same scan, with -Xmx8g finished in 1.5 hours. So the upgrade fixed the issue.
Thanks,
John