Following up on Parallel analysis in MAVEN/SonarScanner feature suggestion would be parallel analysis in MAVEN plugin.
Last time I checked, parallel execution in Maven was broken. Also, typically you only run the scanner once (for the root module), so you would only see any gains if you are running other tasks at the same time.
See comments here: https://jira.sonarsource.com/browse/MSONAR-7 (including a ticket in maven)
Since the referenced ticket has been created roughly 10 years ago, there is a chance that MAVEN has fixed/changed how it internally does parallelization.
However, it’s not necessarily MAVEN that can do parallelization, I believe it could also be done completely in a plugin like Sonar Scanner.
After all, I could start 26 maven builds on a machine with 80 cores with different values for:
in pom.xml and if I were able to consolidate the results into one SonarQube report, then that would probably speed up analysis by a considerable factor.
As a side note, for junit tests we taking a similar approach although the “distribution by first letter” there is used to distribute tests over different hardware (we’re using junit/jupiter parallelization also for speeding things up in a single process).
I understand that some checks might not work reliably with that kind of parallelization, e.g. duplication checks, checks for usages of @depcecated or unused variables etc.
But all checks that do not look beyond a single file should work nicely, and for short-lived branches that are reviewed through pull requests, that would be ok for me.
The ticket has relevant comments dating less than 3 years ago.
Anyway, if we were to do that internally (within the task), it would make sense to multithread the execution of sensors. That way the checks and other computation done by the sensor can be split effectively and the scanner can still aggregate all the results. It wouldn’t be specifically done for maven. That idea comes back from time to time and is one possible solution (with a lot of drawbacks!) to speed up SonarJava in certain running environments.
As you pointed out, there is already a ticket with the goal of speeding up SonarJava 6.x and the team will work on finding a solution. Your suggestion has been linked to it.
Thanks for taking the time to write here!
That’s right, the bug regarding parallel builds seems to have been fixed on maven side:
I’d like to share the view that sonar-scanner itself should be able to do a multi-threaded scan of a codebase without a dependance on a specific build system like maven.
I’ve reverted to using sonar-scanner.bat due to the fact that the maven plugin is fragile in my use-case.
My expectation is that sonar-scanner.bat can be ran with 8-16 threads and analyze all the discovered java files ( over 100,000+ java files ) in parallel.
The fragility in maven is due to the fact that the objective i’m trying to achieve is a high-level , top-down view of the static code analysis concerns for the whole code-base of the entire platform ( all in 1 sonar instance + one project ) , when this is attempted from the root-pom of my project , there are errors raised that are blocking errors when files are missing, my expectation is a fault-resistant analysis, where gaps in coverage, gaps in java code <-- --> java binaries should be reported at the end.
This would mean i’d achieve 99.5% of the code analysis, and error information detailing the failure to analyze the final 0.5% , with the sonar maven plugin today it appears this is not achievable.
So ultimately i wish to use the sonar-scanner CLI tool directly, and i then had an expectation i could mutli-thread it directly too.
If you have errors with the scanner for maven please report them so that we can have a look at it. The sonar-scanner CLI and scanner for maven are a thin layer over common shared components.
@dmeneses I’ll create a thread to share my issues. I’d be really happy to have your feedback on how to overcome my current inability to run sonar successfully on our project when its done top-down.
Parallel execution of the maven scanner seems like a reasonable ask, is there a open SONAR ticket on the topic I should follow?