Improve sonarscan time on python and java monolith repos

SonarQube Server / Community Build
, ,
1

Hi,
We currently use monolith repositories for our services written in java(npm,yarn) and python Wanted to know how can I improve on the time sonar takes to run? Is there away I can only run the scan on the new/changed code. We are currently using the community version but happy to explore the paid versions if this problem is solved.

Thanks

2

Hello,

Are you sure your monolith repos are made of Java? When I see “npm + yarn”, this makes me think about JavaScript or TypeScript but not Java.

Can you share some details about the size of your repos, in particular the number of Lines of Code for each Languages (you can find this info in SQ itself), the number of files and the total time to scan them?
(don’t forget to mention the version of SonarQube you are using also).

We are working to improve the scan time of Pull Requests so that only the changed files in PR are analyzed instead of analyzing all the files (and report issues only on changed files). This is not yet available but hopefully should come in 2022. This will be a feature of SonarQube Developer Edition.

Alex

3

Thanks, yes we have a repo which has both java and JS code. But let me reference the one with javascript and typescript here. Below are the details.
Sonarqube version - 8.9.2
Sonar Scanner version - 4.6.2

Screen Shot 2021-10-27 at 1.54.20 pm
Screen Shot 2021-10-27 at 1.54.20 pm1340Ă—564 17.9 KB

Time for scan - ~4min15secs
I noticed there are some error messages related to eslint, that may have caused the delay. Attaching the log file. Have edited the file to remove some sensative data.
Thanks

4

Hi Sonar team,
Can I have some help here please.

Thanks,
Prerna

6

Hello @Prerna
I think you forgot to attach log file you mentioned above, I see only 1 screenshot.

7

Hi Alex,

to speed up the analysis in general, it would already be very helpful if analysis in parallel (Jenkins pipelines) would be officially supported, but the related Jira ticket is still open, see

Gilbert

8

Hello @anon67236913,

Thanks for the feedback and the push of the “Scanner for Jenkins” ticket (it got lost in our backlog, incorrectly not tagged with the “performance” tag :frowning: ).

Our current plans to make performance better are:

  • analyze only the changed files in a PR so you don’t need to re-analyze all the LOCs of your repo for each PR - this will come first on SonarCloud and will be gradually ported to SonarQube
  • allow each analyzer to benefit from all the Cores of your CPUs: today it’s useless to have a monster machine with a lot of Cores, SonarSource Analyzers use only one

Also, just in case here are some recommendations to get the best performance available as of now:

  • make sure you run the scan on a Linux machine: you can get up to 50% improv. compared to a Windows box
  • if you have no choice and must use a Windows build agent, deactivate Windows Defender on that machine while running the scan
  • run the scan with JDK11+ (better perf compared to JDK8)
  • run the latest version of SonarQube (9.1+), especially if you are running a Commercial Edition, we did huge improvements on SQ DE+ on the security taint analyzer.

Alex

9

[quote=“Sonarqube Scanner for Jenkins, analysis in parallel ? [SONARJNKNS-316], post:1, topic:51415”]
https://docs.sonarqube.org/latest/analysis/scan/sonarscanner-for-jenkins/
[/quote]

Hi @Lena, apologies, I missed the log file. adding it here log (9).txt (8.6 KB). The lines ```java.io.IOException: Cannot run program “/home/jenkins/agent/workspace/Belongweb-sonarqube/.scannerwork/ce228c7b0152b0d6616c9bf171b8aaec-codescan-language-eslint-linux”: error=2,’’’ I think are the problem areas . Can you help please?

Thanks Alex, We are already running sonar on linux machines with JDK 12, since I have already requested a developer edition trail license, not planning on upgrading the CE version.
We are really looking forward to "analyze only the changed files in a PR " feature to come through to the SonarSource analyser , any ETA for now?
Is there a Jira Issue that I can keep an eye on?

Thanks heaps,
Prerna

11

Please provide the full log, not only the final part of it, I want to see which sensors are taking the time. Regarding CodeScanLang, I don’t know what it is, probably some community plugin you have installed, but definitely it’s not taking it long to fail, just 3 seconds :slight_smile:

12

Hi,

https://jira.sonarsource.com/browse/SONARJNKNS-316
has still status OPEN and no Fix version/s and it seems it needs some push again !?
Our developers ask nearly every week for support of scan in parallel.
“Some teams nevertheless use the Sonarqube analysis in parallel, the duration of the analysis
is reduced from 8-9 mins to 2-3 mins.” ain’t bad !?
As i wrote before, this would be a way to speed up the analysis in general - regardless of the language.
Guess it’s not only relevant for Jenkins, but also for similar tools.

Gilbert

1 Like