SCM analysis of large repository fluctuates between 5 minutes and 2 hours

SonarQube
1

Must-share information (formatted with Markdown):

  • Enterprise Edition Version 8.9.3 (build 48735) for SQ server, 3.7.0.1746 for Scanner
  • Scanning a large Java repository.
  • Tweaking to provide more memory, using shallow clones.

Hi there,

We’re trying to scan a large Java 8 repository with SQ using Maven and are running into builds stalling on SQ scanning stage intermittently. The overall scan can take 15-20 minutes normally, but then periodically take 2+ hours and time out entirely. The stage that is causing this problem appears to be SCM scanning:

[INFO] Sensor Zero Coverage Sensor
[INFO] Sensor Zero Coverage Sensor (done) | time=284ms
[INFO] Sensor Java CPD Block Indexer
[INFO] Sensor Java CPD Block Indexer (done) | time=2777ms
[INFO] SCM Publisher SCM provider for this project is: git
[INFO] SCM Publisher 32 source files to be analyzed
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] 
[INFO] SCM Publisher 23/32 source files have been analyzed (done) | time=4828157ms
[WARNING] Missing blame information for the following files:
...

Something is definitely not right here since it appears to take 1.5+ hours to scan what it claims 23 files. The other puzzling variable is that this is completely intermittent, and same exact repository can have the following output for that same SCM stage:

[INFO] Sensor Zero Coverage Sensor
[INFO] Sensor Zero Coverage Sensor (done) | time=240ms
[INFO] Sensor Java CPD Block Indexer
[INFO] Sensor Java CPD Block Indexer (done) | time=2549ms
[INFO] SCM Publisher SCM provider for this project is: git
[INFO] SCM Publisher 23 source files to be analyzed
[INFO] SCM Publisher 14/23 source files have been analyzed (done) | time=3538ms

So here - the same exact operation took only 3 seconds…
Another strange thing - somehow its claiming 23 source code files instead of previous 32, while nothing of significance has changed in the source code. In other builds I see different numbers… Yet I cannot connect that to any meaningful value like number of files changed in a given build.

At this point we’ve tried to increase the amount of memory, which hasn’t helped. And now trying a shallow clone to see if that would help at all, but I realize that would mess with SQ features like blame and automatic issue assignment.

Is there anything else we should be looking at? Any thoughts on what could be causing this strange behaviour?

2

Seeing a bunch of edits on the original question by the SQ community team - but zero responses of any kind is pretty disheartening and does not instill confidence in the product or the community one bit.

2 Likes