Sonar downloads plugins which takes ~70 minutes

  • SonarQube Developer edition 9.5 (build 56709), Scanner (4.6.2.2472)

  • what are you trying to achieve
    => I have integrated sonarqube with GitLab-ci via GitLab runner. For every analysis it downloads lot of
    plugins and takes more than ~75-90 min minutes. I am looking for a way to reuse plugins cache (but once job finishes docker container gets cleanup) or to download only typescript/JavaScript plugin to analyze. Our code has 50k LOC and For every PR sonar takes 100 minutes.

Example log from sonar:

06:37:18.631 INFO: Load/download plugins (done) | time=4169602ms
06:37:19.134 DEBUG: Plugins:
06:37:19.134 DEBUG:   * IaC Code Quality and Security 1.7.0.2012 (iac)
06:37:19.134 DEBUG:   * PL/SQL Code Quality and Security 3.7.0.4372 (plsql)
06:37:19.134 DEBUG:   * Scala Code Quality and Security 1.9.0.3429 (sonarscala)
06:37:19.134 DEBUG:   * C# Code Quality and Security 8.40.0.48530 (csharp)
06:37:19.135 DEBUG:   * Vulnerability Analysis 9.5.1.16463 (security)
06:37:19.135 DEBUG:   * Java Code Quality and Security 7.12.0.29739 (java)
06:37:19.135 DEBUG:   * HTML Code Quality and Security 3.6.0.3106 (web)
06:37:19.136 DEBUG:   * Flex Code Quality and Security 2.7.0.2865 (flex)
06:37:19.136 DEBUG:   * XML Code Quality and Security 2.5.0.3376 (xml)
06:37:19.136 DEBUG:   * Text file Code Quality and Security 1.1.0.282 (text)
06:37:19.137 DEBUG:   * VB.NET Code Quality and Security 8.40.0.48530 (vbnet)
06:37:19.137 DEBUG:   * Swift Code Quality and Security 4.6.0.5406 (swift)
06:37:19.137 DEBUG:   * CFamily Code Quality and Security 6.34.0.48468 (cpp)
06:37:19.137 DEBUG:   * Python Code Quality and Security 3.13.0.9611 (python)
06:37:19.137 DEBUG:   * Dataflow Bug Detection 1.2.2.1720 (dbd)
06:37:19.138 DEBUG:   * Go Code Quality and Security 1.9.0.3429 (go)
06:37:19.138 DEBUG:   * JaCoCo 1.1.1.1157 (jacoco)
06:37:19.138 DEBUG:   * Kotlin Code Quality and Security 2.9.0.1147 (kotlin)
06:37:19.139 DEBUG:   * Dataflow Bug Detection Rules for Java 1.2.2.1720 (dbdjavafrontend)
06:37:19.139 DEBUG:   * T-SQL Code Quality and Security 1.6.0.4844 (tsql)
06:37:19.139 DEBUG:   * JavaScript/TypeScript/CSS Code Quality and Security 9.3.0.18033 (javascript)
06:37:19.139 DEBUG:   * Ruby Code Quality and Security 1.9.0.3429 (ruby)
06:37:19.140 DEBUG:   * Vulnerability Rules for C# 9.5.1.16463 (securitycsharpfrontend)
06:37:19.140 DEBUG:   * Vulnerability Rules for Java 9.5.1.16463 (securityjavafrontend)
06:37:19.140 DEBUG:   * Vulnerability Rules for JS 9.5.1.16463 (securityjsfrontend)
06:37:19.140 DEBUG:   * Vulnerability Rules for Python 9.5.1.16463 (securitypythonfrontend)
06:37:19.140 DEBUG:   * PHP Code Quality and Security 3.23.1.8766 (php)
06:37:19.141 DEBUG:   * ABAP Code Quality and Security 3.10.0.3628 (abap)
06:37:19.141 DEBUG:   * Configuration detection fot Code Quality and Security 1.2.0.267 (config)
06:37:19.141 DEBUG:   * Vulnerability Rules for PHP 9.5.1.16463 (securityphpfrontend)

Hi,

Could you provide the full analysis log, please?

 
Ann

log.txt (619.7 KB)

Hi Ann,

Attached a log file. Please let me know once you’ve accessed it. I will delete it later for security reasons.

Hi,

In general, please redact logs as necessary. Planning to delete them isn’t a great strategy because

  • multiple folks may need to look at them in order to figure out what’s going on
  • others may have downloaded them in the meantime.

So here’s what I see:

07:02:51.795 DEBUG: Download plugin 'abap' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache11888252211239544152.tmp'
07:02:52.376 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=abap&acceptCompressions=pack200 | time=581ms
07:03:22.296 DEBUG: Download plugin 'csharp' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache6857642055859446245.tmp'
07:03:22.879 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=csharp&acceptCompressions=pack200 | time=582ms
07:04:58.535 DEBUG: Download plugin 'cpp' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache9036815492318569160.tmp'
07:04:59.114 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=cpp&acceptCompressions=pack200 | time=577ms
07:20:47.547 DEBUG: Download plugin 'config' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache11907440712011240776.tmp'
07:20:48.137 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=config&acceptCompressions=pack200 | time=589ms
07:20:48.411 DEBUG: Download plugin 'dbd' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache830467713220073461.tmp'
07:20:48.999 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=dbd&acceptCompressions=pack200 | time=587ms
07:21:19.524 DEBUG: Download plugin 'dbdjavafrontend' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache2317550623580279289.tmp'
07:21:20.117 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=dbdjavafrontend&acceptCompressions=pack200 | time=592ms
07:21:59.565 DEBUG: Download plugin 'flex' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache222075443239006892.tmp'
07:22:00.147 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=flex&acceptCompressions=pack200 | time=582ms
07:22:25.334 DEBUG: Download plugin 'go' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache14156780315205978986.tmp'
07:22:25.928 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=go&acceptCompressions=pack200 | time=594ms
07:24:14.543 DEBUG: Download plugin 'web' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache9271133287620042715.tmp'
07:24:15.194 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=web&acceptCompressions=pack200 | time=650ms
07:24:29.856 DEBUG: Download plugin 'iac' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache10746565511143014761.tmp'
07:24:30.474 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=iac&acceptCompressions=pack200 | time=618ms
07:24:50.433 DEBUG: Download plugin 'jacoco' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache11064398298359449373.tmp'
07:24:51.102 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=jacoco&acceptCompressions=pack200 | time=669ms
07:24:51.378 DEBUG: Download plugin 'java' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache13787424018589285941.tmp'
07:24:51.990 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=java&acceptCompressions=pack200 | time=611ms
07:30:09.994 DEBUG: Download plugin 'javascript' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache3122646807294488168.tmp'
07:30:10.579 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=javascript&acceptCompressions=pack200 | time=585ms
07:36:34.662 DEBUG: Download plugin 'kotlin' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache6262153098676614524.tmp'
07:36:35.253 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=kotlin&acceptCompressions=pack200 | time=590ms
07:47:19.227 DEBUG: Download plugin 'php' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache12802669747790969416.tmp'
07:47:19.806 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=php&acceptCompressions=pack200 | time=578ms
07:48:59.652 DEBUG: Download plugin 'plsql' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache8260553626170724950.tmp'
07:49:00.232 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=plsql&acceptCompressions=pack200 | time=580ms
07:49:42.362 DEBUG: Download plugin 'python' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache4294737769230704061.tmp'
07:49:42.952 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=python&acceptCompressions=pack200 | time=590ms
07:51:59.615 DEBUG: Download plugin 'ruby' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache13575753049606940295.tmp'
07:52:00.200 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=ruby&acceptCompressions=pack200 | time=584ms
07:56:23.413 DEBUG: Download plugin 'sonarscala' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache9876941146486545444.tmp'
07:56:23.999 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=sonarscala&acceptCompressions=pack200 | time=585ms
08:00:56.324 DEBUG: Download plugin 'swift' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache2033365630477662513.tmp'
08:00:56.918 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=swift&acceptCompressions=pack200 | time=593ms
08:01:23.109 DEBUG: Download plugin 'tsql' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache3145578852732308979.tmp'
08:01:23.692 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=tsql&acceptCompressions=pack200 | time=582ms
08:02:03.352 DEBUG: Download plugin 'text' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache15887750475707504814.tmp'
08:02:03.945 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=text&acceptCompressions=pack200 | time=593ms
08:02:04.877 DEBUG: Download plugin 'vbnet' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache1631100186802704754.tmp'
08:02:05.482 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=vbnet&acceptCompressions=pack200 | time=604ms
08:03:22.913 DEBUG: Download plugin 'security' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache4944404148469588733.tmp'
08:03:23.531 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=security&acceptCompressions=pack200 | time=617ms
08:06:21.323 DEBUG: Download plugin 'securitycsharpfrontend' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache11685936911326777036.tmp'
08:06:21.921 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=securitycsharpfrontend&acceptCompressions=pack200 | time=598ms
08:06:39.624 DEBUG: Download plugin 'securityjsfrontend' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache7356965238318957167.tmp'
08:06:40.330 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=securityjsfrontend&acceptCompressions=pack200 | time=705ms
08:06:48.870 DEBUG: Download plugin 'securityjavafrontend' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache16525232223308327047.tmp'
08:06:49.450 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=securityjavafrontend&acceptCompressions=pack200 | time=580ms
08:09:33.483 DEBUG: Download plugin 'securityphpfrontend' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache4765486181422369709.tmp'
08:09:34.066 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=securityphpfrontend&acceptCompressions=pack200 | time=583ms
08:10:23.332 DEBUG: Download plugin 'securitypythonfrontend' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache9456846831075702570.tmp'
08:10:23.917 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=securitypythonfrontend&acceptCompressions=pack200 | time=584ms
08:11:12.858 DEBUG: Download plugin 'xml' to '/builds/cardiac-svcs/sv-qa-automation/expertautomation/.sonar/_tmp/fileCache15133005232917748385.tmp'
08:11:13.449 DEBUG: GET 200 http://10.20.71.26:9000/api/plugins/download?plugin=xml&acceptCompressions=pack200 | time=591ms

Yes, the whole thing takes from 7:02 to 8:11, but each individual download is sub-second:

07:02:51.795 : Download plugin 'abap' 
07:02:52.376 : GET 200 ...abap... time=581ms
07:03:22.296 : Download plugin 'csharp' 
07:03:22.879 : GET 200 ...csharp... time=582ms
07:04:58.535 : Download plugin 'cpp' 
07:04:59.114 : GET 200 ...cpp... time=577ms
07:20:47.547 : Download plugin 'config' 
07:20:48.137 : GET 200 ...config... time=589ms
07:20:48.411 : Download plugin 'dbd' 
07:20:48.999 : GET 200 ...dbd... time=587ms
07:21:19.524 : Download plugin 'dbdjavafrontend' 
07:21:20.117 : GET 200 ...dbdjavafrontend... time=592ms
07:21:59.565 : Download plugin 'flex' 
07:22:00.147 : GET 200 ...flex... time=582ms
07:22:25.334 : Download plugin 'go' 
07:22:25.928 : GET 200 ...go... time=594ms
07:24:14.543 : Download plugin 'web' 
07:24:15.194 : GET 200 ...web... time=650ms
07:24:29.856 : Download plugin 'iac' 
07:24:30.474 : GET 200 ...iac... time=618ms
07:24:50.433 : Download plugin 'jacoco' 
07:24:51.102 : GET 200 ...jacoco... time=669ms
07:24:51.378 : Download plugin 'java' 
07:24:51.990 : GET 200 ...java... time=611ms
07:30:09.994 : Download plugin 'javascript' 
07:30:10.579 : GET 200 ...javascript... time=585ms
07:36:34.662 : Download plugin 'kotlin' 
07:36:35.253 : GET 200 ...kotlin... time=590ms
07:47:19.227 : Download plugin 'php' 
07:47:19.806 : GET 200 ...php... time=578ms
07:48:59.652 : Download plugin 'plsql' 
07:49:00.232 : GET 200 ...plsql... time=580ms
07:49:42.362 : Download plugin 'python' 
07:49:42.952 : GET 200 ...python... time=590ms
07:51:59.615 : Download plugin 'ruby' 
07:52:00.200 : GET 200 ...ruby... time=584ms
07:56:23.413 : Download plugin 'sonarscala' 
07:56:23.999 : GET 200 ...sonarscala... time=585ms
08:00:56.324 : Download plugin 'swift' 
08:00:56.918 : GET 200 ...swift... time=593ms
08:01:23.109 : Download plugin 'tsql' 
08:01:23.692 : GET 200 ...tsql... time=582ms
08:02:03.352 : Download plugin 'text' 
08:02:03.945 : GET 200 ...text... time=593ms
08:02:04.877 : Download plugin 'vbnet' 
08:02:05.482 : GET 200 ...vbnet... time=604ms
08:03:22.913 : Download plugin 'security' 
08:03:23.531 : GET 200 ...security... time=617ms
08:06:21.323 : Download plugin 'securitycsharpfrontend' 
08:06:21.921 : GET 200 ...securitycsharpfrontend... time=598ms
08:06:39.624 : Download plugin 'securityjsfrontend' 
08:06:40.330 : GET 200 ...securityjsfrontend... time=705ms
08:06:48.870 : Download plugin 'securityjavafrontend' 
08:06:49.450 : GET 200 ...securityjavafrontend... time=580ms
08:09:33.483 : Download plugin 'securityphpfrontend' 
08:09:34.066 : GET 200 ...securityphpfrontend... time=583ms
08:10:23.332 : Download plugin 'securitypythonfrontend' 
08:10:23.917 : GET 200 ...securitypythonfrontend... time=584ms
08:11:12.858 : Download plugin 'xml' 
08:11:13.449 : GET 200 ...xml... time=591ms

The lags come between the requests:

07:04:59.114 : GET 200 ...cpp... time=577ms
07:20:47.547 : Download plugin 'config' 
...
07:24:51.990 : GET 200 ...java... time=611ms
07:30:09.994 : Download plugin 'javascript' 
07:30:10.579 : GET 200 ...javascript... time=585ms
07:36:34.662 : Download plugin 'kotlin' 
07:36:35.253 : GET 200 ...kotlin... time=590ms
07:47:19.227 : Download plugin 'php' 
&etc...

Since this is not behavior we typically see I have to ask what else is going on on your build agent. It looks to me like the analysis process just isn’t getting enough processing time to complete in a timely manner.

 
HTH,
Ann