CDN for hosted SQE sonar plugins?

SonarQube
1

Case:
There is a hosted Sonarqube Enterprise instance (8.9 LTS). The clients are consuming this service from both, near-site and cloud/far networks.

We are noticing that there is a network latency/throughput problem for the off-network clients when downloading / loading plugin bundle, e.g.:

11:24:59.663 DEBUG: GET 200 https://some-url//api/plugins/installed | time=41ms
11:24:59.701 INFO: Load plugins index (done) | time=79ms
11:24:59.703 DEBUG: Download plugin 'abap' to '/root/.sonar/_tmp/fileCache14037486325711296398.tmp'
11:24:59.865 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=abap&acceptCompressions=pack200 | time=162ms
11:25:01.717 DEBUG: Download plugin 'sonarapex' to '/root/.sonar/_tmp/fileCache5786218597455464228.tmp'
11:25:01.759 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=sonarapex&acceptCompressions=pack200 | time=41ms
11:25:05.182 DEBUG: Download plugin 'csharp' to '/root/.sonar/_tmp/fileCache18026409911252945376.tmp'
11:25:05.224 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=csharp&acceptCompressions=pack200 | time=42ms
11:25:11.552 DEBUG: Download plugin 'cpp' to '/root/.sonar/_tmp/fileCache1412788673414354808.tmp'
11:25:11.599 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=cpp&acceptCompressions=pack200 | time=47ms
11:25:58.481 DEBUG: Download plugin 'cobol' to '/root/.sonar/_tmp/fileCache9995925726605737529.tmp'
11:25:58.544 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=cobol&acceptCompressions=pack200 | time=63ms
11:26:03.409 DEBUG: Download plugin 'cssfamily' to '/root/.sonar/_tmp/fileCache6336623537397404017.tmp'
11:26:03.451 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=cssfamily&acceptCompressions=pack200 | time=42ms
11:26:16.268 DEBUG: Download plugin 'crowd' to '/root/.sonar/_tmp/fileCache2930986939595476342.tmp'
11:26:16.309 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=crowd&acceptCompressions=pack200 | time=40ms
11:26:20.943 DEBUG: Download plugin 'flex' to '/root/.sonar/_tmp/fileCache2280390717408817688.tmp'
11:26:20.998 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=flex&acceptCompressions=pack200 | time=54ms
11:26:22.411 DEBUG: Download plugin 'go' to '/root/.sonar/_tmp/fileCache7902454430238026508.tmp'
11:26:22.451 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=go&acceptCompressions=pack200 | time=40ms
11:26:31.316 DEBUG: Download plugin 'web' to '/root/.sonar/_tmp/fileCache17968117194249367835.tmp'
11:26:31.370 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=web&acceptCompressions=pack200 | time=54ms
11:26:32.809 DEBUG: Download plugin 'jacoco' to '/root/.sonar/_tmp/fileCache10406288320847127330.tmp'
11:26:32.851 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=jacoco&acceptCompressions=pack200 | time=42ms
11:26:32.884 DEBUG: Download plugin 'java' to '/root/.sonar/_tmp/fileCache12311013122112491548.tmp'
11:26:32.934 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=java&acceptCompressions=pack200 | time=49ms
11:26:54.346 DEBUG: Download plugin 'javascript' to '/root/.sonar/_tmp/fileCache6888865307282151551.tmp'
11:26:54.395 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=javascript&acceptCompressions=pack200 | time=49ms
11:27:21.293 DEBUG: Download plugin 'kotlin' to '/root/.sonar/_tmp/fileCache10447975653859556469.tmp'
11:27:21.336 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=kotlin&acceptCompressions=pack200 | time=43ms
11:27:31.484 DEBUG: Download plugin 'license' to '/root/.sonar/_tmp/fileCache17107519240651051769.tmp'
11:27:31.534 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=license&acceptCompressions=pack200 | time=50ms
11:27:31.540 DEBUG: Download plugin 'php' to '/root/.sonar/_tmp/fileCache824119165240911690.tmp'
11:27:31.582 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=php&acceptCompressions=pack200 | time=42ms
11:27:37.830 DEBUG: Download plugin 'pli' to '/root/.sonar/_tmp/fileCache17220274962626273670.tmp'
11:27:37.879 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=pli&acceptCompressions=pack200 | time=49ms
11:27:38.801 DEBUG: Download plugin 'plsql' to '/root/.sonar/_tmp/fileCache16637380352739294331.tmp'
11:27:38.842 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=plsql&acceptCompressions=pack200 | time=41ms
11:27:41.229 DEBUG: Download plugin 'python' to '/root/.sonar/_tmp/fileCache14696758429430013766.tmp'
11:27:41.282 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=python&acceptCompressions=pack200 | time=53ms
11:27:46.494 DEBUG: Download plugin 'rpg' to '/root/.sonar/_tmp/fileCache17684180275419946299.tmp'
11:27:46.548 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=rpg&acceptCompressions=pack200 | time=54ms
11:27:48.809 DEBUG: Download plugin 'ruby' to '/root/.sonar/_tmp/fileCache209858304356783907.tmp'
11:27:48.851 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=ruby&acceptCompressions=pack200 | time=42ms
11:28:03.443 DEBUG: Download plugin 'sonarscala' to '/root/.sonar/_tmp/fileCache9046351620412847718.tmp'
11:28:03.486 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=sonarscala&acceptCompressions=pack200 | time=42ms
11:28:24.365 DEBUG: Download plugin 'swift' to '/root/.sonar/_tmp/fileCache14824788930469276769.tmp'
11:28:24.416 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=swift&acceptCompressions=pack200 | time=51ms
11:28:26.119 DEBUG: Download plugin 'tsql' to '/root/.sonar/_tmp/fileCache8798435168135785417.tmp'
11:28:26.162 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=tsql&acceptCompressions=pack200 | time=43ms
11:28:29.271 DEBUG: Download plugin 'vbnet' to '/root/.sonar/_tmp/fileCache11144165990221315086.tmp'
11:28:29.333 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=vbnet&acceptCompressions=pack200 | time=62ms
11:28:34.553 DEBUG: Download plugin 'vb' to '/root/.sonar/_tmp/fileCache14004356695251675635.tmp'
11:28:34.611 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=vb&acceptCompressions=pack200 | time=58ms
11:28:36.630 DEBUG: Download plugin 'security' to '/root/.sonar/_tmp/fileCache11674847675970114547.tmp'
11:28:36.680 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=security&acceptCompressions=pack200 | time=50ms
11:28:40.234 DEBUG: Download plugin 'securitycsharpfrontend' to '/root/.sonar/_tmp/fileCache9074393743044346547.tmp'
11:28:40.276 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=securitycsharpfrontend&acceptCompressions=pack200 | time=42ms
11:28:41.975 DEBUG: Download plugin 'securityjsfrontend' to '/root/.sonar/_tmp/fileCache12720224686189200036.tmp'
11:28:42.017 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=securityjsfrontend&acceptCompressions=pack200 | time=42ms
11:28:42.488 DEBUG: Download plugin 'securityjavafrontend' to '/root/.sonar/_tmp/fileCache5722509714498422701.tmp'
11:28:42.549 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=securityjavafrontend&acceptCompressions=pack200 | time=60ms
11:28:55.271 DEBUG: Download plugin 'securityphpfrontend' to '/root/.sonar/_tmp/fileCache8586376013557675959.tmp'
11:28:55.311 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=securityphpfrontend&acceptCompressions=pack200 | time=40ms
11:28:58.813 DEBUG: Download plugin 'securitypythonfrontend' to '/root/.sonar/_tmp/fileCache10131456002645437954.tmp'
11:28:58.855 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=securitypythonfrontend&acceptCompressions=pack200 | time=42ms
11:29:02.589 DEBUG: Download plugin 'xml' to '/root/.sonar/_tmp/fileCache8959204243209075451.tmp'
11:29:02.632 DEBUG: GET 200 https://some-url//api/plugins/download?plugin=xml&acceptCompressions=pack200 | time=43ms
11:29:05.529 INFO: Load/download plugins (done) | time=245908ms

I am wondering what is the recommended solution for hosted instances for such a problem (apart from network investigation of course). I see that for SonarCloud, you have implemented CDN caching. Is this also possible for hosted version? I am thinking about e.g. using AWS CDN/S3 for caching, but the question is how to configure the sonar scanner to use it?

I am aware of the caching sonar scanner files docker solution, but I’m wondering if there are other possible solutions applicable for the standalone scanner and as well Maven/Gradle/others.

2

PS. The time= in that DEBUG message is clearly misleading. It looks like time consumed to send a GET request, not to actually download the plugin (which you need to calculate from the timestamp difference).

3

Hi,

It’s not possible to use a CDN with SonarQube.

And I’m not so sure those time counts are wrong. Check the rest of your analysis log & you’ll see that the times presented are typically how long the task took. Are you sure there’s not some other process on your build agent hogging the cycles?

 
Ann

4

Hi @ganncamp,

I am sure they are wrong - I verified it by downloading the plugins manually via curl and they match exactly the timestamp differences in the log.

1 Like