SonarCloud analysis is using the docker image sonarsource/sonar-scanner-cli in drone CI.
I am seeing this happen in overnight CI and still happening now:
The CI pipeline hangs for more than 1 hour and then gives up when trying to get the needed plugin(s).
All the other CI pipelines and steps are working fine.
Is there a known problem with sonarsource/sonar-scanner-cli that started in the last day?
INFO: Load plugins index
INFO: Load plugins index (done) | time=715ms
INFO: Load/download plugins (done) | time=5450256ms
That worked but took more than an hour to fetch the plugins. Is there something throttling the download, or?
java.lang.IllegalStateException: Fail to download plugin [cpp] into /opt/sonar-scanner/.sonar/_tmp/fileCache16221404849936660163.tmp
Happened after about 30 minutes. Happened in the last hour. How can I debug this?
Loading the plugins took 24 seconds:
Load/download plugins (done) | time=24712ms
Other nightly CI jobs were similar and all were successful.
That is much better than taking more than an hour! Something has been fixed since yesterday. If anyone knows what the problem/event was yesterday, please comment. It would be nice to understand for next time.
Hi @phil-davis !
We didn’t made any change regarding the download of plugins. This seems to be related to your network/infrastructure. Could you check first if you had some changes on your side, please? I tested locally here and didn’t found any issue.
@Alexandre_Holzhey This issue is pretty common it seems:
And the response is always “network problem” regarding timeout. But the timeouts listed in all these issues are reasonable. The plugins being downloaded are <10MB, a connection timeout of 60 seconds should be more than enough. In fact, the shorter timeout is helping my team waste less time when this error occurs. It seems that the sonarqube server “forgets” to close the connection somehow.
@Alexandre_Holzhey can you provide any clarification into what the network flow is to download plugins?
- Sonarqube (9.2)
- Deployed as StatefulSet on GKE (using sonarqube provided helm chart, no modifications)
- Serving traffic through GCP HTTPS LB (60 second timeout per-request)
We frequently see builds fail at the 60-second mark. Are all plugins being downloaded via a single long-running request? Our builds run in unique docker instances, so it is not possible to preserve the plugins across builds.
INFO: EXECUTION FAILURE
INFO: Total time: 1:06.773s
INFO: Final Memory: 4M/20M
ERROR: Error during SonarScanner execution
java.lang.IllegalStateException: Fail to download plugin [ruby] into /root/.sonar/_tmp/fileCache17047521401284787132.tmp
The plugin which fails changes every time, so its somewhat random. Also, we don’t use ruby, nor most of the plugins which are failing. Is it possible to disable the download of certain plugins to the scanner?
@jasondamour the scanner needs to download the plugins to proceed with the analysis. We want to improve this for 2022, but meanwhile all plugins are required to be downloaded. The server where the scanner is executed must allow network connection to download them (one request per plugin). Here is an example of simple CURL request to download the JAVA plugin (which is bigger than RUBY that is in your log output):
curl --request GET \
--output /dev/null \
--url https://sonarcloud.io/api/plugins/download \
--header 'Content-Type: multipart/form-data' \
In my machine the output is:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 16.6M 0 16.6M 100 145 11.5M 100 0:00:01 0:00:01 --:--:-- 11.5M
So as you can see usually is pretty fast, but some times we could have a higher load on our servers and because of that i recommend you to increase the timeout you currently have to 180 seconds.
But the log output you provided shows to me that there is a possibility of a file system issue (permissions, available space, etc). This specific message is triggered when:
if destination is a directory
if destination cannot be written
if destination needs creating but can't be
if an IO error occurs during copying (here a timeout could be the reason)
Could you enable DEBUG mode to get more information?
Thanks for the suggestion! I will apply the increased timeout, but I’m concerned about the server performance. This is a self hosted instance, so I’d like for no requests to take longer than 60 seconds. Which I assume from the 3 servers JVMs, this request is handled by the Web process? Would increasing the Memory/CPU of the Web process reduce the duration? Is there a limit of max connections?
I also see these default options in the system administration section for the web process. Do these need to be increased as well?
Ok, yesterday I went all-out in increasing resources:
LB Timeout: 60s → 180s
Web JVM Xmx: 742m → 3072m
CE JVM Xmx: 742m → 3072m
CPU Request/Limit (Kubernetes): 1vcpu → 3vcpu
And I still see the same error. Here are logs i enabled on our LoadBalancer (which now alerts me when a sonarqube request exceeds 2 minutes):
GET | 5.82 MB | 180.1s | ScannerNpm/2.8.1 | http://sonarqube.xxxx.net/api/plugins/download?plugin=go&acceptCompressions=pack200
GET | 2.33 MB | 180.2s | ScannerGradle/2.8-SNAPSHOT/6.9.1 | http://sonarqube.xxxx.net/api/plugins/download?plugin=securitypythonfrontend
GET | 5.82 MB | 180.1s | ScannerGradle/3.0-SNAPSHOT/6.5 | http://sonarqube.xxxx.net/api/plugins/download?plugin=go
And on and on. Happens usually once an hour, spikes up to 10x an hour.
I have also now added a Prometheus adapter sidecar to the sonarqube server, so I have all 865 metrics in a time-series database. If any of those data points help point to whats happening in the JVM at the point in time then the request fails, please let me know what to look for and I’ll provide the data.
Here are the metrics I’ve inspected for correlation so far:
Hello @jasondamour , thanks for trying to change the configuration!
I saw here and you mentioned that you are using sonarqube:
This thread is about SonarCloud and it have a totally different approach to download the plugins. Could you confirm that we are talking about SonarQube and if this is the case can i ask you to create another thread for this topic ?
I’ve have the same issue @phil-davis how to access this ticket " SonarQube failed to download plugin" @jasondamour @Alexandre_Gigleux
[DEBUG] 01:41:09.799 Download plugin ‘kotlin’ to ‘/home/runner/.sonar/_tmp/fileCache4435001357620950364.tmp’
3289[DEBUG] 01:41:09.851 GET 200 https://sonarqubeinternal.xxxxxxx.org/api/plugins/download?plugin=kotlin&acceptCompressions=pack200 | time=52ms
3390[DEBUG] 01:41:30.910 Download plugin ‘php’ to ‘/home/runner/.sonar/_tmp/fileCache4503266867632749061.tmp’
Error: Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:18.104.22.1684:sonar (default-cli) on project jb-hello-world-maven: Fail to download plugin [php] into /home/runner/.sonar/_tmp/fileCache4503266867632749061.tmp: timeout: Socket closed → [Help 1]
@raosri1992 I’m not sure why that link isn’t working, heres the post:
However, I think I found the issue and the solution. Please refer to: