SonarCloud analysis is using the docker image sonarsource/sonar-scanner-cli in drone CI.
I am seeing this happen in overnight CI and still happening now: https://drone.owncloud.com/owncloud/core/33622/35/8 java.lang.IllegalStateException: Fail to download plugin [javascript] into /opt/sonar-scanner/.sonar/_tmp/fileCache17550887843177845817.tmp
The CI pipeline hangs for more than 1 hour and then gives up when trying to get the needed plugin(s).
All the other CI pipelines and steps are working fine.
Is there a known problem with sonarsource/sonar-scanner-cli that started in the last day?
Other nightly CI jobs were similar and all were successful.
That is much better than taking more than an hour! Something has been fixed since yesterday. If anyone knows what the problem/event was yesterday, please comment. It would be nice to understand for next time.
We didn’t made any change regarding the download of plugins. This seems to be related to your network/infrastructure. Could you check first if you had some changes on your side, please? I tested locally here and didn’t found any issue.
And the response is always “network problem” regarding timeout. But the timeouts listed in all these issues are reasonable. The plugins being downloaded are <10MB, a connection timeout of 60 seconds should be more than enough. In fact, the shorter timeout is helping my team waste less time when this error occurs. It seems that the sonarqube server “forgets” to close the connection somehow.
@Alexandre_Holzhey can you provide any clarification into what the network flow is to download plugins?
Our setup:
Sonarqube (9.2)
Deployed as StatefulSet on GKE (using sonarqube provided helm chart, no modifications)
Serving traffic through GCP HTTPS LB (60 second timeout per-request)
We frequently see builds fail at the 60-second mark. Are all plugins being downloaded via a single long-running request? Our builds run in unique docker instances, so it is not possible to preserve the plugins across builds.
INFO: ------------------------------------------------------------------------
INFO: EXECUTION FAILURE
INFO: ------------------------------------------------------------------------
INFO: Total time: 1:06.773s
INFO: Final Memory: 4M/20M
INFO: ------------------------------------------------------------------------
ERROR: Error during SonarScanner execution
java.lang.IllegalStateException: Fail to download plugin [ruby] into /root/.sonar/_tmp/fileCache17047521401284787132.tmp
The plugin which fails changes every time, so its somewhat random. Also, we don’t use ruby, nor most of the plugins which are failing. Is it possible to disable the download of certain plugins to the scanner?
@jasondamour the scanner needs to download the plugins to proceed with the analysis. We want to improve this for 2022, but meanwhile all plugins are required to be downloaded. The server where the scanner is executed must allow network connection to download them (one request per plugin). Here is an example of simple CURL request to download the JAVA plugin (which is bigger than RUBY that is in your log output):
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 16.6M 0 16.6M 100 145 11.5M 100 0:00:01 0:00:01 --:--:-- 11.5M
So as you can see usually is pretty fast, but some times we could have a higher load on our servers and because of that i recommend you to increase the timeout you currently have to 180 seconds.
But the log output you provided shows to me that there is a possibility of a file system issue (permissions, available space, etc). This specific message is triggered when:
if destination is a directory
if destination cannot be written
if destination needs creating but can't be
if an IO error occurs during copying (here a timeout could be the reason)
Could you enable DEBUG mode to get more information?
Thanks for the suggestion! I will apply the increased timeout, but I’m concerned about the server performance. This is a self hosted instance, so I’d like for no requests to take longer than 60 seconds. Which I assume from the 3 servers JVMs, this request is handled by the Web process? Would increasing the Memory/CPU of the Web process reduce the duration? Is there a limit of max connections?
Ok, yesterday I went all-out in increasing resources:
LB Timeout: 60s → 180s
Web JVM Xmx: 742m → 3072m
CE JVM Xmx: 742m → 3072m
CPU Request/Limit (Kubernetes): 1vcpu → 3vcpu
And I still see the same error. Here are logs i enabled on our LoadBalancer (which now alerts me when a sonarqube request exceeds 2 minutes):
GET | 5.82 MB | 180.1s | ScannerNpm/2.8.1 | http://sonarqube.xxxx.net/api/plugins/download?plugin=go&acceptCompressions=pack200
GET | 2.33 MB | 180.2s | ScannerGradle/2.8-SNAPSHOT/6.9.1 | http://sonarqube.xxxx.net/api/plugins/download?plugin=securitypythonfrontend
GET | 5.82 MB | 180.1s | ScannerGradle/3.0-SNAPSHOT/6.5 | http://sonarqube.xxxx.net/api/plugins/download?plugin=go
And on and on. Happens usually once an hour, spikes up to 10x an hour.
I have also now added a Prometheus adapter sidecar to the sonarqube server, so I have all 865 metrics in a time-series database. If any of those data points help point to whats happening in the JVM at the point in time then the request fails, please let me know what to look for and I’ll provide the data.
Here are the metrics I’ve inspected for correlation so far:
Hello @jasondamour , thanks for trying to change the configuration!
I saw here and you mentioned that you are using sonarqube:
This thread is about SonarCloud and it have a totally different approach to download the plugins. Could you confirm that we are talking about SonarQube and if this is the case can i ask you to create another thread for this topic ?
@raosri1992 I’m not sure why that link isn’t working, heres the post: https://community.sonarsource.com/t/analysis-fails-during-plugin-download/55895
However, I think I found the issue and the solution. Please refer to: https://community.sonarsource.com/t/need-ability-to-configure-tomcat-keepalivetimeout/59654