Plugin download download fails

@ganncamp This issue has never been resolved for us. Something about the way specifically sonarqube sends responses to our Load Balancer is causing this failure. None of the other of 100s of servers behind our same LB face this issue. Please schedule resources to investigate this, I can reproduce regularly and provide any logs.

The end user error is always some variant of the following:
java.lang.IllegalStateException: Fail to download plugin [iac] into /opt/sonar-scanner/.sonar/_tmp/fileCache2582033732133438722.tmp

Hi @jasondamour,

Why is your single SonarQube instance behind a load balancer?

 
Ann

It really shouldn’t really matter why we did it some particular way. We have a single load balancer serving as the single entrypoint for several applications. This same loadbalancer handles 50k+ requests/day for our Maven artifact server with no issues. It is only Sonarqube which causes this issue. Here are the load balancer logs. The backend is taking several minutes per request for plugin downloads, so it appears to be a Sonarqube Server issue, not the network layer.

I can, and have already, provide my full server configuration details:

  env:
  - name: SONAR_WEB_HTTP_MAXTHREADS
    value: "200"
  - name: SONAR_WEB_HTTP_ACCEPTCOUNT
    value: "50"
  - name: SONAR_WEB_JAVAADDITIONALOPTS
    value: "-Xms2048m -Xmx2048m"
  - name: SONAR_CE_JAVAADDITIONALOPTS
    value: "-Xms2048m -Xmx2048m"
  - name: SONAR_SEARCH_JAVAADDITIONALOPTS
    value: "-Xms1024m -Xmx1024m"

  # Web Xmx + CE Xmx + Search Xmx = 2048 + 2048x2 + 1024 = 7Gb
  resources:
    requests:
      cpu: 2
      memory: 7Gi
    limits:
      cpu: 4
      memory: 7Gi

I also have months of data on all 527 custom prometheus metrics from sonarqube which could be used to debug. I have meticulously inspected each one already and found no indicator of the issue, but will provide full access if it helps debug this.

We’re an enterprise customer, and have escalated this a couple times to our TAM, who has redirected us here since we will not pay $20k for a support package for this one simple issue. So I’ll continue to escalate here until it’s resolved

I’ve tried debugging this myself for months. I even opened a PR to sonarqube which was accepted and released, which didn’t fix the issue: Need ability to configure Tomcat KeepAliveTimeout - #3 by jasondamour

Adding on. The GCP load balancer logs this for all the failed requests:
statusDetails: “backend_connection_closed_after_partial_response_sent”

@ganncamp What else can I provide to aid identifying the issue here?

Hi,

I’ve flagged this for more expert attention. Hopefully, they’ll be along soon.

 
Ann

Hey @jasondamour ,

Have you tried to use the same endpoints (or just a single one) the same way scanners do on the same machine as SonarQube is installed, so that you could check downloads time by passing the load balancer?

For instance: http://your-sq-instance/api/plugins/download?plugin=javascript&acceptCompressions=pack200

I would like to confirm if that’s a server-side or client-side issue.

Sure @jacek.poreda, I will test it this week

Hi all, are there any news regarding this issue? I have similar issue, but I don’t have load balancer, just reverse proxy. Downloading manually files that my pipeline could not is really fast. And actually downloading of other plugins is also below 20ms each in pipeline, it just stops at random one