Occasional timeouts when running analysis


(Daniel Scott) #1

Hi,

I have SonarQube 6.7.4 (http://sonar-pkg.sourceforge.net/ - as an aside. Does anyone know when new RPMs will be made available, or are they deprecated) running on an AWS EC2 instance behind an ELB. The ELB is also reverse-proxied behind an apache 2.4 server.

Inconsistently, but frequently, starting last week, we have been seeing timeouts from our build agents (Using both maven and gradle - various plugin versions).

The agents are erroring with

timeout: Socket closed

And I’m seeing some exceptions in the /opt/sonar/logs/web.log file:

2019.01.18 13:04:24 WARN  web[AWg4fhAucTKFyaqAA47i][o.s.s.w.ServletRequest] Can't read file part for parameter report
java.io.IOException: org.apache.tomcat.util.http.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. Unexpected EOF read on the socket

The server metrics look good - very little CPU load, and the heap usage seems OK (I’m using default settings of 512MB and it’s showing about 50% used.

Occasionally, after I see a build failure, I try and login to the web UI, and it’s very ‘sluggish’ - slow to load my list of projects, etc. But after a while (several minutes) it recovers, and the UI becomes normal again.

This install has been working fine for a few years now. I was using 6.7.2 until I started seeing this issue last week, after which, I upgraded to 6.7.4.

Does anyone have any ideas?

Thanks,

Dan


(G Ann Campbell) #2

Hi Dan,

Your error message is saying that the data feed to SonarQube ended abruptly, so this is mostly likely somewhere between your analysis machine and your SQ instance.

 
Ann


(Daniel Scott) #3

Yes, my AWS ELB has a 60s timeout - I suppose my question is more accurately asked as “Why do some analyses suddenly take far longer (The vast majority complete within milliseconds) than others?”


(G Ann Campbell) #4

Hi,

This is actually two different questions. One is about why the analysis report upload is sometimes curtailed, and the other is about how long it takes to produce the analysis report.

Briefly, the scanner contacts the server via web services at the beginning of analysis to request data to set up analysis (which languages, rules, …), then analysis runs independently of and unconnected to the server. At the end of that process, an analysis report is compiled and uploaded, again via web services, to the server.

That last piece is what you initially reported trouble with. If you’re additionally wondering about a variable analysis time, that would probably be best in a separate thread, although I would start by looking at whether the results per project were consistent, and if not I’d take a look at the build/analysis infrastructure.

 
HTH,
Ann


(Daniel Scott) #5

Hi,

Thanks for the information - I think I’m probably using the wrong terminology. As far as I can see, there is one problem, which is that occasionally (randomly as far as I can tell) the sonar-maven-plugin ‘hangs’ during connection to the server. The hang occurs immediately after the client logs Load global settings. In all occasions when this completed successfully, it has taken <300ms. The ‘hanging’ processes are killed after 60s due to the AWS ELB timeout.

Thanks,

Dan


(G Ann Campbell) #6

Hi Dan,

And is there an error in the analysis logs for these failing jobs?

 
Ann


(Daniel Scott) #7

The only error logs that I see are those I provided at the start. The server side web.log has the IOFileUploadException, and the client receives the timeout. There are no errors in ce.log, es.log or sonar.log.


(G Ann Campbell) #8

Hi Dan,

Both ends of analysis say that the upload process died. I would have to look at whatever’s in the middle, especially since this started happening abruptly last week after apparently no changes to either server or analysis, right?

 
Ann


(Daniel Scott) #9

I’m not certain it’s an ‘analysis’ problem. The network connection seems to hang before it even gets that far. It never gets to Load global settings (done), implying that it’s trying to download config from the server. Is that correct?

Yes, this started happening abruptly last week - no changes on either server or analysis side.