Socket timeouts using maven sonar plugin

SonarQube
,
1

I am having socket timeouts in Enterprise Edition Version 8.9 (build 43852) and
org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.1.1688.

It seems whatever is used as default (30sec?) is not enough running sonar maven from Azure Devops against external sonarqube server.
Increasing sonar.ws.timeout to high values did not help.
I may be due its the write not the read timeout.

mvn
verify
org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.1.1688:sonar
-Pcoverage
–batch-mode
–file ${AGENT_BUILDDIRECTORY}/source/pom.xml
–settings ${AGENT_BUILDDIRECTORY}/pipelinescripts/pipelines/ci/maven/settings.xml
-Dmaven.repo.local=${MAVEN_CACHE}
-Dsonar.projectKey=${APP_KEY^^}
-Dsonar.projectName=${APP_NAME}
-Dsonar.projectVersion=${APP_VERSION}
-Dsonar.login=${SONAR_TOKEN}
-Dsonar.host.url=${SONAR_URL}
-Dsonar.branch.name=${BUILD_SOURCEBRANCHNAME}
-Djavax.net.ssl.keyStore=${SONAR_CERT_PATH}
-Djavax.net.ssl.keyStoreType=PKCS12
-Djavax.net.ssl.keyStorePassword=${SONAR_CERT_PASSPHRASE}
-Dsonar.qualitygate.wait=true
-Dsonar.ws.timeout=900
-Dmaven.test.failure.ignore=true
-Dspring.profiles.active=ci
-Ddependency-check.skip=true

See the logs

Can we publish some metrics like chunk sizes of transfered packages or any other reasons which would explain why we get the sockets timeout. I can not imagine that we generally have an azure cloud network isuse on the link to our sonarqube. Doing hundreds of analysis per day successful. Just two project which have a rather big testbed with some loops seems to challenge the sonar client protocol design into the edge. More or less 50% of the runs for this projects it runs and the other 50% it does not …

2

The overall chain is as follows: SQ maven plugin → sonarqube.ws.client class → OkHttpClient → SQ server
The OkHttpClient library support both readTimeOut and writeTimeOut. Unfortunately, today the sonarqube.ws.client implementation only support configuration for readTimeOut, but the writeTimeOut is hardcoded - it was 10 sec before and recently they increased it up to 30 sec, which is still not always enough according to user reports in jira and this community forum. The solution is to make writeTimeOut also configurable: I have implemented the solution and raised pull request 3252, but it was declined with explanation:
We try to avoid adding more configuration. We think 30s is a good compromise.

I think now is a time to review my implementation again.
Regards.

3

I still don’t get how the changes of the server binary will make it into the maven sonar plugin i need to tweak
If its a client fix, then i would expect i need to compile a new org.sonarsource.scanner.maven:sonar-maven-plugin (currently i use :3.6.1.1688 version)
see GitHub - SonarSource/sonar-scanner-maven: SonarQube Scanner for Maven

Seems like i need to release a custom

<dependency>
<groupId>org.sonarsource.scanner.api</groupId>
<artifactId>sonar-scanner-api</artifactId>
</dependency>

and depend on it

On what factors is the write write timeout for submission report? depending?
Why do we have that only one some projects?
All projects use the same pool of vms client to server, so it seems not to depend (only) on the latency or bandwidth. But actually the testbed in this two projects are big.
If sonar clients defaults are limiting the number of tests i can transmit in one project that would be a rather bad experience and an antipattern for qs.

4

Having the full logs with debug enabled would help see what is the effective timeout being used with the timestamps.
The response of the webservice in the logs (new_code_periods) is very small, so surely it’s not related to the data size.
Do you really think it’s acceptable to take more than 30s to get a simple WS response? I don’t think you guys are tackling the right problem here.

5

That log is huge - what exactly do you need (from-to) or whcih key words i need to search for?

6

I just wanted to confirm is it took 30seconds to time out by comparing the timestamp of the previous log entry (the request) with the timestamp of the error that you posted.
If it is taking 30seconds, I’d say there’s a network problem.

7

Everything is happening within two minutes, isn’it?
see yourself gist:f0100e5fc1344ae6e7ef8ad74f84440d · GitHub

I can not cure Ms Azure networks, if so - neither i in charge of the sonar backend. Just a user.
I can’t change this facts easily and i don’t care if sonar client retries until to finally get it done. That is kind of a resilient pattern i would expect.
I would rather allow the client to take longer (which happens every 3rd time) by letting the user allow to configure what is accepted instead of trying to find the “worlds default”.

8

It actually looks like it waited 15 minutes before timing out.

2021-06-21T07:07:36.9277934Z [INFO] 07:07:36.695 Load New Code definition
2021-06-21T07:22:36.7104206Z

9

Load New Code definition - what is happening there?
Is it waiting for the server? Is it possible not of transmission but server (maybe sql) response latency?
Why 15 minutes? Why not more or less? What 15 min timeout is this?

10

It’s getting https://mysonar.com/sonar/api/new_code_periods/show.protobuf?project=TTR&branch=master.
This is a simple webservice, I doubt there is any delay on the server side. You can see the response and check how long it takes from your machine by opening it on your browser or using a tool like curl.

I don’t know why it’s taking 15min. It could be that the connection is extremely slow, keeping it alive for 15min until is finally times out because no more data is received.

11

“errors”:[{“msg”:“Insufficient privileges”}]}
15 minutes grant timeout …? … why is sometimes working sometimes not?

12

Ok … i have used the wrong user for my replay test… now as i used the CI user i was well able to download content (via the browser)
So the question is again …why does it rest there for 15 mins from time to time …
I am running the sonar maven build on azure devops hosted services.

13

Returns

image

14

That’s the expected reply (protobuf format). I’m assuming that the reply didn’t have any delay.

I think there’s a network problem between azure and SonarQube but I can’t help with that.
Maybe it always takes very long for the scanner to consumer SonarQube’s webservices, even when it succeeds, but sometimes not long enough to time out.

15

I think there’s a network problem between azure and SonarQube but I can’t help with that.

I doubt this in general as we run tons of other jobs on this pool, and only sonar with this dedicated project seems to have sporadically an issue. Is it possible that there is a flaky response latency at the sonarqube server under certain situation. especially for this project i know i have linked additional js, html, cs, typescript additional profiles. Which server logs i could ask the admin to check on their side. What shall they search for?
Which timeouts of the tcp stack and which bon sonar/sonarqube might play a role and needs to fulfill certain relations and which one?

16

Hi,
sorry for the late reply. You can check the server side logs in {sonarqube_home}/logs/web.log. If you want higher verbosity, see here our to configure it: Server Logs & System Info | SonarQube Docs.

As I said before, you can replay yourself the webservices requested by the scanner and check what is the delay.