SonarQube 6.2 --> 6.7.7 Maven goal: "Broken pipe (write failed)"

mirzmaster · July 3, 2019, 4:00am

Hi, all. I’m in the process of upgrading our organization from SonarQube 6.2 to 6.7.7 (LTS). Our primary repository is a Java codebase, and we employ the org.codehaus.mojo:sonar-maven-plugin:3.2 scanner plugin in the following way:

mvn sonar:sonar -Dsonar.host.url=https://sonar.example.com

The Sonar scanner has been working just fine with SonarQube 6.2, but the same job fails against the new SonarQube 6.7.7 host I’ve configured. Note: I did not upgrade SonarQube from 6.2 to 6.7.7, but rather the 6.7.7 LTS is a new deployment.

The Maven plugin throws the following exception when running the sonar:sonar goal:

[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.2:sonar (default-cli) on project foo: Fail to request https://sonar.example.com/api/ce/submit?projectKey=com.example:foo&projectName=bar: Broken pipe (Write failed) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.2:sonar (default-cli) on project foo: Fail to request https://sonar.example.com/api/ce/submit?projectKey=com.example:foo&projectName=bar
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
	at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: Fail to request https://sonar.example.com/api/ce/submit?projectKey=com.example:foo&projectName=bar
	at org.sonarsource.scanner.maven.bootstrap.ExceptionHandling.handle(ExceptionHandling.java:36)
	at org.sonarsource.scanner.maven.bootstrap.ScannerBootstrapper.execute(ScannerBootstrapper.java:81)
	at org.sonarsource.scanner.maven.SonarQubeMojo.execute(SonarQubeMojo.java:122)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
	... 20 more
Caused by: java.lang.IllegalStateException: Fail to request https://sonar.example.com/api/ce/submit?projectKey=com.example:foo&projectName=bar
	at org.sonarqube.ws.client.HttpConnector.doCall(HttpConnector.java:190)
	at org.sonarqube.ws.client.HttpConnector.post(HttpConnector.java:155)
	at org.sonarqube.ws.client.HttpConnector.call(HttpConnector.java:114)
	at org.sonar.scanner.bootstrap.ScannerWsClient.call(ScannerWsClient.java:73)
	at org.sonar.scanner.report.ReportPublisher.upload(ReportPublisher.java:189)
	at org.sonar.scanner.report.ReportPublisher.execute(ReportPublisher.java:135)
	at org.sonar.scanner.phases.PublishPhaseExecutor.publishReportJob(PublishPhaseExecutor.java:73)
	at org.sonar.scanner.phases.PublishPhaseExecutor.executeOnRoot(PublishPhaseExecutor.java:55)
	at org.sonar.scanner.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:93)
	at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:177)
	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)
	at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121)
	at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:291)
	at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:286)
	at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:264)
	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)
	at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121)
	at org.sonar.scanner.task.ScanTask.execute(ScanTask.java:48)
	at org.sonar.scanner.task.TaskContainer.doAfterStart(TaskContainer.java:84)
	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)
	at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121)
	at org.sonar.scanner.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:121)
	at org.sonar.batch.bootstrapper.Batch.doExecuteTask(Batch.java:116)
	at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:111)
	at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:62)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
	at com.sun.proxy.$Proxy23.execute(Unknown Source)
	at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:233)
	at org.sonarsource.scanner.api.EmbeddedScanner.runAnalysis(EmbeddedScanner.java:151)
	at org.sonarsource.scanner.maven.bootstrap.ScannerBootstrapper.execute(ScannerBootstrapper.java:78)
	... 23 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
	at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
	at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:876)
	at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:847)
	at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
	at okio.Okio$1.write(Okio.java:78)
	at okio.AsyncTimeout$1.write(AsyncTimeout.java:179)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:171)
	at okio.RealBufferedSink.write(RealBufferedSink.java:41)
	at okhttp3.internal.http1.Http1Codec$FixedLengthSink.write(Http1Codec.java:286)
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.java:171)
	at okio.RealBufferedSink.writeAll(RealBufferedSink.java:99)
	at okhttp3.RequestBody$3.writeTo(RequestBody.java:118)
	at okhttp3.MultipartBody.writeOrCountBytes(MultipartBody.java:171)
	at okhttp3.MultipartBody.writeTo(MultipartBody.java:113)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:62)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at org.sonarqube.ws.client.OkHttpClientBuilder.addHeaders(OkHttpClientBuilder.java:207)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185)
	at okhttp3.RealCall.execute(RealCall.java:69)
	at org.sonarqube.ws.client.HttpConnector.doCall(HttpConnector.java:188)
	... 56 more

I have tried the Maven example at sonar-scanning-examples, which works, even after downgrading sonar-maven-plugin to 3.2.

I have also tried passing -Dsonar.ws.timeout=6000 to mvn sonar:sonar in case it’s a timeout issue, but that didn’t work either.

(In case it’s relevant, I’m running SonarQube in an EKS cluster, with an Aurora PostgreSQL-compatible DB, deployed with a modified version of the sonarqube Helm chart.)

ganncamp · July 3, 2019, 6:25pm

Hi,

First, you should be aware that the current version of the Scanner for Maven is 3.6. Unless you have specific reasons for pinning an old version of the scanner in your pom, you usually want whatever the latest is.

That said, it’s interesting that the sample project works with the old scanner. I’m not sure what to do with that tho.

What happens if you manually request the URL that the scanner fails on? And since you said this is a new instance (with an empty DB, right?) is this the first analysis of the project or were you previously successful analyzing this project on the server?

And finally, it’s incumbent upon me to point out that 7.9 was released on Monday, and it’s the new/current LTS, so since you’re dealing with a brand new instance anyway, you probably want to be starting with 7.9 instead of 6.7.

Ann

mirzmaster · July 3, 2019, 10:56pm

Thanks for your response @ganncamp.

Pinning the scanner version to 3.2 isn’t strictly necessary, but since it worked with the sample project I haven’t invested in the effort to upgrade to 3.6.

When I manually request the URL the scanner fails on it does seem to hit the service since SonarQube complains about the fact that I’m submitting a GET request and need to be POSTing to that endpoint instead. As I’m not sure what is being POSTed, I haven’t tried to submit a POST request.

Yes, this is the first analysis for this project, though I assume the DB has been initialized due to the successful analysis of the sample project.

And finally, yes, I’m aware that 7.9 was released this week. I’d still like to get what I have in place working before investing further effort in shifting to 7.9.

mirzmaster · July 6, 2019, 6:42pm

Finally figured this issue out.

The broken pipe was an indication that communication between the scanner and the server was being interrupted, though I focused on looking for clues in the scanner output and server logs, but didn’t find a root cause there. After coming across another issue here on the forums (SonarQube report upload throwing SocketTimeOutException), it occurred to me that the Kubernetes nginx ingress controller could be interfering.

Checking the logs there showed that nginx was not happy with the request body size (our generated reports are > 160M when compressed. Adding the proxy-body-size annotation to our ingress resource corrected the problem.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  ...
  annotations:
    nginx.ingress.kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "200m"
...

Et, voila!

Hope this helps others that might experience the same issue.

ganncamp · July 8, 2019, 12:24pm

Thanks for the followup! I’m sure this will help others in the future.