Bitbucket Server PR Decoration not consistent

[vdinicola (Vincent Dinicola)]

Must-share information:

• Sonarqube Server version: 8.7.0.41497
• Sonarqube Scanner version: 4.3.8.2102

We want to use the Sonarqube report as a way to gate our Bitbucket PRs. We have configured the Bitbucket PR decoration in accordance with this doc: Bitbucket Server Integration | SonarQube Docs

The PR decoration works for a short amount of time as you would expect. However, after a variable amount of time, the decoration starts to fail. The error message shown on Sonarqube is:
“Pull request decoration did not happen. Failed to access Bitbucket Server, the repository or the pull request”

Upon enabling DEBUG logs and looking at the ce.log file, I can see that an exception is thrown:

2021.05.20 19:41:06 WARN  ce[AXmLSskOKnPc9HCg8IoL][c.s.F.D.B.E] Pull request decoration did not happen. Failed to access Bitbucket Server, the repository or the pull request
java.lang.IllegalArgumentException: Unable to contact Bitbucket server
	at org.sonar.alm.client.bitbucketserver.BitbucketServerRestClient.doCall(BitbucketServerRestClient.java:133)
	at org.sonar.alm.client.bitbucketserver.BitbucketServerRestClient.doGet(BitbucketServerRestClient.java:114)
	at com.sonarsource.F.D.B.A.C.getPullRequest(Unknown Source)
	at com.sonarsource.F.D.B.E.A(Unknown Source)
	at com.sonarsource.F.D.B.E.A(Unknown Source)
	at com.sonarsource.F.D.B.B.A(Unknown Source)
	at com.sonarsource.F.D.c.A(Unknown Source)
	at java.base/java.util.Optional.ifPresent(Optional.java:183)
	at com.sonarsource.F.D.c.B(Unknown Source)
	at com.sonarsource.F.D.c.A(Unknown Source)
	at org.sonar.ce.async.SynchronousAsyncExecution.addToQueue(SynchronousAsyncExecution.java:27)
	at com.sonarsource.F.D.c.A(Unknown Source)
	at java.base/java.util.Optional.ifPresent(Optional.java:183)
	at com.sonarsource.F.D.c.finished(Unknown Source)
	at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.executeTask(PostProjectAnalysisTasksExecutor.java:118)
	at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.finished(PostProjectAnalysisTasksExecutor.java:109)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeListener(ComputationStepExecutor.java:91)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:63)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:235)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:217)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:162)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:137)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:89)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:120)
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at org.sonar.alm.client.bitbucketserver.BitbucketServerRestClient.doCall(BitbucketServerRestClient.java:127)
	... 33 common frames omitted
	Suppressed: java.net.SocketTimeoutException: connect timed out
		... 57 common frames omitted
2021.05.20 19:41:06 INFO  ce[AXmLSskOKnPc9HCg8IoL][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Pull Request decoration | status=SUCCESS | time=60044ms

When we restart the Sonarqube server, the PR decoration starts to work again. We initially setup a cronjob to restart Sonarqube every 2 days, but now that doesn’t even appear long enough. After about 24 hours after being restarted, the PR decoration starts to fail once again.

These errors would seem to indicate that there is an issue connecting to Bitbucket from Sonarqube, but as mentioned, this decoration works reliably for a short amount of time. More so, logs state that the PR decoration was actually successful.

I have verified that our connection to Bitbucket works and that the user we are using for the integration has the proper permissions. Any help identifying what might be causing this would be great.

[Colin]

Hey there.

Is it possible that the IP Address of your Bitbucket server (masked behind a hostname) is changing regularly?

You might be finding yourself in this situation:

Sonarqube Developer - intermittant issues connecting to Github Enterprise

Spent a decent amount of time troubleshooting this today and it appears to be caused by the IP address of our GitHub Enterprise load balancer changing. Managed to validate that was occurring by utilizing sysdig to capture network/system calls and saw the old GHE IP address was present. Fully fixed the issue by adding networkaddress.cache.ttl=60 to $JAVA_HOME/conf/security/java.security to force the Sonarqube ce process to only cache DNS results for 60 seconds.

[vdinicola (Vincent Dinicola)]

It’s very likely. Our bitbucket server is behind an AWS load balancer so the underlying IPs are out of our control. Let me try putting this in place and seeing how it works. How long does Sonar set this cache by default?

[Colin]

It should be set by the JVM, not SonarQube. Admittedly, I am not an expert here, and I’m scratching my head over how any java-based apps integrating with other applications function if the DNS is cached indefinitely.

networkaddress.cache.ttl null in openjdk

I look forward to hearing if it resolves your issue.

[Colin]

Ah, I believe we introduced a Security Manager in v8.1 (SONAR-12617) which triggers the condition to have the DNS time to live be forever.

# default value is forever (FOREVER). For security reasons, this
# caching is made forever when a security manager is set. When a security
# manager is not set, the default behavior in this implementation
# is to cache for 30 seconds.

I’m not sure this was an intended consequence of this change, and maybe there’s something that can be done SonarQube-side. I’ll ping some more intelligent folks and see what they think

[dmeneses (Duarte Meneses)]

Here is the bug ticket: [SONAR-14870] - Jira.
I was able to reproduce and test a fix. You were right about the cause of the problem, @Colin.

@vdinicola, meanwhile you can use the workaround described in the post that @Colin had already mentioned.

Sonarqube Developer - intermittant issues connecting to Github Enterprise

Spent a decent amount of time troubleshooting this today and it appears to be caused by the IP address of our GitHub Enterprise load balancer changing. Managed to validate that was occurring by utilizing sysdig to capture network/system calls and saw the old GHE IP address was present. Fully fixed the issue by adding networkaddress.cache.ttl=60 to $JAVA_HOME/conf/security/java.security to force the Sonarqube ce process to only cache DNS results for 60 seconds.

[vdinicola (Vincent Dinicola)]

I’ve had the proposed fix in place for 2 days now and the decoration looks consistent. I will keep monitoring in case this changes but so far so good. Thanks for looking into this and filing a ticket.

[vdinicola (Vincent Dinicola)]

Just to follow up, it’s been over a week since I put this fix in and Bitbucket pull request decorations have continued to work. Thank you all for your help.