Process exited with code 255 - The remote server returned an error: (503) Server Unavailable

RyanK · March 19, 2019, 11:33am

Recently we started to have issues with Team City (TC) communicating to our SonarQube (SQ) server.
Originally we were using the DNS server name, my log below is IP address and still does not work.

Things to point out:

I am still able to login and view SQ server and view project results
I do not see any errors on the SQ server when I RDP onto it and Windows I checked Windows Event viewer
Only the TC projects that require to build C# (“SonarCanner for MSBuild”) fails. Our other projects that are non-C# projects (“SonarQube Runner”) is still able to communicate and log the results SQ server. Both DNS name and IP address.
When I click on the url 'http://172.24.21.150:9000/sonar/api/server/version" it loads and provides the version.
We are using Community Edition v7.4.18908
I have restarted SQ server and host OS server

the cut off line in the screen shot " Starting: C:\BuildAgent\tools\sonar-scanner-msbuild.4.0.2.892\MSBuild.SonarQube.Runner.exe “/d:project.home=.” “/d:sonar.host.url=http://172.24.21.150:9000/sonar” “/k:Customer_ProfileApi” “/n:Profile API” “/v:6-rel” “/d:sonar.login=d320bcb3d6c44ccc7f9cdc4059c7dabdd6aaaf38” /d:sonar.analysis.mode=publish /d:sonar.cs.dotcover.reportsPaths=“CoverageReport.html” /d:sonar.log.level=DEBUG /d:sonar.sourceEncoding=UTF-8 /d:sonar.verbose=true begin"

RyanK · March 22, 2019, 12:01pm

We have tried the new version Community Edition Version 7.7 (build 23042)
still encounter the same issue.

ganncamp · March 26, 2019, 7:37pm

Hi,

What’s in your SonarQube logs?

Ann

RyanK · March 27, 2019, 6:47pm

Sonar logs did not really say anything about the issue.
Windows event viewer had something

ganncamp · March 27, 2019, 7:06pm

Hi,

These errors are from scanners, not the server. If the analysis log shows a 503 error, there really should be some trace of it in the server logs. Perhaps they’ve rolled since the error happened?

Ann

RyanK · April 3, 2019, 1:12pm

Hi Ann

Can you please confirm I am looking in the correct directories. I ran one test build in Team City first thing in the morning.

From the SQ installed directory:

\Logs\Access.log
Do not see any entry close to current date or time
\Logs\ec.log
\Logs\es.log
\Logs\sonar.log
\Logs\web.log

Current date logs were not created for

Access.log
ce.log
es.log

Is there anywhere else I should be looking?

ganncamp · April 3, 2019, 1:28pm

Hi,

Assuming you’re in the right SonarQube instance, (and assuming your paths were auto-corrected from “logs” to “Logs”) you’re looking in the right place. However, since the recommended upgrade procedure is to explode the zip for the new version next to the old version and update configs in the new directory, it might be the case that you’re looking in the wrong SonarQube directory. This sounds particularly likely since you don’t see recent log entries for your running instance.

Ann

RyanK · April 3, 2019, 4:31pm

We had three versions installed on our server - 7.3, 7.4, 7.7.

For each conf directory I did a folder and file compare on sonar.properties and wrapper.config file.
There was no differences other

the comments you guys made
sonar.path.data=data and sonar.path.temp=temp were commented out in v7.7

Is there any other directory and files I should make a comparison to?

ganncamp · April 3, 2019, 5:32pm

Hi,

I’m really not going to be able to help you any further until you find the underlying cause of the 503 returned to analysis. If it’s not in your SonarQube server logs, then perhaps you have some proxy or firewall sitting in the middle that’s the source of the error?

Ann

RyanK · April 30, 2019, 2:36pm

I have made some progress with this issue.

Based on this URL it got me thinking if we also have the same issue that the " Several of the properties weren’t honored from sonar-project.properties , so I had to declare them explicitly:"

At the step calling “SonarScanner for MSBuild: being Analysis” I added

sonar.host.url
sonar.login
sonar.password
sonar.jbdc.username
sonar.jdbc.password

Now the scanner starts and it fails trying to load the xml file

I checked the SQ logs and Windows Events. Nothing was actually logged.

Any guidance on where I should be looking.

ganncamp · April 30, 2019, 3:24pm

Hi,

Congratulations on your progress.

You’re not going to find details on your current error in the SonarQube logs. Instead, you need to try to get more verbose logging on the analysis side. Try adding /d:sonar.verbose=true to your start command & report back.

BTW, I’m aware of the joy that is copying out of a Windows command window, and yet it’s far easier to help with code-formatted text logs, rather than screenshots.

Ann

RyanK · April 30, 2019, 3:27pm

This is a copy and paste of from Team City log. To show you I have been using verbose and debug level logging.

Starting: C:\BuildAgent\tools\sonar-scanner-msbuild.4.6.0.1930\MSBuild.SonarQube.Runner.exe “/d:project.home=.” “/d:sonar.host.url=http://sonarqube.pinnacle.com:9000/sonar” “/d:sonar.jdbc.url=jdbc:postgresql://localhost/sonarqube” “/d:sonar.jdbc.username=sonarqube” “/k:Customer_ProfileApi” “/n:Profile API” “/v:45-hotfix” "/d:sonar.password=" “/d:sonar.login=Admin” /d:sonar.host.url=“http://sonarqube.pinnacle.com:9000/sonar” /d:sonar.login=“fbba62febfc0f51a335b080bce730349301c6a2f” /d:sonar.password="" /d:sonar.jdbc.username=“sonarqube” /d:sonar.jdbc.password="*****" /d:sonar.analysis.mode=publish /d:sonar.cs.dotcover.reportsPaths=“CoverageReport.html” /d:sonar.log.level=DEBUG /d:sonar.sourceEncoding=UTF-8 /d:sonar.verbose=true begin

ganncamp · April 30, 2019, 3:37pm

Hi,

Thanks for sharing your analysis command. Let’s unpack it a little (I’ve rearranged for coherence of my comments):

Starting: C:\BuildAgent\tools\sonar-scanner-msbuild.4.6.0.1930\MSBuild.SonarQube.Runner.exe 
“/k:Customer_ProfileApi” “/n:Profile API” “/v:45-hotfix” 
“/d:sonar.host.url=http://sonarqube.pinnacle.com:9000/sonar” 
/d:sonar.cs.dotcover.reportsPaths=“CoverageReport.html” 
/d:sonar.sourceEncoding=UTF-8

so far so good

“/d:project.home=.”

I don’t know what this is

“/d:sonar.jdbc.url=jdbc:postgresql://localhost/sonarqube” 
“/d:sonar.jdbc.username=sonarqube”

IIRC, we stopped using these values in … 5.3?

"/d:sonar.password=* ******" “/d:sonar.login=Admin”

It would really be better to use a token, but okay

/d:sonar.host.url=“http://sonarqube.pinnacle.com:9000/sonar”

no need to do this twice

/d:sonar.login=“fbba62febfc0f51a335b080bce730349301c6a2f” /d:sonar.password="****** *"

Now I’m confused, and I’m pretty sure analysis is too. Not sure which of the two value sets will be used.

/d:sonar.jdbc.username=“sonarqube” /d:sonar.jdbc.password="* ***** *"

a) these are still obsolete
b) no need to set values twice in a single command

/d:sonar.analysis.mode=publish

This parameter doesn’t do anything any more, and even if it did, publish was always the default value

/d:sonar.log.level=DEBUG 
/d:sonar.verbose=true begin*

these do the same thing.

After all that, we’re back to needing more logs. There was nothing after that last red line in your screenshot?

RyanK · April 30, 2019, 5:13pm

We upgraded SonarScanner for MSBuild to 4.6.0.1930. So we encounter the deprecated message and unable to run the proper exe.

Screen shot shows you the third red line… “check logs…”

In a previous version these did provide different outputs.

/d:sonar.log.level=DEBUG 
/d:sonar.verbose=true*

Adding this was the ONLY way we moved to the next step. Regardless what SQ logs stated on already existing.

/d:sonar.host.url=“http://sonarqube.pinnacle.com:9000/sonar” 
/d:sonar.password=* ******" “/d:sonar.login=Admin”

ganncamp · April 30, 2019, 5:30pm

Hi,

Sorry. I was remembering this from the docs description of sonar.log.level (emphasis added)

Similar to sonar.verbose=true .

RyanK:

Adding this was the ONLY way we moved to the next step. Regardless what SQ logs stated on already existing.
/d:sonar.host.url=“http://sonarqube.pinnacle.com:9000/sonar” 
/d:sonar.password=* ******" “/d:sonar.login=Admin” 

For the host URL it’s probably a question of where the open quote is. You really should remove one of the key/value pairs, and the same for login and password; figure out which set of credentials works and remove the other one.

But none of that helps with the real problem, and at this point I’m out of my depth. I’ll see if I can find help.

Ann

RyanK · May 2, 2019, 11:55am

I have provided the SQ system info… see if this can help.

SQ System Info.txt (25.0 KB)

RyanK · May 17, 2019, 6:11pm

I have been making small progress.
I have recently notice when I log into SQ as administrator I see on the project page “master”

The last analysis has failed. More details available on the [Background Tasks](http://SQ.com:9000/sonar/project/background_tasks?id=Api) page.

The error is “java.lang.OutOfMemoryError: GC overhead limit exceeded”

In the logs this is what I found

2019.05.17 05:27:12 WARN  ce[][o.e.t.TcpTransport] send message failed [channel: Netty4TcpChannel{localAddress=/127.0.0.1:58506, remoteAddress=/127.0.0.1:9001}]
io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown
	at io.netty.channel.AbstractChannel$AbstractUnsafe.shutdownOutput(AbstractChannel.java:645)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:952)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:360)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:905)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1396)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
	at io.netty.handler.logging.LoggingHandler.flush(LoggingHandler.java:265)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
	at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
	at io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
	at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
	at io.netty.channel.AbstractChannelHandlerContext.access$1500(AbstractChannelHandlerContext.java:38)
	at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:1152)
	at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:1075)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:474)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.nio.DirectByteBuffer.duplicate(Unknown Source)
	at io.netty.buffer.PooledUnsafeDirectByteBuf.newInternalNioBuffer(PooledUnsafeDirectByteBuf.java:70)
	at io.netty.buffer.PooledUnsafeDirectByteBuf.newInternalNioBuffer(PooledUnsafeDirectByteBuf.java:31)
	at io.netty.buffer.PooledByteBuf.internalNioBuffer(PooledByteBuf.java:156)
	at io.netty.buffer.PooledUnsafeDirectByteBuf.internalNioBuffer(PooledUnsafeDirectByteBuf.java:333)
	at io.netty.channel.ChannelOutboundBuffer.nioBuffers(ChannelOutboundBuffer.java:433)
	at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:389)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:938)
	... 21 common frames omitted

2019.05.17 06:15:55 ERROR ce[AWrF5P8WSSwVJcv2rcKT][o.s.c.t.CeWorkerImpl] Failed to execute task AWrF5P8WSSwVJcv2rcKT
java.lang.OutOfMemoryError: GC overhead limit exceeded
2019.05.17 06:15:59 INFO  ce[AWrF5P8WSSwVJcv2rcKT][o.s.c.t.CeWorkerImpl] Executed task | project=Customer_ProfileApi | type=REPORT | id=AWrF5P8WSSwVJcv2rcKT | submitter=admin | status=FAILED | time=669784ms

I have so far made changes to sonar.properties. I have uncomment “sonar.web.javaOpts”, “sonar.ce.javaOpts”, “sonar.search.javaOpts”, and added “sonar.skipPackageDesign=true” and “sonar.skipDesign=true” with no changes. SQ was also restarted.

RyanK · November 1, 2019, 12:11pm

I am wondering has anyone else encounter this issue since I posted?

RyanK · February 16, 2020, 12:25pm

I found some event error logs on the build agent.

Application: MSBuild.SonarQube.Runner.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Net.WebException
   at System.Net.WebClient.DownloadDataInternal(System.Uri, System.Net.WebRequest ByRef)
   at System.Net.WebClient.DownloadString(System.Uri)
   at SonarScanner.MSBuild.PreProcessor.SonarWebService+<>c__DisplayClass15_0.<DownloadServerVersion>b__0()
   at SonarScanner.MSBuild.PreProcessor.SonarWebService.DoLogExceptions[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]](System.Func`1<System.__Canon>, System.String, System.Action`1<System.Exception>)
   at SonarScanner.MSBuild.PreProcessor.SonarWebService.DownloadServerVersion()
   at SonarScanner.MSBuild.PreProcessor.SonarWebService.GetServerVersion()
   at SonarScanner.MSBuild.PreProcessor.SonarWebService.GetProperties(System.String, System.String)
   at SonarScanner.MSBuild.PreProcessor.TeamBuildPreProcessor.FetchArgumentsAndRulesets(SonarScanner.MSBuild.PreProcessor.ISonarQubeServer, SonarScanner.MSBuild.PreProcessor.ProcessedArgs, SonarScanner.MSBuild.TFS.TeamBuildSettings, System.Collections.Generic.IDictionary`2<System.String,System.String> ByRef, System.Collections.Generic.List`1<SonarScanner.MSBuild.Common.AnalyzerSettings> ByRef)
   at SonarScanner.MSBuild.PreProcessor.TeamBuildPreProcessor.DoExecute(SonarScanner.MSBuild.PreProcessor.ProcessedArgs)
   at SonarScanner.MSBuild.BootstrapperClass.PreProcess()
   at SonarScanner.MSBuild.BootstrapperClass.Execute()
   at SonarScanner.MSBuild.Program.Execute(System.String[], SonarScanner.MSBuild.Common.ILogger)
   at MSBuild.SonarQube.Runner.Program.Main(System.String[])

and

Faulting application name: MSBuild.SonarQube.Runner.exe, version: 4.7.1.2311, time stamp: 0xc359daed
Faulting module name: KERNELBASE.dll, version: 6.1.7601.24499, time stamp: 0x5d0115f1
Exception code: 0xe0434352
Fault offset: 0x000000000000b87d
Faulting process id: 0x2228
Faulting application start time: 0x01d5e4bebe72e7dc
Faulting application path: D:\BuildAgent\tools\sonar-scanner-msbuild.4.7.1.2311\MSBuild.SonarQube.Runner.exe
Faulting module path: C:\Windows\system32\KERNELBASE.dll
Report Id: 295dfec8-50b2-11ea-9020-0050568c7ddc

The agent has these .NET library framework installed.

From the earlier post I have upgraded the SQ runner to sonar-scanner-msbuild.4.7.1.2311

mickaelcaro · March 2, 2020, 10:12am

Hi @RyanK,

First thing first, pleast update the MSBuild.SonarQube.Runner.exe, as it is deprecated and soon be removed, i suggest to use the SonarScanner.MSBuild.exe instead.

Second, i have quite few questions :

do you have the possibility to curl your SonarQube server from where your build agent is running ?
does this error happen on every build?
do you have a proxy or a firewall between your build agent and the SonarQube instance ?

Thank you.

Topic		Replies	Views
On prem - msbuild api get /api/server/version failed to request and prase SonarQube Server / Community Build	6	730	January 8, 2020
SonarQube server [https://sq.domain.com] can not be reached SonarQube Server / Community Build scanner	3	1117	September 7, 2021
Teamcity, sonarqube error - javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.secu SonarQube Server / Community Build scanner , ssl	3	628	June 21, 2023
ERROR: Caused by: You must install a plugin that supports the language 'C#' SonarQube Server / Community Build	7	1084	December 18, 2020
New project Not able to be added to SQ SonarQube Server / Community Build	4	332	January 9, 2019

Process exited with code 255 - The remote server returned an error: (503) Server Unavailable

Related topics