The specified CGI application encountered an error and the server terminated the process

SonarQube Server / Community Build
1

which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
9.9LTS Enterprise Version

how is SonarQube deployed: zip, Docker, Helm
Azure App Service with GitHub - vanderby/SonarQube-AzureAppService: Instructions and files to host SonarQube on an Azure App Service without a container.

So far the application works great for the past year, but something did happen, also metrics for the resources are at max 50%.

Had a downtime today about 10 minutes, then restarted the web app. I have taken over the SonarQube from colleague that has left the company and not many notes left. So, would appreciate any advice here. For what should I look if I have this type of error and people could not get into SonarQube. What logs could tell this specifically? Thanks.

2

Hey there.

For specifics about deploying with GitHub - vanderby/SonarQube-AzureAppService: Instructions and files to host SonarQube on an Azure App Service without a container., you’ll want to reach out to the maintainer.

Normally you can check out the logs to see why SonarQube crashed, if that’s indeed what happened.

1 Like
3

Yes, it did crash and I am trying to understand what exactly did. May I share a log files from the specific timeframe SQ was down?

4

It’s your job to look into the log file and see if you can figure it out. If you can’t, feel free to share to see if somebody can help you :slight_smile:

1 Like
5

Thanks Colin. Of course I do understand. I will look into this and share something that bugs me.

6

One of the errors that I have found in this time period when App Service was showing http errors in the logs is following. Can you explain these ones?

2023.09.12 10:35:29 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted
Caused by: java.net.SocketException: Permission denied: no further information
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
	at java.base/java.lang.Thread.run(Thread.java:833)
2023.09.12 10:35:30 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
org.elasticsearch.ElasticsearchException: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2695)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105)
	at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:151)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:64)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:92)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:77)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:62)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:223)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:288)
Caused by: java.util.concurrent.ExecutionException: java.net.SocketException: Permission denied: no further information
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
	at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:244)
	at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:75)
	at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2692)
	... 10 common frames omitted

Also, here are the es.log from that time frame when problems started:

2023.09.12 10:34:34 INFO  es[][o.e.n.Node] version[7.17.8], pid[6480], build[unknown/unknown/120eabe1c8a0cb2ae87cffc109a5b65d213e9df1/2022-12-02T17:33:09.727072865Z], OS[Windows Server 2016/10.0/amd64], JVM[Microsoft/OpenJDK 64-Bit Server VM/17.0.4.1/17.0.4.1+1-LTS]
2023.09.12 10:34:34 INFO  es[][o.e.n.Node] JVM home [C:\Program Files\Java\microsoft-jdk-17.0.4.1]
2023.09.12 10:34:34 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseG1GC, -Djava.io.tmpdir=C:\home\site\wwwroot\sonarqube-9.9.0.65466\temp, -XX:ErrorFile=C:\home\site\wwwroot\sonarqube-9.9.0.65466\logs\es_hs_err_pid%p.log, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=C:\home\site\wwwroot\sonarqube-9.9.0.65466\temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Dcom.redhat.fips=false, -Des.enforce.bootstrap.checks=true, -Xmx2G, -Xms2G, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=C:\home\site\wwwroot\sonarqube-9.9.0.65466\elasticsearch, -Des.path.conf=C:\home\site\wwwroot\sonarqube-9.9.0.65466\temp\conf\es]
2023.09.12 10:34:37 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2023.09.12 10:34:37 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2023.09.12 10:34:37 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2023.09.12 10:34:37 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2023.09.12 10:34:37 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2023.09.12 10:34:37 INFO  es[][o.e.p.PluginsService] no plugins loaded
2023.09.12 10:34:37 ERROR es[][o.e.b.Bootstrap] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\home\site\wwwroot\sonarqube-9.9.0.65466\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:328) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.node.Node.<init>(Node.java:429) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) [elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) [elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.17.8.jar:7.17.8]
	at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) [elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.17.8.jar:7.17.8]
2023.09.12 10:34:37 ERROR es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\home\site\wwwroot\sonarqube-9.9.0.65466\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:173) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:160) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) ~[elasticsearch-cli-7.17.8.jar:7.17.8]
	at org.elasticsearch.cli.Command.main(Command.java:77) ~[elasticsearch-cli-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:125) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) ~[elasticsearch-7.17.8.jar:7.17.8]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[C:\home\site\wwwroot\sonarqube-9.9.0.65466\data\es7]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:328) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.node.Node.<init>(Node.java:429) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.17.8.jar:7.17.8]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:169) ~[elasticsearch-7.17.8.jar:7.17.8]
	... 6 more
7

This can happen if multiple SonarQube instances are started against the same database, or if the file system becomes unwritable. In the context of Azure App Services, I’m not sure when that would happen, and I would defer to the maintainer if it becomes a recurring issue.

1 Like
8

Understood. There is only one instance connected to that database. We use test db for our test sonarqube, this is totally on prod and has a dedicated sql database.

9

What would be the most efficient way to deploy SonarQube in Azure? So it’s a bit less hustle when it comes to the upgrades?

10

Effectively you have two options: Deploying on a VM or deploying via container. Both are described here.

I don’t have any particular guidance to share about what makes the most sense in your environment or with Azure.

1 Like
11

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.