Problem installing on Azure App Service with Azure SQL Server

SonarQube Server / Community Build
1

I have deployed SonarQube 7.8 to Azure from https://github.com/vanderby/SonarQube-AzureAppService and the site worked fine with internal database. I’m trying to change to use Azure SQL Server now. I’ve updated sonar.properties file to set sonar.jdbc.username, sonar.jdbc.password and sonar.jdbc.url. SonarQube connected to the database and created tables but I get following errors in sonar.log:

2019.07.04 05:49:29 WARN  app[][o.s.application.App] SonarQube will require Java 11+ starting on next version
2019.07.04 05:49:29 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory D:\home\site\wwwroot\sonarqube-7.8\temp
2019.07.04 05:49:29 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.07.04 05:49:29 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [D:\home\site\wwwroot\sonarqube-7.8\elasticsearch]: D:\Program Files\Java\zulu8.31.0.2-jre8.0.181-win_x64\bin\java -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=D:\home\site\wwwroot\sonarqube-7.8\temp -XX:ErrorFile=../logs/es_hs_err_pid%p.log -Des.enforce.bootstrap.checks=true -Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=D:\home\site\wwwroot\sonarqube-7.8\elasticsearch -Des.path.conf=D:\home\site\wwwroot\sonarqube-7.8\temp\conf\es -cp lib/* org.elasticsearch.bootstrap.Elasticsearch
2019.07.04 05:49:29 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2019.07.04 05:49:30 INFO  app[][o.e.p.PluginsService] no modules loaded
2019.07.04 05:49:30 INFO  app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
Unable to set the process priority:  Access is denied. (0x0)
Attempt to set the console title failed: The handle is invalid. (0x6)
--> Wrapper Started as Console
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

What do I need to do to fix this?

1 Like
2

You might want to check the es.log file in your SonarQube logs directory, since the failure looks to be elasticsearch related.

3

Here’s my es.log errors. What do they mean?

2019.07.04 04:58:32 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[Windows (D:)]], net usable_space [2.5gb], net total_space [31.6gb], types [NTFS]
2019.07.04 04:58:32 INFO  es[][o.e.e.NodeEnvironment] heap size [494.9mb], compressed ordinary object pointers [true]
2019.07.04 04:58:34 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [tcKh-3_uRRWkRtK2yfZWgQ]
2019.07.04 04:58:34 INFO  es[][o.e.n.Node] version[6.8.0], pid[12644], build[unknown/unknown/65b6179/2019-05-15T20:06:13.172855Z], OS[Windows Server 2016/10.0/amd64], JVM[Azul Systems, Inc./OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b02]
2019.07.04 04:58:34 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=D:\home\site\wwwroot\sonarqube-7.8\temp, -XX:ErrorFile=../logs/es_hs_err_pid%p.log, -Des.enforce.bootstrap.checks=true, -Xms512m, -Xmx512m, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=D:\home\site\wwwroot\sonarqube-7.8\elasticsearch, -Des.path.conf=D:\home\site\wwwroot\sonarqube-7.8\temp\conf\es]
2019.07.04 04:58:37 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2019.07.04 04:58:37 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2019.07.04 04:58:37 INFO  es[][o.e.p.PluginsService] loaded module [mapper-extras]
2019.07.04 04:58:37 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2019.07.04 04:58:37 INFO  es[][o.e.p.PluginsService] loaded module [percolator]
2019.07.04 04:58:37 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2019.07.04 04:58:37 INFO  es[][o.e.p.PluginsService] loaded module [repository-url]
2019.07.04 04:58:37 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2019.07.04 04:58:37 INFO  es[][o.e.p.PluginsService] no plugins loaded
2019.07.04 04:58:42 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2019.07.04 04:58:46 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and host providers [settings]
2019.07.04 04:58:48 INFO  es[][o.e.n.Node] initialized
2019.07.04 04:58:48 INFO  es[][o.e.n.Node] starting ...
2019.07.04 04:58:48 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2019.07.04 04:58:48 INFO  es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2019.07.04 04:58:51 INFO  es[][o.e.c.s.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {sonarqube}{tcKh-3_uRRWkRtK2yfZWgQ}{UNgED36BSzaG8P2ppomevw}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}
2019.07.04 04:58:51 INFO  es[][o.e.c.s.ClusterApplierService] new_master {sonarqube}{tcKh-3_uRRWkRtK2yfZWgQ}{UNgED36BSzaG8P2ppomevw}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: apply cluster state (from master [master {sonarqube}{tcKh-3_uRRWkRtK2yfZWgQ}{UNgED36BSzaG8P2ppomevw}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2019.07.04 04:58:51 INFO  es[][o.e.n.Node] started
2019.07.04 04:58:53 INFO  es[][o.e.g.GatewayService] recovered [7] indices into cluster_state
2019.07.04 04:59:21 WARN  es[][o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [tcKh-3_uRRWkRtK2yfZWgQ][sonarqube][D:\home\site\wwwroot\sonarqube-7.8\data\es6\nodes\0] free: 2.5gb[8%], shards will be relocated away from this node
2019.07.04 04:59:21 INFO  es[][o.e.c.r.a.DiskThresholdMonitor] rerouting shards: [high disk watermark exceeded on one or more nodes]
2019.07.04 04:59:52 WARN  es[][o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [tcKh-3_uRRWkRtK2yfZWgQ][sonarqube][D:\home\site\wwwroot\sonarqube-7.8\data\es6\nodes\0] free: 2.5gb[8%], shards will be relocated away from this node
2019.07.04 05:00:22 WARN  es[][o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [tcKh-3_uRRWkRtK2yfZWgQ][sonarqube][D:\home\site\wwwroot\sonarqube-7.8\data\es6\nodes\0] free: 2.5gb[8%], shards will be relocated away from this node
2019.07.04 05:00:22 INFO  es[][o.e.c.r.a.DiskThresholdMonitor] rerouting shards: [high disk watermark exceeded on one or more nodes]
2019.07.04 05:00:52 WARN  es[][o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [tcKh-3_uRRWkRtK2yfZWgQ][sonarqube][D:\home\site\wwwroot\sonarqube-7.8\data\es6\nodes\0] free: 2.5gb[8%], shards will be relocated away from this node
2019.07.04 05:01:22 WARN  es[][o.e.c.r.a.DiskThresholdMonitor] high disk watermark [90%] exceeded on [tcKh-3_uRRWkRtK2yfZWgQ][sonarqube][D:\home\site\wwwroot\sonarqube-7.8\data\es6\nodes\0] free: 2.5gb[8%], shards will be relocated away from this node
2019.07.04 05:01:22 INFO  es[][o.e.c.r.a.DiskThresholdMonitor] rerouting shards: [high disk watermark exceeded on one or more nodes]
4

And more:

2019.07.04 06:25:07 WARN  es[][o.e.g.G.InternalPrimaryShardAllocator] [metadatas][0]: failed to list shard for shard_started on node [tcKh-3_uRRWkRtK2yfZWgQ]
org.elasticsearch.action.FailedNodeException: Failed node [tcKh-3_uRRWkRtK2yfZWgQ]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:236) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:151) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:210) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1114) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1226) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1200) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TransportService$7.onFailure(TransportService.java:703) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:736) [elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-6.8.0.jar:6.8.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.elasticsearch.transport.RemoteTransportException: [sonarqube][127.0.0.1:9001][internal:gateway/local/started_shards[n]]
Caused by: org.elasticsearch.ElasticsearchException: failed to load started shards
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:169) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:61) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:138) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:259) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:255) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.0.jar:6.8.0]
	... 3 more
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:1766, file:D:\home\site\wwwroot\sonarqube-7.8\data\es6\nodes\0\indices\vuMFhuTOQ1-vFKf6fKY2fw\_state\state-1766.st]
	at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:165) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:127) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:61) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:138) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:259) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:255) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.0.jar:6.8.0]
	... 3 more
Caused by: java.io.IOException: failed to read [id:1766, file:D:\home\site\wwwroot\sonarqube-7.8\data\es6\nodes\0\indices\vuMFhuTOQ1-vFKf6fKY2fw\_state\state-1766.st]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:127) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:61) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:138) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:259) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:255) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.0.jar:6.8.0]
	... 3 more
Caused by: java.nio.file.NoSuchFileException: D:\home\site\wwwroot\sonarqube-7.8\data\es6\nodes\0\indices\vuMFhuTOQ1-vFKf6fKY2fw\_state\state-1766.st
	at sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79) ~[?:?]
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97) ~[?:?]
	at sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102) ~[?:?]
	at sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_181]
	at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_181]
	at org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28]
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:182) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:127) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.gateway.TransportNodesListGatewayStartedShards.nodeOperation(TransportNodesListGatewayStartedShards.java:61) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:138) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:259) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:255) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:692) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-6.8.0.jar:6.8.0]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.8.0.jar:6.8.0]
	... 3 more
5

@Alexander_Zhidkov, actually, yes.

Eventially it will get worse and finally sonarqube would reject to do it’s job (would just fail on startup).

I googled around quite a bit and found out that there is a little flexibility in terms of how you can adjust sonarqube’s elasticsearch using configuration files - especially if we are talking about it’s shard settings.

@Colin, I hope I am wrong.

One of solutions would require allocation.disk.threshold_enabled elastic search’s setting adjustment (there are some more), but all of them need to be done using curl which I could not manage to do with SonarQube deployed as a part of Azure AppService.

Please advise