Error upgrading from SonarQube 7.3 to 7.9.6 (prerequisite upgrade for 8.9.7 LTS)

SonarQube Server / Community Build
,
1

Current system:

  • SonarQube v7.3 Community
  • MS SQL 2016
  • JDK 8.0
  • Running on Windows Server 2016 VM, with IIS reverse proxy for SSL

Attempting to upgrade to 8.9.7 LTS, but requires intermediary upgrade to previous 7.9.6 LTS. Preparation steps:

  • Install JDK 11.0.13
  • Update sonar.properties
  • Update wrapper.conf (points to JDK 11)

Prior to installing and running NT service, I performed startsonar to test the initialization. This step fails always. sonar.log reports failed initialization of elastic search.

sonar.log says:

--> Wrapper Started as Console
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

java.net.SocketException: Permission denied: connect
2022.04.02 16:10:22 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory C:\sonarqube\v7.9.6\temp
2022.04.02 16:10:22 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2022.04.02 16:10:22 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [C:\sonarqube\v7.9.6\elasticsearch]: C:\SDK\Java\jdk11.0.13.8\bin\java -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=C:\sonarqube\v7.9.6\temp -XX:ErrorFile=../logs/es_hs_err_pid%p.log -Des.enforce.bootstrap.checks=true -Xms512m -Xmx512m -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=C:\sonarqube\v7.9.6\elasticsearch -Des.path.conf=C:\sonarqube\v7.9.6\temp\conf\es -cp lib/* org.elasticsearch.bootstrap.Elasticsearch
2022.04.02 16:10:22 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2022.04.02 16:10:23 INFO  app[][o.e.p.PluginsService] no modules loaded
2022.04.02 16:10:23 INFO  app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
Exception in thread "elasticsearch[_client_][generic][T#1]" java.lang.NullPointerException
	at org.elasticsearch.transport.netty4.Netty4Transport.lambda$stopInternal$2(Netty4Transport.java:255)
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103)
	at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85)
	at org.elasticsearch.common.lease.Releasables.close(Releasables.java:36)
	at org.elasticsearch.common.lease.Releasables.close(Releasables.java:46)
	at org.elasticsearch.common.lease.Releasables.close(Releasables.java:51)
	at org.elasticsearch.transport.netty4.Netty4Transport.stopInternal(Netty4Transport.java:254)
	at org.elasticsearch.transport.TcpTransport.lambda$doStop$11(TcpTransport.java:578)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2022.04.02 16:10:26 ERROR app[][o.s.a.p.EsManagedProcess] Failed to check status
java.lang.IllegalStateException: failed to create a child event loop
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
	at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:77)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:72)
	at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:59)
	at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:131)
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:72)
	at org.elasticsearch.transport.TransportService.doStart(TransportService.java:228)
	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:72)
	at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:227)
	at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:277)
	at org.sonar.application.es.EsConnectorImpl$MinimalTransportClient.<init>(EsConnectorImpl.java:103)
	at org.sonar.application.es.EsConnectorImpl.buildTransportClient(EsConnectorImpl.java:89)
	at org.sonar.application.es.EsConnectorImpl.getTransportClient(EsConnectorImpl.java:74)
	at org.sonar.application.es.EsConnectorImpl.getClusterHealthStatus(EsConnectorImpl.java:61)
	at org.sonar.application.process.EsManagedProcess.checkStatus(EsManagedProcess.java:88)
	at org.sonar.application.process.EsManagedProcess.checkOperational(EsManagedProcess.java:73)
	at org.sonar.application.process.EsManagedProcess.isOperational(EsManagedProcess.java:58)
	at org.sonar.application.process.ManagedProcessHandler.refreshState(ManagedProcessHandler.java:201)
	at org.sonar.application.process.ManagedProcessHandler$EventWatcher.run(ManagedProcessHandler.java:258)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:168)
	at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:127)
	at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36)
	at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
	... 21 common frames omitted
Caused by: java.io.IOException: Unable to establish loopback connection
	at java.base/sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
	at java.base/sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.nio.ch.PipeImpl.<init>(PipeImpl.java:171)
	at java.base/sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
	at java.base/java.nio.channels.Pipe.open(Pipe.java:155)
	at java.base/sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:142)
	at java.base/sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
	at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:166)
	... 25 common frames omitted
Caused by: java.net.SocketException: Permission denied: listen
	at java.base/sun.nio.ch.Net.listen(Native Method)
	at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:228)
	at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
	at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:73)
	at java.base/sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(PipeImpl.java:121)
	at java.base/sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:76)
	... 33 common frames omitted
2022.04.02 16:10:27 INFO  app[][o.e.p.PluginsService] no modules loaded
2022.04.02 16:10:27 INFO  app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2022.04.02 16:11:03 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2022.04.02 16:11:03 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [C:\sonarqube\v7.9.6]: C:\SDK\Java\jdk11.0.13.8\bin\java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=C:\sonarqube\v7.9.6\temp --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -server -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/common/*;C:\sonarqube\v7.9.6\lib\jdbc\mssql\mssql-jdbc-7.2.2.jre11.jar org.sonar.server.app.WebServer C:\sonarqube\v7.9.6\temp\sq-process12458333604459337062properties
2022.04.02 16:11:05 INFO  app[][o.s.a.SchedulerImpl] Process[web] is stopped
2022.04.02 16:11:05 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
2022.04.02 16:11:05 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped

es.log says:

2022.04.02 16:10:27 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[(C:)]], net usable_space [57gb], net total_space [126.9gb], types [NTFS]
2022.04.02 16:10:27 INFO  es[][o.e.e.NodeEnvironment] heap size [495.3mb], compressed ordinary object pointers [true]
2022.04.02 16:10:27 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [2GSZZTI8TqqRtssdJ_qcfw]
2022.04.02 16:10:27 INFO  es[][o.e.n.Node] version[6.8.0], pid[2984], build[unknown/unknown/65b6179/2019-05-15T20:06:13.172855Z], OS[Windows Server 2016/10.0/amd64], JVM[ojdkbuild/OpenJDK 64-Bit Server VM/11.0.13/11.0.13+8-LTS]
2022.04.02 16:10:27 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=C:\sonarqube\v7.9.6\temp, -XX:ErrorFile=../logs/es_hs_err_pid%p.log, -Des.enforce.bootstrap.checks=true, -Xms512m, -Xmx512m, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=C:\sonarqube\v7.9.6\elasticsearch, -Des.path.conf=C:\sonarqube\v7.9.6\temp\conf\es]
2022.04.02 16:10:28 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2022.04.02 16:10:28 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2022.04.02 16:10:28 INFO  es[][o.e.p.PluginsService] loaded module [mapper-extras]
2022.04.02 16:10:28 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2022.04.02 16:10:28 INFO  es[][o.e.p.PluginsService] loaded module [percolator]
2022.04.02 16:10:28 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2022.04.02 16:10:28 INFO  es[][o.e.p.PluginsService] loaded module [repository-url]
2022.04.02 16:10:28 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2022.04.02 16:10:28 INFO  es[][o.e.p.PluginsService] no plugins loaded
2022.04.02 16:10:30 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2022.04.02 16:10:32 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and host providers [settings]
2022.04.02 16:10:32 INFO  es[][o.e.n.Node] initialized
2022.04.02 16:10:32 INFO  es[][o.e.n.Node] starting ...
2022.04.02 16:10:33 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2022.04.02 16:10:33 INFO  es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2022.04.02 16:10:37 INFO  es[][o.e.c.s.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {sonarqube}{2GSZZTI8TqqRtssdJ_qcfw}{lq_vcVWuSeG9lRPy1Yt4bQ}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}
2022.04.02 16:10:37 INFO  es[][o.e.c.s.ClusterApplierService] new_master {sonarqube}{2GSZZTI8TqqRtssdJ_qcfw}{lq_vcVWuSeG9lRPy1Yt4bQ}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: apply cluster state (from master [master {sonarqube}{2GSZZTI8TqqRtssdJ_qcfw}{lq_vcVWuSeG9lRPy1Yt4bQ}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2022.04.02 16:10:37 INFO  es[][o.e.n.Node] started
2022.04.02 16:10:37 INFO  es[][o.e.g.GatewayService] recovered [0] indices into cluster_state

Seems elasticsearch is unable to complete port binding. Sometimes, the webserver initialization starts, but it fails for a similar exception. I cannot find any specific posts on resolving this issue. An upgrade from v6.9 to v7.3 was completed a long time ago on the same machine without issues. My suspicions are:

  1. Firewall - However, there are no special rules defined for the existing v7.3 (and older) instances. Those work fine, and only localhost binding is used as a reverse proxy is configured to route external connections.
  2. JDK 11 (?) - Could all connection attempts via JDK 11 be blocked for some reason? What can be done?

ADDITIONAL INFO:
I can safely revert to v7.3 again on the same system; verified this several times by attempting to initialize v7.9.6 and then back to v7.3. There are no parameter differences in the properties of both instances (excluding comments and any additional parameters introduced in v7.9.6).

I do not know how to resolve this issue. Please help.

Regards,
Jeev

2

Did some additional investigations. The problem does seem to stem from the JDK11 package. I tried from several sources (OpenJDK Adopt, Microsoft) of v11.0.13 OR v11.0.14. None of them work.

However, v11.0.4 of the JDK works with the installation perfectly. This confirms that the JDK is causing the issue, although why is still unknown. Unfortunately, v11.0.4 is rather old and I would prefer to get it working with the newest GA releases of the JDK.

Anyone else encountering this situation?

1 Like
3

Hi,

You’ve narrowed this down to the JDK and you’re only hitting 7.9.6 on the way to 8.9.8 (just released a new version today!). So I’d just use what you know works to get 7.9.6 up & running, and then move on to 8.9.8. From there you should be able to use the newest GA releases of the JDK.

 
HTH,
Ann

4

That’s the problem! What I know ‘works’ doesn’t actually work in different environments. I forgot to mention that the test on JDK 11.0.4 only worked in a mirrored dev server, but the same does not work on a production server.

All i have confirmed is the problem is rooted around the JDK, but not exactly why and how to work around it given that I cannot get any valid JDK 11 installation working on this setup.

5

Just to be clear again:

  • Dev server, JDK 11.0.4 works. Newer versions do not.
  • Prod server, JDK 11.0.4 and newer ALL do not work.

Although I am doing only a transitionary upgrade, SQ8 also requires JDK11, so it is imperative for me to get this working anyway :frowning:

6

Hi,

Okay, going back to the 7.9 docs, it looks like Java 11 isn’t supported at all for SonarQube 7.9. So you’ll need to point your 7.9 at Java 8 to get it up and running. Then you should be fine with Java 11 when you get to SQ 8.9.

 
HTH,
Ann

7

You referenced the wrong version (7.6) of the upgrade document. Version 7.9 explicitly states the requirement for JDK11, and I have got this working in the DEV environment. I just can’t get it to work, for some reason, in the PROD environment. It keeps throwing a SocketException, indicating failure to bind to the sockets.

Again, I can reproduce the behavior in DEV using newer versions of JDK11, but the vresion that works in DEV still fails in PROD.