9.8 to 9.9 upgrade. Webserver failing to start

Nikita · February 26, 2023, 5:12pm

Hi guys,

What version are you upgrading from?
-9.8
System information (Operating system, Java version, Database provider/version)
Windows Server 2019, openjdk version “17.0.2”, MSSQL2017
What’s the issue you’re facing?
I can’t start SonarQube service after upgrade, log messages -
…

2023.02.26 18:00:33 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023.02.26 18:00:47 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2023.02.26 18:00:47 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [C:\SonarQube\sonarqube-9.9.0.65466]: C:\SonarQube\JDK\bin\java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=c:\SonarQube\SonarHome\temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -server -Xmx768m -Xms256m -XX:MaxPermSize=160m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-9.9.0.65466.jar;C:\SonarQube\sonarqube-9.9.0.65466\lib\jdbc\mssql\mssql-jdbc-10.2.1.jre11.jar org.sonar.server.app.WebServer c:\SonarQube\SonarHome\temp\sq-process12244382321443300548properties
2023.02.26 18:00:47 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [Web Server]: 1
2023.02.26 18:00:47 INFO  app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.02.26 18:00:47 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.02.26 18:00:47 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

We are using Microsoft SQL Server with Integrated Authentication, SQL connection string -

sonar.jdbc.url=jdbc:sqlserver://SERVER1NAME;databaseName=DBNAME;failoverPartner=SERVER2NAME;integratedSecurity=true;encrypt=false

As I understand besides other reasons problem might be in SQL connection. How can I troubleshoot that?
Any help would be appreciated!

ganncamp · February 27, 2023, 3:10pm

Hi,

Anything interesting in your other server logs?

Ann

Nikita · February 27, 2023, 4:28pm

I’m afraid I don’t have other service logs. Only sonar.log and es.log are being created after starting service.

As it was recommended in “Release 9.6 upgrade notes” I have replaced mssql-jdbc_auth dll with mssql-jdbc_auth-10.2.1.x64.dll when I was upgrading to 9.6. Now I tried to replace jdbc_auth-10.2.1.x64.dll with latest available jdbc_auth dll but that hadn’t helped either.

ganncamp · February 27, 2023, 5:45pm

Hi,

There’s no web.log file in your $SONARQUBE-HOME/logs directory?

Since the web process is what’s failing your server startup, its log will hold the key.

If the server is unable to create that log, that would be the first thing to investigate. If you have to, you might even try touch-ing the file and chowning it to the user running SonarQube. Or better yet, make sure the $SONARQUBE-HOME directory is owned to the SonarQube user, recursively.

Ann

Nikita · February 27, 2023, 6:22pm

SonarQube running on Windows Sever. So I might have troubles touch-ing and chown-ing. But yes server is not creating that log. That’s why I think that issue can be related to MSSQL connection.

ganncamp · February 27, 2023, 6:24pm

Hi,

Those are two different things. First you need to sort out SonarQube being able to create its log files. Once that’s taken care of, if there’s still a problem starting up we can check those logs for details.

Ann

Nikita · February 27, 2023, 6:28pm

Since sonar.log and es.log creating just fine I believe the issue lay not in the permissions area. So what you’re suggesting me to do?

ganncamp · February 27, 2023, 6:32pm

Hi,

SonarQube won’t start if it can’t create its logs. You need to figure out why web.log wasn’t created.

Ann

Nikita · February 27, 2023, 9:24pm

I think you confuse cause and effect, but okay. How can I check why web.log wasn’t created?
User account running SonarQube service has full access to all SQ folders.

ganncamp · February 28, 2023, 1:04pm

Hi,

The web process won’t start unless it can write to its logs. And the server won’t start without the web process.

That’s on your OS side & I’m particularly unqualified to advise you on Windows. It’s been quite a very long time indeed since I had to use it.

Ann

Nikita · February 28, 2023, 4:14pm

That’s definitely not on the “OS side”.
The reason of the issue could be connected to the JDK version.
As soon as I restored backup with SonarQube 9.8 and configured environment variables to point to openjdk version “11.0.13” service started successfully and web.log and all other log files were created. But when I changed environments to point to openjdk version “17.0.2” SonarQube 9.8 started and stopped after creating only two log files es.log and sonar.log with the same lines in sonar.log as before -

2023.02.28 17:03:15 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023.02.28 17:03:30 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2023.02.28 17:03:30 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [C:\SonarQube\sonarqube-9.8.0.63668]: C:\SonarQube\JDK\bin\java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=c:\SonarQube\SonarHome\temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -server -Xmx768m -Xms256m -XX:MaxPermSize=160m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-9.8.0.63668.jar;C:\SonarQube\sonarqube-9.8.0.63668\lib\jdbc\mssql\mssql-jdbc-10.2.1.jre11.jar org.sonar.server.app.WebServer c:\SonarQube\SonarHome\temp\sq-process15730836310318985955properties
2023.02.28 17:03:30 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [Web Server]: 1
2023.02.28 17:03:30 INFO  app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.02.28 17:03:30 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.02.28 17:03:30 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

So we can assume that Web Server of SonarQube 9.9 failing to start on openjdk version “17.0.2” due to some unknown issue. What can I do to fix that?

ganncamp · February 28, 2023, 5:02pm

Hi,

Can you see if this is any different with the latest Java 17?

Ann

Nikita · March 2, 2023, 9:29am

Nope, I installed openjdk version “17.0.6”, but still behavior the same, only two log files are created and in sonar.log the same records -

2023.03.02 10:26:51 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory c:\SonarQube\SonarHome\temp
2023.03.02 10:26:52 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:62926]
2023.03.02 10:26:52 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [C:\SonarQube\sonarqube-9.9.0.65466\elasticsearch]: C:\SonarQube\JDK\bin\java -XX:+UseG1GC -Djava.io.tmpdir=c:\SonarQube\SonarHome\temp -XX:ErrorFile=C:\SonarQube\SonarHome\logs\es_hs_err_pid%p.log -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djna.tmpdir=c:\SonarQube\SonarHome\temp -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=COMPAT -Dcom.redhat.fips=false -Des.enforce.bootstrap.checks=true -Xmx512m -Xms512m -XX:MaxDirectMemorySize=256m -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=C:\SonarQube\sonarqube-9.9.0.65466\elasticsearch -Des.path.conf=c:\SonarQube\SonarHome\temp\conf\es -cp lib/* org.elasticsearch.bootstrap.Elasticsearch
2023.03.02 10:26:52 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023.03.02 10:27:07 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2023.03.02 10:27:07 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [C:\SonarQube\sonarqube-9.9.0.65466]: C:\SonarQube\JDK\bin\java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=c:\SonarQube\SonarHome\temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -server -Xmx768m -Xms256m -XX:MaxPermSize=160m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-9.9.0.65466.jar;C:\SonarQube\sonarqube-9.9.0.65466\lib\jdbc\mssql\mssql-jdbc-11.2.2.jre17.jar org.sonar.server.app.WebServer c:\SonarQube\SonarHome\temp\sq-process9539204467816193879properties
2023.03.02 10:27:08 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [Web Server]: 1
2023.03.02 10:27:08 INFO  app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.03.02 10:27:08 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.03.02 10:27:08 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

ganncamp · March 2, 2023, 6:30pm

Hi,

Thanks for trying. I’ve called for a second pair of eyes.

Ann

Matteo_Mara · March 6, 2023, 1:55pm

Hello @Nikita,

If I understand correctly you have the same type of failure running both versions 9.9 and 9.8 as soon as you switch to jdk17?

Nikita · March 7, 2023, 4:15pm

Hi Matteo,
That’s right, both versions display the same behavior on openjdk 17.

Matteo_Mara · March 7, 2023, 4:54pm

Which versions of the jdbc auth dll are you using?
According to the docs it is supposed to be:

mssql-jdbc_auth-11.2.2.x64.dll for version 9.9
mssql-jdbc_auth-11.2.1.x64.dll for version 9.8

Nikita · March 8, 2023, 10:03am

I had a copy of old mssql-jdbc_auth dll in different PATH location so I doublechecked environments.

9.8 works on mssql-jdbc_auth-11.2.1.x64.dll, openjdk version “11.0.13”. No issues.
Then I replaced dll file, variables and tried to start 9.9 -
9.9 stopping after few seconds on mssql-jdbc_auth-11.2.2.x64.dll, openjdk version “17.0.6” with error

2023.03.08 10:57:01 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory c:\SonarQube\SonarHome\temp
2023.03.08 10:57:01 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:49880]
2023.03.08 10:57:01 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [C:\SonarQube\sonarqube-9.9.0.65466\elasticsearch]: C:\SonarQube\JDK\bin\java -XX:+UseG1GC -Djava.io.tmpdir=c:\SonarQube\SonarHome\temp -XX:ErrorFile=C:\SonarQube\SonarHome\logs\es_hs_err_pid%p.log -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djna.tmpdir=c:\SonarQube\SonarHome\temp -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=COMPAT -Dcom.redhat.fips=false -Des.enforce.bootstrap.checks=true -Xmx512m -Xms512m -XX:MaxDirectMemorySize=256m -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=C:\SonarQube\sonarqube-9.9.0.65466\elasticsearch -Des.path.conf=c:\SonarQube\SonarHome\temp\conf\es -cp lib/* org.elasticsearch.bootstrap.Elasticsearch
2023.03.08 10:57:01 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023.03.08 10:57:15 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2023.03.08 10:57:15 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [C:\SonarQube\sonarqube-9.9.0.65466]: C:\SonarQube\JDK\bin\java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=c:\SonarQube\SonarHome\temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -server -Xmx768m -Xms256m -XX:MaxPermSize=160m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-9.9.0.65466.jar;C:\SonarQube\sonarqube-9.9.0.65466\lib\jdbc\mssql\mssql-jdbc-11.2.2.jre17.jar org.sonar.server.app.WebServer c:\SonarQube\SonarHome\temp\sq-process16517586405101355814properties
2023.03.08 10:57:15 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [Web Server]: 1
2023.03.08 10:57:15 INFO  app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.03.08 10:57:15 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.03.08 10:57:15 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

Only two log files created in log dir - sonar.log, es.log.

joris · March 8, 2023, 10:26am

Just asking, officially SQ 9.9 support only Java 17 for server side? (where SQ9.8 still supported Java 11)

So i guess a rollback to Java 11 will not be a viable solution for this issue described here.

Prerequisites and overview (sonarqube.org)

Matteo_Mara · March 8, 2023, 10:42am

@joris correct.
While SQ 9.8 supports both JDK11 and JDK17, SQ 9.9 supports only JDK17.
I am trying to understand how to reproduce locally, but with the amount of logs available it is quite hard at the moment.