Unable to start Sonarqube service after upgrade from version 9.3 to any latest version

ashwinb · March 6, 2023, 6:26am

Good Day,

Trying to upgrade from SonarQube version 9.3 to any latest version after that.
OS: Windows Server 2016, Java Version: 11, Database: MSSQL Server 2017
When installing the service of the new version the service fails to start up. Before was as easy as just replacing the sonar and wrapper files and since 9.6.1 it is no longer the case.

Rebse · March 6, 2023, 9:41am

Hi,

the Sonarqube 9.9.0 LTS / Latest server needs to run on Java17 now, whereas the Sonarqube analysis may run with Java 11 or Java 17.

As you’re using MSSQL, the version of the incorportated jdbc driver changed with nearly every Sonarqube 9.x version lately.
When using MSSQL with integrated authentication you need to adapt the dll on path to the version that works with the Sonarqube jdbc version, i.e. for Sonarqube 9.9.0 it’s mssql-jdbc_auth-11.2.2.x64.dll

see Install the server => Installing the database => Microsoft SQL Server

Gilbert

ashwinb · March 6, 2023, 2:16pm

Good Day Gilbert
Thank you so much for the guidance.

Somehow when starting up the service I keep getting this error and the service doesn’t stay running.

Child process [10276 - C:\Program Files\Java\jdk-17.0.5\bin\java.exe -Xms8m -Xmx32m -Djava.awt.headless=true --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -cp “…....\lib\sonar-application-9.9.0.65466.jar” “org.sonar.application.App” ] finished with 0

Rebse · March 6, 2023, 3:05pm

We need to have a look at the logs, %SONARQUBE_HOME%/logs = sonar.log, web.log
Sensitive parts as Sonarqube url … should be detained / anonymized

ashwinb · March 6, 2023, 3:47pm

This is the error from the sonar.log.

2023.03.06 16:13:31 INFO app[o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [E:\sonarqube\sonarqube-9.9.0.65466\elasticsearch]: C:\Program Files\Java\jdk-17.0.5\bin\java -XX:+UseG1GC -Djava.io.tmpdir=E:\sonarqube\sonarqube-9.9.0.65466\temp -XX:ErrorFile=E:\sonarqube\sonarqube-9.9.0.65466\logs\es_hs_err_pid%p.log -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djna.tmpdir=E:\sonarqube\sonarqube-9.9.0.65466\temp -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=COMPAT -Dcom.redhat.fips=false -Des.enforce.bootstrap.checks=true -Xmx512m -Xms512m -XX:MaxDirectMemorySize=256m -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=E:\sonarqube\sonarqube-9.9.0.65466\elasticsearch -Des.path.conf=E:\sonarqube\sonarqube-9.9.0.65466\temp\conf\es -cp lib/* org.elasticsearch.bootstrap.Elasticsearch
2023.03.06 16:13:31 INFO app[o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023.03.06 16:13:40 INFO app[o.s.a.SchedulerImpl] Process[es] is up
2023.03.06 16:13:40 INFO app[o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [E:\sonarqube\sonarqube-9.9.0.65466]: C:\Program Files\Java\jdk-17.0.5\bin\java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=E:\sonarqube\sonarqube-9.9.0.65466\temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-9.9.0.65466.jar;E:\sonarqube\sonarqube-9.9.0.65466\lib\jdbc\mssql\mssql-jdbc-11.2.2.jre17.jar org.sonar.server.app.WebServer E:\sonarqube\sonarqube-9.9.0.65466\temp\sq-process14502463895822180458properties
2023.03.06 16:13:44 INFO app[o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.03.06 16:13:44 INFO app[o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.03.06 16:13:44 INFO app[o.s.a.SchedulerImpl] SonarQube is stopped

ashwinb · March 6, 2023, 3:48pm

And another from the web.log

2023.03.06 16:13:44 WARN web[o.s.c.a.AnnotationConfigApplicationContext] Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘jdk.internal.loader.ClassLoaders$AppClassLoader@659e0bfd-org.sonar.db.DefaultDatabase’: Initialization of bean failed; nested exception is java.lang.IllegalStateException: Fail to connect to database
2023.03.06 16:13:44 ERROR web[o.s.s.p.PlatformImpl] Web server startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name ‘jdk.internal.loader.ClassLoaders$AppClassLoader@659e0bfd-org.sonar.db.DefaultDatabase’: Initialization of bean failed; nested exception is java.lang.IllegalStateException: Fail to connect to database
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:628)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:955)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:187)
at org.sonar.server.platform.platformlevel.PlatformLevel.start(PlatformLevel.java:80)
at org.sonar.server.platform.platformlevel.PlatformLevel1.start(PlatformLevel1.java:164)
at org.sonar.server.platform.PlatformImpl.start(PlatformImpl.java:196)
at org.sonar.server.platform.PlatformImpl.startLevel1Container(PlatformImpl.java:162)
at org.sonar.server.platform.PlatformImpl.init(PlatformImpl.java:76)
at org.sonar.server.platform.web.PlatformServletContextListener.contextInitialized(PlatformServletContextListener.java:43)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4764)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5222)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:835)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1393)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1383)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:916)
at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:265)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardService.startInternal(StandardService.java:430)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:486)
at org.sonar.server.app.EmbeddedTomcat.start(EmbeddedTomcat.java:72)
at org.sonar.server.app.WebServer.start(WebServer.java:55)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:97)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:81)
at org.sonar.server.app.WebServer.main(WebServer.java:104)
Caused by: java.lang.IllegalStateException: Fail to connect to database
at org.sonar.db.DefaultDatabase.start(DefaultDatabase.java:130)
at org.sonar.core.platform.StartableBeanPostProcessor.postProcessBeforeInitialization(StartableBeanPostProcessor.java:33)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620)
… 44 common frames omitted
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: “PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”. ClientConnectionId:eb7856d1-840e-4319-a0bd-d424cb62bc54
at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:596)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:582)
at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:100)
at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:81)
at org.sonar.db.DefaultDatabase.createHikariDataSource(DefaultDatabase.java:159)
at org.sonar.db.DefaultDatabase.initDataSource(DefaultDatabase.java:148)
at org.sonar.db.DefaultDatabase.start(DefaultDatabase.java:126)
… 48 common frames omitted
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: “PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target”. ClientConnectionId:eb7856d1-840e-4319-a0bd-d424cb62bc54
at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:3806)
at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1906)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:3329)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:2950)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:2790)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1663)
at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1064)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:121)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:359)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:470)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:561)
… 53 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458)
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:201)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1505)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1420)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1795)
… 63 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:242)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:113)
at com.microsoft.sqlserver.jdbc.HostNameOverrideX509TrustManager.checkServerTrusted(SQLServerTrustManager.java:86)
at java.base/sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1441)
at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:638)
… 75 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
… 82 common frames omitted
2023.03.06 16:13:44 INFO web[o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 8080
2023.03.06 16:13:44 INFO web[o.s.p.ProcessEntryPoint] Hard stopping process

Rebse · March 6, 2023, 4:03pm

ok, this means the keystore of Java17 installation your Sonarqube server runs on is missing the certs of your organization, %JAVA_HOME%\lib\security\cacerts

ashwinb · March 6, 2023, 6:50pm

Made the following change to the properties file and it was able to start the service afterwards.
Then it allowed me to proceed with the upgrade. All looking happy from my side now.
Appreciate the guidance. Thank you.

url=jdbc:sqlserver://:;databasename=;encrypt=false

Rebse · March 6, 2023, 10:38pm

That’s no real solution, you should fix your keystore for https connection.

Jersen · March 17, 2023, 5:42pm

May I see what exactly did you changed? I am encountering the same issues