Sonar upgrade 7.0 to 7.9.1 LTS is failing community version

Must-share information (formatted with Markdown):

  1. which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    SonarQube Version 7.0 (build 36138) Community version, Scanner 4.0

  2. what are you trying to achieve
    Upgrade to latest version 7.9.1 LTS version

  3. what have you tried so far to achieve this

Hi Team,

I am facing an issue while upgrading SonarQube to latest version 7.9.1 LTS from 7.0
While upgrading Sonar 7.0 to 7.9.1 hosted on Redhat Linux server, upgrade is failing due to below mentioned error, this is happening when I try to start latest sonar version 7.9.1 service on same old database: Sonar2 , If I start old sonar version service 7.0 , I don’t see this issue, it’s up and running fine…SPN & Kerberos have been enabled and working fine…

Documents followed:

https://docs.sonarqube.org/latest/setup/install-server/
https://docs.sonarqube.org/latest/setup/upgrading/
https://docs.sonarqube.org/latest/requirements/requirements/

Server specifications:

service account: sonarxxx-> it has full read/write permissions
webserver: Red Hat Enterprise Linux Server release 7.7 (Maipo)
Linux 3.10.0-1062.el7.x86_64 #1 SMP Thu Jul 18 20:25:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
DB server: Windows server 2016
collation set to SQL_Latin1_General_CP1_CS_AS
we created a service account for web to database communication and gave permissions as per your upgrade document
Database: Sonar2
Placed mssql-jdbc-7.2.2.jre11.jar under /opt/sonarqube-7.9.1/extensions/jdbc-driver/mssql
Sonar old version: 7.0
Sonar new version: 7.9.1
Java installed : Java 12
java version “12.0.2” 2019-07-16
Java™ SE Runtime Environment (build 12.0.2+10)
Java HotSpot™ 64-Bit Server VM (build 12.0.2+10, mixed mode, sharing)

Enabled properties in Sonar.properties conf file

sonar.jdbc.url=jdbc:sqlserver://xxx.xxx.COM:1433;integratedSecurity=true;authenticationScheme=JavaKerberos;databaseName=Sonar2
sonar.jdbc.maxActive=60
sonar.jdbc.maxIdle=5
sonar.jdbc.minIdle=2
sonar.jdbc.maxWait=5000
sonar.web.host=0.0.0.0
sonar.web.port=9000
sonar.web.sessionTimeoutInMinutes=4320
sonar.search.port=9002

Wrapper.conf

wrapper.java.command=/usr/java/jdk-12.0.2/bin/java
wrapper.java.additional.1=-Dsonar.wrapped=true
wrapper.java.additional.2=-Djava.awt.headless=true
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp
wrapper.java.classpath.1=../../lib/jsw/*.jar
wrapper.java.classpath.2=../../lib/common/*.jar
wrapper.java.classpath.3=../../lib/*.jar
wrapper.java.library.path.1=./lib
wrapper.app.parameter.1=org.sonar.application.App
wrapper.java.initmemory=8
wrapper.java.maxmemory=32
wrapper.console.format=PM
wrapper.console.loglevel=INFO
wrapper.logfile=../../logs/sonar.log
wrapper.logfile.format=M
wrapper.logfile.loglevel=INFO
wrapper.syslog.loglevel=NONE
wrapper.console.title=SonarQube
wrapper.single_invocation=true
wrapper.ntservice.name=SonarQube
wrapper.ntservice.displayname=SonarQube
wrapper.ntservice.description=SonarQube
wrapper.ntservice.dependency.1=
wrapper.ntservice.starttype=AUTO_START
wrapper.ntservice.interactive=false
wrapper.disable_restarts=TRUE
wrapper.ping.timeout=0
wrapper.shutdown.timeout=300
wrapper.jvm_exit.timeout=300

Error:

Removed server name for security reason

**2019.09.09 10:44:12 INFO  web[][o.sonar.db.Database] Create JDBC data source for jdbc:sqlserver://xxx.xxx.COM:1433;integratedSecurity=true;authenticationScheme=JavaKerberos;databaseName=Sonar2**

2019.09.09 10:47:13 ERROR web[][o.s.s.p.Platform] Web server startup failed

java.lang.IllegalStateException: Fail to connect to database

        at org.sonar.db.DefaultDatabase.start(DefaultDatabase.java:90)

        at org.sonar.core.platform.StartableCloseableSafeLifecyleStrategy.start(StartableCloseableSafeLifecyleStrategy.java:40)

        at org.picocontainer.injectors.AbstractInjectionFactory$LifecycleAdapter.start(AbstractInjectionFactory.java:84)

        at org.picocontainer.behaviors.AbstractBehavior.start(AbstractBehavior.java:169)

        at org.picocontainer.behaviors.Stored$RealComponentLifecycle.start(Stored.java:132)

        at org.picocontainer.behaviors.Stored.start(Stored.java:110)

        at org.picocontainer.DefaultPicoContainer.potentiallyStartAdapter(DefaultPicoContainer.java:1016)

        at org.picocontainer.DefaultPicoContainer.startAdapters(DefaultPicoContainer.java:1009)

        at org.picocontainer.DefaultPicoContainer.start(DefaultPicoContainer.java:767)

        at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)

        at org.sonar.server.platform.platformlevel.PlatformLevel.start(PlatformLevel.java:90)

        at org.sonar.server.platform.platformlevel.PlatformLevel1.start(PlatformLevel1.java:160)

        at org.sonar.server.platform.Platform.start(Platform.java:211)

        at org.sonar.server.platform.Platform.startLevel1Container(Platform.java:170)

        at org.sonar.server.platform.Platform.init(Platform.java:86)

        at org.sonar.server.platform.web.PlatformServletContextListener.contextInitialized(PlatformServletContextListener.java:43)

        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4817)

        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5283)

        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1423)

        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1413)

        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

        at java.base/java.lang.Thread.run(Thread.java:835)

Caused by: java.lang.IllegalStateException: Can not connect to database. Please check connectivity and settings (see the properties prefixed by 'sonar.jdbc.').

        at org.sonar.db.DefaultDatabase.checkConnection(DefaultDatabase.java:134)

        at org.sonar.db.DefaultDatabase.start(DefaultDatabase.java:87)

        ... 24 common frames omitted

Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (Integrated authentication failed. ClientConnectionId:xxx)

        at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2385)

        at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2110)

        at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1563)

        at org.sonar.db.profiling.NullConnectionInterceptor.getConnection(NullConnectionInterceptor.java:31)

        at org.sonar.db.profiling.ProfiledDataSource.getConnection(ProfiledDataSource.java:317)

        at org.sonar.db.DefaultDatabase.checkConnection(DefaultDatabase.java:131)

        ... 25 common frames omitted

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication failed. ClientConnectionId:xxx

        at com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2892)

        at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthHandShake(KerbAuthentication.java:182)

        at com.microsoft.sqlserver.jdbc.KerbAuthentication.GenerateClientContext(KerbAuthentication.java:357)

        at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:4789)

        at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:3668)

        at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:3627)

        at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7194)

        at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2935)

        at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:2456)

        at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:2103)

        at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:1950)

        at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1162)

        at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:735)

        at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:53)

        at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:291)

        at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2395)

        at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2381)

        ... 30 common frames omitted

Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Receive timed out)

        at java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:776)

        at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:266)

        at java.security.jgss/sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:196)

        at com.microsoft.sqlserver.jdbc.KerbAuthentication.intAuthHandShake(KerbAuthentication.java:165)

        ... 45 common frames omitted

Caused by: java.net.SocketTimeoutException: Receive timed out

        at java.base/java.net.PlainDatagramSocketImpl.receive0(Native Method)

        at java.base/java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:181)

        at java.base/java.net.DatagramSocket.receive(DatagramSocket.java:814)

        at java.security.jgss/sun.security.krb5.internal.UDPClient.receive(NetClient.java:205)

        at java.security.jgss/sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:402)

        at java.security.jgss/sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:362)

        at java.base/java.security.AccessController.doPrivileged(AccessController.java:552)

        at java.security.jgss/sun.security.krb5.KdcComm.send(KdcComm.java:346)

        at java.security.jgss/sun.security.krb5.KdcComm.sendIfPossible(KdcComm.java:251)

        at java.security.jgss/sun.security.krb5.KdcComm.send(KdcComm.java:227)

        at java.security.jgss/sun.security.krb5.KdcComm.send(KdcComm.java:198)

        at java.security.jgss/sun.security.krb5.KrbTgsReq.send(KrbTgsReq.java:246)

        at java.security.jgss/sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:261)

        at java.security.jgss/sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:308)

        at java.security.jgss/sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:126)

        at java.security.jgss/sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)

        at java.security.jgss/sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:695)

        ... 48 common frames omitted

Hi,

Welcome to the community!

Congratulations and thank you for so thoroughly providing the background data and relevant logs!

The docs give a little guidance on reading Java stacktraces. Here’s what I get from yours:

This is about authenticating SonarQube to your Database. I’ll be quite honest and say I know nothing about SQLServer integrated authentication. However, I would start by making sure all the sonar.jdbc.* properties were configured letter-for-letter exactly the same in both instances.

 
HTH,
Ann

Hi G Ann Campbell,

Thank you for the quick response and directions.

I double checked the sonar.properties file for both instances letter-by-letter by diffing as well as sonar.jdbc.* properties, both are looks same. Attaching the both instance sonar.properties for your reference.

sonar7.0.txt (17.4 KB) sonar7.9.1.txt (19.6 KB) .

Hi,

Since your configurations are the same then I think you need to look outside SonarQube to the integrated authentication. Unfortunately, I can’t help you there.

 
Ann

I understood. Thank you Ann for your time.