9.8 to 9.9 upgrade. Webserver failing to start

SonarQube Server / Community Build 9.9 LTA Upgrade
23

In my case SQ 9.8 doesn’t work on JDK17. I left jdbc 11.2.1, changed only ENVs to JDK17 and when I started SQ 9.8 it’s stopped with the same error -

2023.03.08 12:20:12 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory c:\SonarQube\SonarHome\temp
2023.03.08 12:20:12 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:56321]
2023.03.08 12:20:12 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [C:\SonarQube\sonarqube-9.8.0.63668\elasticsearch]: C:\SonarQube\JDK\bin\java -XX:+UseG1GC -Djava.io.tmpdir=c:\SonarQube\SonarHome\temp -XX:ErrorFile=../logs/es_hs_err_pid%p.log -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djna.tmpdir=c:\SonarQube\SonarHome\temp -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=COMPAT -Dcom.redhat.fips=false -Des.enforce.bootstrap.checks=true -Xmx512m -Xms512m -XX:MaxDirectMemorySize=256m -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=C:\SonarQube\sonarqube-9.8.0.63668\elasticsearch -Des.path.conf=c:\SonarQube\SonarHome\temp\conf\es -cp lib/* org.elasticsearch.bootstrap.Elasticsearch
2023.03.08 12:20:12 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023.03.08 12:20:27 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2023.03.08 12:20:27 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [C:\SonarQube\sonarqube-9.8.0.63668]: C:\SonarQube\JDK\bin\java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=c:\SonarQube\SonarHome\temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -server -Xmx768m -Xms256m -XX:MaxPermSize=160m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-9.8.0.63668.jar;C:\SonarQube\sonarqube-9.8.0.63668\lib\jdbc\mssql\mssql-jdbc-11.2.1.jre11.jar org.sonar.server.app.WebServer c:\SonarQube\SonarHome\temp\sq-process5727997908424542798properties
2023.03.08 12:20:27 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [Web Server]: 1
2023.03.08 12:20:27 INFO  app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.03.08 12:20:27 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.03.08 12:20:27 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
24

We tried to verify with the dev team if this behavior was ever noticed in similar situations, and we identified only one case that was not possible to reproduce.

Apparently, there was one recorded case on a Windows machine when the start of the instance failed very quickly in the same way, producing only the sonar and the es log files in the logs folder.

The theory is that there were hanging processes that were not properly terminated from a previous execution.
At the next restart, the problem was not there anymore and it was impossible to reproduce it.

The setup where this happened was based on:

  • Windows 11
  • JDK Temurin 17.0.6
  • PostgreSQL database

We did many attempts with MSSQL, integrated security and the JDBC dll but we never managed to end up in the same situation again.

I am wondering if the working execution of version 9.8 on JDK 11 is leaving some processes hanging that block the execution of the 9.9 on JDK17.
We will try to reproduce this error again, but at the moment there is not a clear idea about what can be the cause of your issue.

25

I am having pretty much the same issue…the 9.9 upgrade failed due to not having OpenJDK17 installed (had OPENJDK11 running with 9.8). After upgrading to OPENJDK17, I get message that I need to update database manually…and after trying to go back to 9.8 (with database restored) I get a message that I cannot use an older Node of Elasticsearch. So I cannot go to 9.9 nor go back to 9.8. This was on my test server.

26

Hello @bufbooth thanks for sharing,

I think your issue is not related to the one described in the first message of the topic.
In your case the server is starting properly and it is correctly telling you to update the database since you are moving one version up.
What is the issue after updating the database?

27

The message was a one liner that stated something like “Need to update database manually” with no other information. The strange thing was that I was prompted to continue with the database update, and it started and looked like it completed successfully, and then the SonarQube service failed and in the log was that simple message that I needed to update the database manually. Right now I am trying to get back to version 9.8 (getting a message that I cannot use an older Node of Elasticsearch). Not sure how it knows that I have a new version of elasticsearch. Is elasticsearch stored anywhere besides in the C:\sonarqube\elasticsearch folder?

28

Hi,

So you ran the DB upgrade and then the server shut down with nothing in the logs? Any of them?

Not by us.

 
Ann

29

Is the instance still asking to update the database? I would expect that the update was triggered by hitting the /setup endpoint of your sonarqube instance. In the logs you should see the full list of patches applied to your database at the moment of the upgrade.
If it is still asking for the upgrade at the startup, it means that the upgrade did not started/completed.

30

To solve that “manually need to update the database” message, (was appearing in the access.log) I had to restore the database from a backup. The database update failed mid process due to the SonarQube 9.9 database upgrade starting while under Open JDK 11 (vs the needed 17). I missed the required JDK upgrade needed from going from SonarQube 9.8 to 9.9. The other issue I was experience was trying to get back to 9.8, elasticsearch was blocking me from using an older version of JDK, I had to delete the folder where the elasticsearch data is stored (sonar.path.data = ??? and sonar.path.temp = ???). So once I got all the elasticsearch data deleted, my database restored, and JDK 11 back on my server, I finally got sonarqube 9.8 to work again. I then removed JDK 11 and installed 17 and did the SonarQube 9.9 deployment and all finally worked.

31

Elasticsearch is stored in that folder (C:\sonarqube\elasticsearch) and the 2nd place is sonar.path.data and sonar.path.temp (under the sonar.properties.config file, at the bottom under OTHERS).
I need to delete everything in those two folder locations in order to go back to SonarQube 9.8.

32

@Nikita

We realised that there was an information we are missing here, how are you starting your instance?
As a Windows service? Or as a standalone application?

When analyzing more carefully your logs we noticed that there are a lot of paths shared between the two installations, like the temporary files folder, I think it is worth to test 9.9 as separate as possible from 9.8.
Also for the logs, having 9.9 pointing to a specific separate logs folder not shared with 9.8 would help in debugging what is going on here.

33

I’m starting as Windows service under domain account. To upgrade from previous version I follow upgrade guide (Upgrading from the ZIP file) so it’s not surprise that all versions share the same Log and Temp directories since they are located in the home dir.
I want to try to install fresh SQ 9.9 with new DB and everything on the same server, maybe it give more info. I’ll keep you posted.

34

Thanks a lot @Nikita!

All our attempts at reproducing this issue failed at the moment.
But all our tests were completed with installations living in isolation in their own folders.

What is interesting is that starting as a service you should get at least the logs of the wrapper used for launching the service. Have you checked if more logs are inside the folder of the installation C:\SonarQube\sonarqube-9.9.0.65466\logs rather than in the folder C:\SonarQube\SonarHome\logs?

35

@Matteo_Mara, after I compared config file with newly installed one line by line I found reason of the error. In our regular sonar.properties file we had line

sonar.web.javaOpts=-server -Xmx768m -Xms256m -XX:MaxPermSize=160m -XX:+HeapDumpOnOutOfMemoryError -Djava.net.preferIPv4Stack=true

As soon as I removed “-XX:MaxPermSize=160m” I successfully started SQ9.9 service.

Thank you very much for your assistance!

2 Likes
36

@Nikita thanks a lot for letting us know!
We will mimic your setup for understanding why no useful logs were printed out. Because that was an unexpected side effect.

1 Like
37

5 posts were split to a new topic: SonarQube 9.9 upgrade fails due to “Unsupported value for sonar.log.rollingPolicy: NONE”