after upgrading from 8.9.9 to 9.9.1 Enterprise a few weeks ago, the server keeps going offline after an unspecified time.
Currently we’re running on Windows Server 2016 with OpenJDK Temurin 17.0.6 connected to MS SQL Server 13.0.5893. We’re using Nginx as reverse proxy.
During the DB migration no errors occurred, however afterwards we noticed that the Quality Gate page did not load, only a message ‘empty String’ showed up.
Fortunately I was able to fix that with the help of:
Issue upgrading 8.9 → 9.9 (LTS to LTS)
I found out that one cell of the column ‘value_error’ in the table ‘quality_gate_conditions’ was empty. After setting it to a valid value, the Quality Page was working again.
Since then we keep getting downtimes and because there are no errors in the log, all I can do is restarting the service. At the beginning that happened several times a day, but I could somehow improve the situation by reducing
sonar.jdbc.maxLifetime from 1800000 (default) to 1200000.
I did this because of the following error message, which appeared one day after the update for the first time:
2023.05.06 15:14:39 WARN ce[c.z.h.pool.PoolBase] HikariPool-1 - Failed to validate connection ConnectionID:611 ClientConnectionId: fbf91e1d-6fee-4ff6-9c18-e278745e871a (The connection is closed.). Possibly consider using a shorter maxLifetime value.
In the meantime we found out that the client connections to the server are apparently not closed anymore for some reason. The number of clients keeps increasing until some connection pool limit is reached and no new connection can be opened. (This could also explain why the service is still runing without errors and only the web service is unreachable)
We can see this in the System Info under
Server Push Connections > SonarLint Connected Clients:
as well as on the nginx status page and error log:
2023/05/09 17:20:16 [alert] 3060#3792: 1024 worker_connections are not enough
sonar.properties which are not on default are as follows:
# DATABASE sonar.jdbc.maxActive=90 sonar.jdbc.minIdle=10 sonar.jdbc.maxLifetime=1200000 #WEB SERVER sonar.web.javaOpts=-Xmx3G -Xms512m -XX:+HeapDumpOnOutOfMemoryError sonar.web.http.maxThreads=75 sonar.web.http.minThreads=10
Do you have any idea what this could be or what we could try?
It’s quite hard to reconstruct everything after 3 weeks, but without a specific point it’s hard to ask for help. I’m a bit afraid by now that the database or something else got a bit more damaged than we initially thought. If there’s some kind of DB health check we can run, I would totally appreciate learning it.
Hopefully it’s just some config/infrastructure problem.
Thank you for your help!