SonarQube keeps going offline after upgrading to 9.9.1 LTS

SonarQube 9.9 LTS Upgrade
1

Hi community,

after upgrading from 8.9.9 to 9.9.1 Enterprise a few weeks ago, the server keeps going offline after an unspecified time.
Currently we’re running on Windows Server 2016 with OpenJDK Temurin 17.0.6 connected to MS SQL Server 13.0.5893. We’re using Nginx as reverse proxy.

During the DB migration no errors occurred, however afterwards we noticed that the Quality Gate page did not load, only a message ‘empty String’ showed up.
Fortunately I was able to fix that with the help of:
Issue upgrading 8.9 → 9.9 (LTS to LTS)
I found out that one cell of the column ‘value_error’ in the table ‘quality_gate_conditions’ was empty. After setting it to a valid value, the Quality Page was working again.

Since then we keep getting downtimes and because there are no errors in the log, all I can do is restarting the service. At the beginning that happened several times a day, but I could somehow improve the situation by reducing sonar.jdbc.maxLifetime from 1800000 (default) to 1200000.
I did this because of the following error message, which appeared one day after the update for the first time:

 2023.05.06 15:14:39 WARN  ce[][c.z.h.pool.PoolBase] HikariPool-1 - Failed to validate connection ConnectionID:611 ClientConnectionId: fbf91e1d-6fee-4ff6-9c18-e278745e871a (The connection is closed.). Possibly consider using a shorter maxLifetime value.

In the meantime we found out that the client connections to the server are apparently not closed anymore for some reason. The number of clients keeps increasing until some connection pool limit is reached and no new connection can be opened. (This could also explain why the service is still runing without errors and only the web service is unreachable)
We can see this in the System Info under Server Push Connections > SonarLint Connected Clients:
Connected Clients

as well as on the nginx status page and error log:
nginx_status

2023/05/09 17:20:16 [alert] 3060#3792: 1024 worker_connections are not enough

The relevant sonar.properties which are not on default are as follows:

# DATABASE
sonar.jdbc.maxActive=90
sonar.jdbc.minIdle=10
sonar.jdbc.maxLifetime=1200000
#WEB SERVER
sonar.web.javaOpts=-Xmx3G -Xms512m -XX:+HeapDumpOnOutOfMemoryError
sonar.web.http.maxThreads=75
sonar.web.http.minThreads=10

Do you have any idea what this could be or what we could try?
It’s quite hard to reconstruct everything after 3 weeks, but without a specific point it’s hard to ask for help. I’m a bit afraid by now that the database or something else got a bit more damaged than we initially thought. If there’s some kind of DB health check we can run, I would totally appreciate learning it.
Hopefully it’s just some config/infrastructure problem.

Thank you for your help!
Julian

3

Hello Julian,

Could you please give us more details on the following points:

  • Which nginx version are you using?
  • Around how many users do you have with SonarLint?
  • If possible, it would be interesting to know if it’s always the same subset of IP addresses that are not correctly closed, or if it can be any address. It might indicate a compatibility issue between specific versions of SonarLint and SonarQube.

So far we believe it might be an infrastructure issue related to nginx.

Thanks!
Eric

6

Hi Eric,

thank you for your answer.
We decided to switch to a fresh machine with Windows Server 2019 and nginx 1.25.0 since this was planned anyway. It seems to be stable now, at least the number of connections is no longer increasing so dramatically. (But that could also be because several colleagues are on vacation now due to school holidays, so we’ll have to wait for a final statement)

  • Which nginx version are you using?
    tried 1.22.0 and 1.23.4 on the old machine (both worked before), now using 1.25.0
  • Around how many users do you have with SonarLint?
    that’s hard to say, mostly because I couldn’t rely on the number viewed in the UI
    we have approx. 278 active users, today 45 SonarLint clients are connected
  • If possible, it would be interesting to know if it’s always the same subset of IP addresses that are not correctly closed, or if it can be any address. It might indicate a compatibility issue between specific versions of SonarLint and SonarQube.
    I still noticed some errors, but they don’t seem to be a bigger issue currently. If it starts again, I’ll try to track down the exact IPs. 
INFO  web[][o.s.s.p.ServerPushClient] The server push client 127.0.0.1 gone without notice, closing the connection (java.io.IOException: An established connection was aborted by the software in your host machine)

2023.06.01 10:38:36 ERROR web[AWkdMc1nuvPaq8TDXb6Y][o.s.s.w.WebServiceEngine] Fail to process request http://sonarqube/sonar/api/issues/search?resolved=false&rules=csharpsquid%3AS1451&facets=projects&ps=1&additionalFields=_all
java.lang.NullPointerException: Component has not been found for uuid 'AYQnhD2tiiq_xM9sBEnx'

Let’s wait and see how it runs. I will also be on vacation until 19th of June and I’ll try to give you an update when I’m back. At least I can probably enjoy my holidays in peace now :sunglasses:

Thank you for your help so far!
BR,
Julian

7

Hi Julian,
Thank you for the update. Please let us know if the issue comes back.

Enjoy your vacation!
Eric

8

Hi Eric,

I just wanted to let you know that the system has been running without problems since we moved.
We still don’t know the exact root cause, somehow the old machine was corrupt.
I’m just happy that it’s working again :slightly_smiling_face:

Thank you for your support!
Julian

9

Hi Julian,
That’s great to hear, thanks for letting us know!