SonarQube sometimes can't connect to Postgres (Docker Setup)

SonarQube
, ,
1

Hi,

I’m running the latest official docker image for SonarQube (8.8 at time of writing) and everything appears to be working as expected. I can add projects, run analysis whatnot.

However, I’ve noticed my CI/CD pipeline sometimes randomly failing with " SonarQube server can not be reached". This seems to happen when Dependabot bumps a group of projects and they all send their analysis’ to Sonar in a short period of time.

While this is going on, the Sonar UI is still working and responsive. Nothing indicates anything is wrong.

Taking a look at the logs from Sonar, it looks like it’s dropping the database connection and unable to reconnect.

Caused by: org.postgresql.util.PSQLException: Connection to sonarqube-postgres:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:303)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223)
	at org.postgresql.Driver.makeConnection(Driver.java:465)
	at org.postgresql.Driver.connect(Driver.java:264)
	at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55)
	at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355)
	at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
	at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
	at org.apache.commons.dbcp2.BasicDataSource$PaGetConnection.run(BasicDataSource.java:73)
	at org.apache.commons.dbcp2.BasicDataSource$PaGetConnection.run(BasicDataSource.java:69)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:744)
	at org.sonar.db.profiling.NullConnectionInterceptor.getConnection(NullConnectionInterceptor.java:31)
	at org.sonar.db.profiling.ProfiledDataSource.getConnection(ProfiledDataSource.java:317)
	at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:139)
	at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:61)
	at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:337)
	at org.apache.ibatis.executor.ReuseExecutor.prepareStatement(ReuseExecutor.java:88)
	at org.apache.ibatis.executor.ReuseExecutor.doQuery(ReuseExecutor.java:59)
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
	... 23 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
	at java.base/java.net.Socket.connect(Unknown Source)
	at org.postgresql.core.PGStream.createSocket(PGStream.java:231)
	at org.postgresql.core.PGStream.<init>(PGStream.java:95)
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:98)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
	... 49 common frames omitted

If I take a look a the database logs:

PostgreSQL Database directory appears to contain a database; Skipping initialization

2021-04-12 15:11:23.762 UTC [1] LOG:  starting PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-04-12 15:11:23.763 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2021-04-12 15:11:23.763 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2021-04-12 15:11:23.767 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-04-12 15:11:23.775 UTC [26] LOG:  database system was shut down at 2021-04-12 15:11:21 UTC
2021-04-12 15:11:23.784 UTC [1] LOG:  database system is ready to accept connections

Everything seems to be in order.

Here is how I’m running both Sonar and Postgres behind a reverse proxy with context path mapping, have I just messed up somewhere?

Is there a way I can debug this further and work out where the problem sits?

2

Hello @peavers and welcome to the community :wave:

your config looks okay for now even thou i would recommend you switching to our environment variables for configuration and maybe remove the port forwarding of sonarqube in favor of routing everything throu traefik.

I assume that the connection drop is a result of a burst of resources. you can try to set limits to the sonarqube deployment to make sure that on high load there is enough cpu/ram available for underlying the system processes:

  sonarqube:
    container_name: sonarqube
    image: sonarqube:latest
    restart: unless-stopped
    cpus: 0.5
    mem_limit: 4096M
    mem_reservation: 1024M

hope that helps

3

Thanks a lot for the response, I did wonder if CPU/Memory was an issue so I’ll implement your suggestions and report back.

Quick question though: I was changing over to environment variables as suggested but I can’t find the var for to replace sonar.core.serverBaseURL, do you happen to know what this should be?

4

This setting should be configured in the UI as it will persist in the database, so there is no need to have it as an env var permanently. if you want to set it automatically you could use the API for this (api/settings/set).