Kubernetes, Sonarqube 8.9, database error: INSERT in a read-only transaction

[Czeslaw_Kazimierczak]

We have installed Sonarqube 8.9 from the helm chart in our Kubernetes and configured it to use an external Postgres database. We can see and after few dags of operation something happens to the database connection and it reports this error on user login:

2022.11.09 08:27:59 ERROR web[][o.s.s.p.w.RootFilter] Processing of request /api/authentication/login failed
org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: org.postgresql.util.PSQLException: ERROR: cannot execute INSERT in a read-only transaction
### The error may exist in org.sonar.db.user.SessionTokenMapper
### The error may involve org.sonar.db.user.SessionTokenMapper.insert-Inline
### The error occurred while setting parameters
### SQL: insert into session_tokens     (       uuid,       user_uuid,       expiration_date,       created_at,       updated_at     )     values (       ?,       ?,       ?,       ?,       ?     )
### Cause: org.postgresql.util.PSQLException: ERROR: cannot execute INSERT in a read-only transaction
        at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:199)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.insert(DefaultSqlSession.java:184)
        at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:62)

After restart/pod recreation all is good next few days…

Any idea what might be wrong?

[carmine.vassallo]

Dear @Czeslaw_Kazimierczak,

Thanks for reaching out! Your problem seems to be on the external database side. We actually found a StackOverflow post that might be related to the issue you’re experiencing.

Specifically, when you experience this error, you might want to check if your db suddenly goes into recovery mode (check if select pg_is_in_recovery() returns false) and if there are no read-only transactions ( show default_transaction_read_only is empty).

Of course, please let us know.

Carmine