After a node drain in my kubernetes cluster, sonarqube server wont start.
Errors are mulitple, here are two examples (see bottom for full logs):
Here is the first one:
2025.03.13 13:06:07 INFO web[][o.s.s.p.p.PlatformLevelStartup] Running Developer Edition
2025.03.13 13:06:07 WARN web[][c.z.h.p.ProxyConnection] HikariPool-1 - Connection org.postgresql.jdbc.PgConnection@21e20ad5 marked as broken because of SQLSTATE(08P01), ErrorCode(0)
org.postgresql.util.PSQLException: Expected command status BEGIN, got COMMIT.
at org.postgresql.core.v3.QueryExecutorImpl$1.handleCommandStatus(QueryExecutorImpl.java:643)
at org.postgresql.core.v3.QueryExecutorImpl.interpretCommandStatus(QueryExecutorImpl.java:2778)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2368)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:372)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:517)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:434)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:194)
at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:180)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.execute(HikariProxyPreparedStatement.java)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:65)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:80)
at org.apache.ibatis.executor.ReuseExecutor.doQuery(ReuseExecutor.java:62)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:336)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:158)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:110)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:90)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:154)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:142)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:75)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87)
at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:141)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
at jdk.proxy2/jdk.proxy2.$Proxy39.selectByKey(Unknown Source)
at org.sonar.db.property.PropertiesDao.selectGlobalProperty(PropertiesDao.java:132)
at org.sonar.db.property.PropertiesDao.selectGlobalProperty(PropertiesDao.java:138)
at org.sonar.server.setting.DatabaseSettingLoader.load(DatabaseSettingLoader.java:40)
at org.sonar.server.setting.ThreadLocalSettings.load(ThreadLocalSettings.java:128)
at org.sonar.server.setting.ThreadLocalSettings.get(ThreadLocalSettings.java:109)
at org.sonar.api.config.internal.Settings.getRawString(Settings.java:88)
at org.sonar.api.config.internal.Settings.getString(Settings.java:140)
at org.sonar.server.config.ConfigurationProvider$ServerConfigurationAdapter.get(ConfigurationProvider.java:51)
at org.sonar.api.config.Configuration.getInt(Configuration.java:137)
at com.sonarsource.i.D.殊(Unknown Source)
at com.sonarsource.i.f.Y.쨭(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2025.03.13 13:06:07 INFO app[][o.s.a.SchedulerImpl] Process[web] is up
Then i have lots of those
### Error querying database. Cause: org.postgresql.util.PSQLException: ERROR: prepared statement "S_14" already exists
### The error may exist in org.sonar.db.property.PropertiesMapper
### The error may involve org.sonar.db.property.PropertiesMapper.selectByKey-Inline
### The error occurred while setting parameters
### SQL: select p.prop_key as "key", p.is_empty as empty, p.text_value as textValue, p.clob_value as clobValue, p.entity_uuid as entityUuid, p.user_uuid as userUuid from properties p where p.prop_key=? and p.entity_uuid is null and p.user_uuid is null
### Cause: org.postgresql.util.PSQLException: ERROR: prepared statement "S_14" already exists
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:156)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:142)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:75)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87)
at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:141)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
at jdk.proxy2/jdk.proxy2.$Proxy39.selectByKey(Unknown Source)
at org.sonar.db.property.PropertiesDao.selectGlobalProperty(PropertiesDao.java:132)
at org.sonar.db.property.PropertiesDao.selectGlobalProperty(PropertiesDao.java:138)
at org.sonar.server.setting.ws.LoginMessageAction.isMessageDisplayEnabled(LoginMessageAction.java:69)
at org.sonar.server.setting.ws.LoginMessageAction.handle(LoginMessageAction.java:58)
at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:114)
at org.sonar.server.platform.web.WebServiceFilter.doFilter(WebServiceFilter.java:84)
at org.sonar.server.platform.web.MasterServletFilter$JavaxFilterAdapter.doFilter(MasterServletFilter.java:194)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:165)
at org.sonar.server.platform.web.MasterServletFilter$HttpFilterChainAdapter.doFilter(MasterServletFilter.java:208)
at org.sonar.server.platform.web.SonarLintConnectionFilter.doFilter(SonarLintConnectionFilter.java:66)
at org.sonar.server.platform.web.MasterServletFilter$JavaxFilterAdapter.doFilter(MasterServletFilter.java:194)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:165)
at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:126)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:83)
at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.sonar.server.platform.web.CspFilter.doFilter(CspFilter.java:67)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:60)
at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:56)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.sonar.server.platform.web.EndpointPathFilter.doFilter(EndpointPathFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.sonar.server.platform.web.RequestIdFilter.doFilter(RequestIdFilter.java:66)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:65)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:115)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:268)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:905)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: org.sonar.api.utils.text.WriterException: Fail to write JSON
at org.sonar.api.utils.text.JsonWriter.rethrow(JsonWriter.java:399)
at org.sonar.api.utils.text.JsonWriter.close(JsonWriter.java:394)
at org.sonar.server.setting.ws.LoginMessageAction.handle(LoginMessageAction.java:56)
... 57 common frames omitted
Caused by: java.io.IOException: Incomplete document
at org.sonar.api.internal.google.gson.stream.JsonWriter.close(JsonWriter.java:713)
at org.sonar.api.utils.text.JsonWriter.close(JsonWriter.java:392)
... 58 common frames omitted
Sonar Server was installed with helm:
helm repo add sonarqube https://sonarSource.github.io/helm-chart-sonarqube
helm repo update
kubectl create namespace sonarqube
helm upgrade --install -n sonarqube sonarqube sonarqube/sonarqube -f values.yaml
#values.yaml
# Configure the edition of SonarQube Server to deploy: developer or enterprise
edition: "developer"
appVersion: 2025.1.0
image:
pullPolicy: IfNotPresent
repository: sonarqube
tag: 2025.1.0-developer
ingress-nginx:
enabled: false
httproute:
enabled: false
nodeSelector:
k8s.scaleway.com/sonarqube: "true"
ingress:
enabled: true
ingressClassName: "nginx"
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-body-size: 64m
hosts:
- name: sonarqube.XXXXXXXXXXXXX
path: "/"
pathType: "Prefix"
tls:
- hosts:
- sonarqube.XXXXXXXXXXXXXXXXX
secretName: sonarqube-tls-secret
## a monitoring passcode needs to be defined in order to get reasonable probe results
# not setting the monitoring passcode will result in a deployment that will never be ready
monitoringPasscode: "XXXXXXXXXXXXXXXXXXXX"
resources:
limits:
cpu: 800m
memory: 6144M
ephemeral-storage: 512000M
requests:
cpu: 400m
memory: 2048M
ephemeral-storage: 1536M
persistence:
enabled: false
jdbcOverwrite:
enabled: true
jdbcUrl: "jdbc:postgresql://postgres-db-pgbouncer.heyliot-postgres-ops.svc.cluster.local/sonar"
jdbcUsername: "sonar"
jdbcPassword: "XXXXXXXXXXXXXXXXXXXXXX"
postgresql:
enabled: false
serviceAccount:
create: false
setAdminPassword:
newPassword: XXXXXXXXXXXXXXXXX
currentPassword: admin
I tried multiple times to restart the statefulset.
I tried multiple time to empty sql statement.
Steps i followed:
- Stop sonarqube (kubectl scale deployment sonarqube --replicas=0)
- Wait for pg connexion to ble closed (through pg_stat_activity)
- Clean prepared statements with
DEALLOCATE ALL;sql command - Start soanrqube again
Nothing works.
Here are 2 log files (1 restart between both)
- all-pods.log (165.8 KB)
- all-pods_2.log (114.5 KB)