We have an issue on 1 of our SonarQube servers which is having 3 SonarQube configured and was working fine for months but suddenly it crashed having some issues where it was having trouble connecting to the posgres.
java.lang.IllegalStateException: Fail to connect to database
Caused by: java.lang.IllegalStateException: Can not connect to database. Please check connectivity and settings (see the properties prefixed by 'sonar.jdbc.').
By aligning permissions and config we weren’t able to fix it, so we reverted the machine with backup of date the instances were running fine on. Now that it is fixed and pods are coming up fine, we are not able to access it on browser UI.
Must-share information (formatted with Markdown):
- which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension) > Community 8.3
- how is SonarQube deployed: zip, Docker, Helm > Helm
- what are you trying to achieve > Our production instances UI were inaccessible suddenly, even after the machine restore the issue did not fix, How to troubleshoot to find root cause?
- what have you tried so far to achieve this > Machine Restore, Recreating the pods on which instances were running, Restarting docker daemon processes etc.
Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!
Sonar.log >
2023.09.13 06:12:51 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2023.09.13 06:12:51 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube]: /usr/lib/jvm/java-11-amazon-corretto.x86_64/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Xmx4g -Xms4g -Djava.security.egd=file:/dev/./urandom -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/common/*:/opt/sonarqube/lib/jdbc/postgresql/postgresql-42.2.11.jar org.sonar.server.app.WebServer /opt/sonarqube/temp/sq-process10793634492405887371properties
2023.09.13 06:13:05 INFO app[][o.s.a.SchedulerImpl] Process[web] is up
2023.09.13 06:13:05 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/opt/sonarqube]: /usr/lib/jvm/java-11-amazon-corretto.x86_64/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/common/*:/opt/sonarqube/lib/jdbc/postgresql/postgresql-42.2.11.jar org.sonar.ce.app.CeServer /opt/sonarqube/temp/sq-process13342030150770185501properties
2023.09.13 06:13:11 INFO app[][o.s.a.SchedulerImpl] Process[ce] is up
2023.09.13 06:13:11 INFO app[][o.s.a.SchedulerImpl] SonarQube is up
ce.log >
2023.09.13 06:13:06 INFO ce[][o.s.p.ProcessEntryPoint] Starting ce
2023.09.13 06:13:06 INFO ce[][o.s.ce.app.CeServer] Compute Engine starting up...
2023.09.13 06:13:06 INFO ce[][o.s.c.e.CoreExtensionsLoader] Loaded core extensions: Community Branch Plugin
2023.09.13 06:13:06 INFO ce[][o.e.p.PluginsService] no modules loaded
2023.09.13 06:13:06 INFO ce[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2023.09.13 06:13:06 INFO ce[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2023.09.13 06:13:06 INFO ce[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2023.09.13 06:13:08 INFO ce[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9001]
2023.09.13 06:13:08 INFO ce[][o.sonar.db.Database] Create JDBC data source for jdbc:postgresql://exp-postgresql:5432/sonardb
2023.09.13 06:13:09 INFO ce[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /opt/sonarqube
2023.09.13 06:13:09 INFO ce[][o.s.c.c.CePluginRepository] Load plugins
2023.09.13 06:13:11 INFO ce[][o.s.c.c.ComputeEngineContainerImpl] Running Developer edition
2023.09.13 06:13:11 INFO ce[][o.s.ce.app.CeServer] Compute Engine is operational
web.log >
2023.09.13 06:13:05 INFO web[][o.s.s.q.ProjectsInWarningDaemon] Counting number of projects in warning is not started as there are no projects in this situation.
2023.09.13 06:13:05 INFO web[][o.s.s.p.p.PlatformLevelStartup] Running Community Edition
2023.09.13 06:13:05 INFO web[][o.s.s.p.Platform] WebServer is operational