SonarQube 10.1 on CentOS7
Deployed from zip file
Postgresql 15
Server was working fine. I updated Linux packages using “yum update”. SonarQube would not start. Turned out that Postgresql service became disabled after its package was updated. Enabled and restarted Postgresql service and rebooted. SonarQube got past the database error, but still does not start.
web.log has two errors:
2023.08.29 15:15:56 INFO web[][o.s.p.ProcessEntryPoint] Starting Web Server
2023.08.29 15:15:57 INFO web[][o.s.s.p.LogServerVersion] SonarQube Server / 10.1.0.73491 / 53c01c35c264c7e3d76cf5fb955de406f36b115e
2023.08.29 15:15:57 INFO web[][o.s.d.DefaultDatabase] Create JDBC data source for jdbc:postgresql://localhost/sonar
2023.08.29 15:15:57 INFO web[][c.z.h.HikariDataSource] HikariPool-1 - Starting...
2023.08.29 15:15:57 INFO web[][c.z.h.p.HikariPool] HikariPool-1 - Added connection org.postgresql.jdbc.PgConnection@726a17c4
2023.08.29 15:15:57 INFO web[][c.z.h.HikariDataSource] HikariPool-1 - Start completed.
2023.08.29 15:15:58 INFO web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /opt/sonarqube-10.1.0.73491
2023.08.29 15:15:58 INFO web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy C# Code Quality and Security / 9.3.0.71466 / e47cf88a6286a446a098754e5775535a330f58d7
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Clean as You Code / 2.0.0.334 / 424c67de80b14c8c1b168104a6749b9d280ef4f9
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Configuration detection fot Code Quality and Security / 1.2.0.267 / 4f37ba9ffb37a96d5883e52ad392ed32c5c6eaab
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Flex Code Quality and Security / 2.9.0.3375 / 3fb9c557effa2d3f6e4017984611e8dae60d030a
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Go Code Quality and Security / 1.13.0.4374 / dc92e7c248e31ecae6e18d94781588cc6264e517
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy HTML Code Quality and Security / 3.8.0.3510 / c8f3d4a50984c6a0b0699616f98ce81a9d2893db
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy IaC Code Quality and Security / 1.17.0.3976 / 5089f396ccda1f2ea0a6f2961b10042ea8c9cd69
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy JaCoCo / 1.3.0.1538 / 74a7798c7cea687c72ed9df40c93eb7ea2a58c49
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Java Code Quality and Security / 7.20.0.31692 / b40c64f1075774f7232608df6636586009ab8ab3
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy JavaScript/TypeScript/CSS Code Quality and Security / 10.3.1.21905 / f95f3d172103f7e438556705c96fb336b33900b4
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Kotlin Code Quality and Security / 2.15.0.2579 / 6ea5707d5a5600fc808653075f9d9cb6c21a4788
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy PHP Code Quality and Security / 3.30.0.9766 / 783200fd5af4e0f8ccbf027a97ebb5f377d5014f
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Python Code Quality and Security / 4.3.0.11660 / 7914a5be1ad88eab1ebd2a3c2f7e606404601e30
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Ruby Code Quality and Security / 1.13.0.4374 / dc92e7c248e31ecae6e18d94781588cc6264e517
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Scala Code Quality and Security / 1.13.0.4374 / dc92e7c248e31ecae6e18d94781588cc6264e517
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy Text Code Quality and Security / 2.1.0.1163 / 849cef77adc8d1a50235d2ef7005e23f67e2838a
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy VB.NET Code Quality and Security / 9.3.0.71466 / e47cf88a6286a446a098754e5775535a330f58d7
2023.08.29 15:15:59 INFO web[][o.s.s.p.ServerPluginManager] Deploy XML Code Quality and Security / 2.8.1.4006 / 80e5629b173cd7a47c04e97cbe1a263db84ff844
2023.08.29 15:16:00 INFO web[][o.s.s.p.d.m.c.PostgresCharsetHandler] Verify that database charset supports UTF8
2023.08.29 15:16:00 INFO web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [http://localhost:9001]
2023.08.29 15:16:01 WARN web[][o.s.a.s.w.WebService$Action] Description is not set on action api/monitoring/metrics
2023.08.29 15:16:01 WARN web[][o.s.a.s.w.WebService$Action] Since is not set on action api/monitoring/metrics
2023.08.29 15:16:01 WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/monitoring/metrics
2023.08.29 15:16:01 WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/system/liveness
2023.08.29 15:16:01 INFO web[][o.s.s.p.DetectPluginChange] Detect plugin changes
2023.08.29 15:16:01 INFO web[][o.s.s.p.DetectPluginChange] No plugin change detected
2023.08.29 15:16:02 INFO web[][o.s.s.s.LogServerId] Server ID: A7EE8CF2-AW2S23ox99jsq_1q6FNX
2023.08.29 15:16:02 INFO web[][o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties
2023.08.29 15:16:03 INFO web[][o.s.a.l.LdapSettingsManager] User mapping: LdapUserMapping{2023.08.29 15:16:03 INFO web[][o.s.a.l.LdapSettingsManager] User mapping: LdapUserMapping{OBFUSCATED}
2023.08.29 15:16:03 INFO web[][o.s.a.l.LdapSettingsManager] Groups will not be synchronized, because property 'ldap.group.baseDn' is empty.
2023.08.29 15:16:03 INFO web[][o.s.a.l.LdapContextFactory] Test LDAP connection on ldaps://OBFUSCATED: OK
2023.08.29 15:16:03 WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action saml/validation_init
2023.08.29 15:16:03 WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/system/liveness
2023.08.29 15:16:03 WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/plugins/download
2023.08.29 15:16:03 WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/analysis_cache/get
2023.08.29 15:16:03 WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/alm_integrations/check_pat
2023.08.29 15:16:03 WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/push/sonarlint_events
2023.08.29 15:16:03 WARN web[][o.s.a.s.w.WebService$Action] Since is not set on action api/cayc/issues_creation_histogram
2023.08.29 15:16:03 WARN web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/cayc/issues_creation_histogram
2023.08.29 15:16:03 WARN web[][o.s.s.p.d.CheckAnyonePermissionsAtStartup] Authentication is not enforced, and permissions assigned to the 'Anyone' group globally expose the instance to security risks. Unauthenticated visitors may unintentionally have permissions on projects.
2023.08.29 15:16:03 INFO web[][o.s.s.a.p.ExpiredSessionsCleaner] Purge of expired session tokens has removed 0 elements
2023.08.29 15:16:03 INFO web[][o.s.s.a.p.ExpiredSessionsCleaner] Purge of expired SAML message ids has removed 0 elements
2023.08.29 15:16:03 INFO web[][o.s.s.n.NotificationDaemon] Notification service started (delay 60 sec.)
2023.08.29 15:16:03 INFO web[][o.s.s.t.TelemetryDaemon] Sharing of SonarQube statistics is enabled.
2023.08.29 15:16:04 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
java.lang.NullPointerException: Cannot invoke "javax.servlet.ServletConfig.getServletContext()" because the return value of "javax.servlet.GenericServlet.getServletConfig()" is null
at javax.servlet.GenericServlet.getServletContext(GenericServlet.java:123)
at org.springframework.web.servlet.FrameworkServlet.destroy(FrameworkServlet.java:863)
at org.sonar.server.platform.web.ApiV2Servlet.destroyDispatcherSafeMode(ApiV2Servlet.java:121)
at org.sonar.server.platform.web.ApiV2Servlet.initDispatcherLevel4(ApiV2Servlet.java:81)
at org.sonar.server.platform.PlatformImpl$1.lambda$doRun$0(PlatformImpl.java:115)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:362)
at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:115)
at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:346)
at java.base/java.lang.Thread.run(Thread.java:833)
2023.08.29 15:16:04 INFO web[][o.s.p.ProcessEntryPoint] Hard stopping process
2023.08.29 15:16:05 WARN web[][o.s.p.ProcessEntryPoint$HardStopperThread] Can not stop in 1000ms
2023.08.29 15:19:41 WARN web[][o.a.c.u.SessionIdGeneratorBase] Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [220,424] milliseconds.
2023.08.29 15:19:41 INFO web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.platform.web.WebServiceFilter@6cc64028 [pattern=UrlPattern{inclusions=[/api/system/migrate_db.*, ...], exclusions=[/api/components/update_key, ...]}]
2023.08.29 15:19:41 INFO web[][o.s.w.s.DispatcherServlet] Initializing Servlet 'app'
2023.08.29 15:19:42 INFO web[][o.s.w.s.DispatcherServlet] Completed initialization in 895 ms
2023.08.29 15:19:42 INFO web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2023.08.29 15:19:42 WARN web[][o.s.s.t.TelemetryDaemon] Termination of pool TelemetryDaemon failed
java.lang.InterruptedException: null
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1660)
at java.base/java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1464)
at java.base/java.util.concurrent.Executors$DelegatedExecutorService.awaitTermination(Executors.java:743)
at org.sonar.server.util.AbstractStoppableExecutorService.stop(AbstractStoppableExecutorService.java:52)
at org.sonar.core.platform.StartableBeanPostProcessor.postProcessBeforeDestruction(StartableBeanPostProcessor.java:48)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:197)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1163)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1156)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1108)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1077)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1023)
at org.sonar.core.platform.SpringComponentContainer.stopComponents(SpringComponentContainer.java:196)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.sonar.core.platform.SpringComponentContainer.stopChildren(SpringComponentContainer.java:208)
at org.sonar.core.platform.SpringComponentContainer.stopComponents(SpringComponentContainer.java:194)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.sonar.core.platform.SpringComponentContainer.stopChildren(SpringComponentContainer.java:208)
at org.sonar.core.platform.SpringComponentContainer.stopComponents(SpringComponentContainer.java:194)
at org.sonar.server.platform.platformlevel.PlatformLevel.stop(PlatformLevel.java:88)
at org.sonar.server.platform.PlatformImpl.stopLevel234Containers(PlatformImpl.java:238)
at org.sonar.server.platform.PlatformImpl.doStop(PlatformImpl.java:267)
at org.sonar.server.platform.web.PlatformServletContextListener.contextDestroyed(PlatformServletContextListener.java:68)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4540)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5179)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1348)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1337)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:936)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1348)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1337)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:936)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.core.StandardService.stopInternal(StandardService.java:491)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:970)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:257)
at org.apache.catalina.startup.Tomcat.stop(Tomcat.java:496)
at org.sonar.server.app.EmbeddedTomcat.terminate(EmbeddedTomcat.java:105)
at org.sonar.server.app.WebServer.hardStop(WebServer.java:86)
at org.sonar.process.ProcessEntryPoint$HardStopperThread.lambda$new$0(ProcessEntryPoint.java:219)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
2023.08.29 15:19:42 INFO web[][o.s.s.n.NotificationDaemon] Notification service stopped
2023.08.29 15:19:42 INFO web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown initiated...
2023.08.29 15:19:42 INFO web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown completed.
I don’t see other errors in the logs. I tried to set sonar.log.level to DEBUG, but that did not shed more light on the cause to me. What can I do to find the cause of the problem?
Do you know what version of Java is installed on your server after the yum update (the output of java --version`).
SonarQube 10.1 only officially supports Java 17 as the runtime… so I wonder if there’s some compatibility issue going on if Java updated itself to Java 20, for example.
Are you getting this error consistently when starting SonarQube?
Can you check on the sonar.log / nohup.log file and give us the exact Java version that is used to run the application?
What is the SonarQube version that you are running? Are you using a multiple-nodes setup?
Yes. Consistent error after multiple reboots. SonarQube also appears to try to restart non-stop and fails in the same way, so the two errors I posted repeat in the log.
SonarQube version is 10.1.0.73491.
Not using multiple nodes.
nohup.log:
2023.08.31 10:39:35 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube-10.1.0.73491/temp
2023.08.31 10:39:35 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:41224]
2023.08.31 10:39:35 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/opt/sonarqube-10.1.0.73491/elasticsearch]: /usr/lib/jvm/java-17-amazon-corretto/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/opt/sonarqube-10.1.0.73491/elasticsearch -Des.path.conf=/opt/sonarqube-10.1.0.73491/temp/conf/es -Des.distribution.type=tar -cp /opt/sonarqube-10.1.0.73491/elasticsearch/lib/*:/opt/sonarqube-10.1.0.73491/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
2023.08.31 10:39:35 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023.08.31 10:39:46 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2023.08.31 10:39:46 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [/opt/sonarqube-10.1.0.73491]: /usr/lib/jvm/java-17-amazon-corretto/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube-10.1.0.73491/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-10.1.0.73491.jar:/opt/sonarqube-10.1.0.73491/lib/jdbc/postgresql/postgresql-42.6.0.jar org.sonar.server.app.WebServer /opt/sonarqube-10.1.0.73491/temp/sq-process17729179442898305935properties
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.sonar.process.PluginSecurityManager (file:/opt/sonarqube-10.1.0.73491/lib/sonar-application-10.1.0.73491.jar)
WARNING: Please consider reporting this to the maintainers of org.sonar.process.PluginSecurityManager
WARNING: System::setSecurityManager will be removed in a future release
sonar.log:
2023.08.31 10:33:36 INFO app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.08.31 10:33:36 INFO app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.08.31 10:33:36 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 143
2023.08.31 10:33:36 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
2023.08.31 10:33:37 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube-10.1.0.73491/temp
2023.08.31 10:33:37 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:40014]
2023.08.31 10:33:37 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/opt/sonarqube-10.1.0.73491/elasticsearch]: /usr/lib/jvm/java-17-amazon-corretto/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/opt/sonarqube-10.1.0.73491/elasticsearch -Des.path.conf=/opt/sonarqube-10.1.0.73491/temp/conf/es -Des.distribution.type=tar -cp /opt/sonarqube-10.1.0.73491/elasticsearch/lib/*:/opt/sonarqube-10.1.0.73491/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
2023.08.31 10:33:37 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023.08.31 10:33:47 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2023.08.31 10:33:47 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [/opt/sonarqube-10.1.0.73491]: /usr/lib/jvm/java-17-amazon-corretto/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube-10.1.0.73491/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-10.1.0.73491.jar:/opt/sonarqube-10.1.0.73491/lib/jdbc/postgresql/postgresql-42.6.0.jar org.sonar.server.app.WebServer /opt/sonarqube-10.1.0.73491/temp/sq-process12512761400800444852properties
2023.08.31 10:39:34 INFO app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.08.31 10:39:34 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 143
2023.08.31 10:39:34 INFO app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.08.31 10:39:34 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
2023.08.31 10:39:35 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube-10.1.0.73491/temp
2023.08.31 10:39:35 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:41224]
2023.08.31 10:39:35 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/opt/sonarqube-10.1.0.73491/elasticsearch]: /usr/lib/jvm/java-17-amazon-corretto/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/opt/sonarqube-10.1.0.73491/elasticsearch -Des.path.conf=/opt/sonarqube-10.1.0.73491/temp/conf/es -Des.distribution.type=tar -cp /opt/sonarqube-10.1.0.73491/elasticsearch/lib/*:/opt/sonarqube-10.1.0.73491/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
2023.08.31 10:39:35 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2023.08.31 10:39:46 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2023.08.31 10:39:46 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[WEB_SERVER] from [/opt/sonarqube-10.1.0.73491]: /usr/lib/jvm/java-17-amazon-corretto/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube-10.1.0.73491/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.management/sun.management=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dcom.redhat.fips=false -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/sonar-application-10.1.0.73491.jar:/opt/sonarqube-10.1.0.73491/lib/jdbc/postgresql/postgresql-42.6.0.jar org.sonar.server.app.WebServer /opt/sonarqube-10.1.0.73491/temp/sq-process17729179442898305935properties
This is a weird behavior, and this is the first time that it is being reported.
I was not able to reproduce it on a fully updated CentOs7 with java 17 Corretto.
Would you be able to identify what libraries were updated during the yum update ?
Since Postgres was shutdown after update, I guess that it has also been updated. Could you please check what version it was updated to?
---> Package apr-util.x86_64 0:1.5.2-6.el7 will be updated
---> Package apr-util.x86_64 0:1.5.2-6.el7_9.1 will be an update
---> Package bind-export-libs.x86_64 32:9.11.4-26.P2.el7_9.10 will be updated
---> Package bind-export-libs.x86_64 32:9.11.4-26.P2.el7_9.14 will be an update
---> Package cronie.x86_64 0:1.4.11-24.el7_9 will be updated
---> Package cronie.x86_64 0:1.4.11-25.el7_9 will be an update
---> Package cronie-anacron.x86_64 0:1.4.11-24.el7_9 will be updated
---> Package cronie-anacron.x86_64 0:1.4.11-25.el7_9 will be an update
---> Package diffutils.x86_64 0:3.3-5.el7 will be updated
---> Package diffutils.x86_64 0:3.3-6.el7_9 will be an update
---> Package httpd.x86_64 0:2.4.6-97.el7.centos.5 will be updated
---> Package httpd.x86_64 0:2.4.6-99.el7.centos.1 will be an update
---> Package httpd-tools.x86_64 0:2.4.6-97.el7.centos.5 will be updated
---> Package httpd-tools.x86_64 0:2.4.6-99.el7.centos.1 will be an update
---> Package java-11-openjdk-headless.x86_64 1:11.0.17.0.8-2.el7_9 will be updated
---> Package java-11-openjdk-headless.x86_64 1:11.0.20.0.8-1.el7_9 will be an update
---> Package kernel.x86_64 0:3.10.0-1160.95.1.el7 will be installed
---> Package kernel-tools.x86_64 0:3.10.0-1160.81.1.el7 will be updated
---> Package kernel-tools.x86_64 0:3.10.0-1160.95.1.el7 will be an update
---> Package kernel-tools-libs.x86_64 0:3.10.0-1160.81.1.el7 will be updated
---> Package kernel-tools-libs.x86_64 0:3.10.0-1160.95.1.el7 will be an update
---> Package libXpm.x86_64 0:3.5.12-1.el7 will be updated
---> Package libXpm.x86_64 0:3.5.12-2.el7_9 will be an update
---> Package libzstd.x86_64 0:1.5.2-1.el7 will be updated
---> Package libzstd.x86_64 0:1.5.5-1.el7 will be an update
---> Package microcode_ctl.x86_64 2:2.1-73.15.el7_9 will be updated
---> Package microcode_ctl.x86_64 2:2.1-73.16.el7_9 will be an update
---> Package mod_ssl.x86_64 1:2.4.6-97.el7.centos.5 will be updated
---> Package mod_ssl.x86_64 1:2.4.6-99.el7.centos.1 will be an update
---> Package nss.x86_64 0:3.79.0-4.el7_9 will be updated
---> Package nss.x86_64 0:3.79.0-5.el7_9 will be an update
---> Package nss-pem.x86_64 0:1.0.3-7.el7 will be updated
---> Package nss-pem.x86_64 0:1.0.3-7.el7_9.1 will be an update
---> Package nss-sysinit.x86_64 0:3.79.0-4.el7_9 will be updated
---> Package nss-sysinit.x86_64 0:3.79.0-5.el7_9 will be an update
---> Package nss-tools.x86_64 0:3.79.0-4.el7_9 will be updated
---> Package nss-tools.x86_64 0:3.79.0-5.el7_9 will be an update
---> Package open-vm-tools.x86_64 0:11.0.5-3.el7_9.4 will be updated
---> Package open-vm-tools.x86_64 0:11.0.5-3.el7_9.6 will be an update
---> Package openssh.x86_64 0:7.4p1-22.el7_9 will be updated
---> Package openssh.x86_64 0:7.4p1-23.el7_9 will be an update
---> Package openssh-clients.x86_64 0:7.4p1-22.el7_9 will be updated
---> Package openssh-clients.x86_64 0:7.4p1-23.el7_9 will be an update
---> Package openssh-server.x86_64 0:7.4p1-22.el7_9 will be updated
---> Package openssh-server.x86_64 0:7.4p1-23.el7_9 will be an update
---> Package openssl.x86_64 1:1.0.2k-25.el7_9 will be updated
---> Package openssl.x86_64 1:1.0.2k-26.el7_9 will be an update
---> Package openssl-libs.x86_64 1:1.0.2k-25.el7_9 will be updated
---> Package openssl-libs.x86_64 1:1.0.2k-26.el7_9 will be an update
---> Package pgdg-redhat-repo.noarch 0:42.0-28 will be updated
---> Package pgdg-redhat-repo.noarch 0:42.0-32 will be an update
---> Package postgresql15.x86_64 0:15.1-1PGDG.rhel7 will be updated
---> Package postgresql15.x86_64 0:15.4-1PGDG.rhel7 will be an update
---> Package postgresql15-libs.x86_64 0:15.1-1PGDG.rhel7 will be updated
---> Package postgresql15-libs.x86_64 0:15.4-1PGDG.rhel7 will be an update
---> Package postgresql15-server.x86_64 0:15.1-1PGDG.rhel7 will be updated
---> Package postgresql15-server.x86_64 0:15.4-1PGDG.rhel7 will be an update
---> Package python.x86_64 0:2.7.5-92.el7_9 will be updated
---> Package python.x86_64 0:2.7.5-93.el7_9 will be an update
---> Package python-libs.x86_64 0:2.7.5-92.el7_9 will be updated
---> Package python-libs.x86_64 0:2.7.5-93.el7_9 will be an update
---> Package python-perf.x86_64 0:3.10.0-1160.81.1.el7 will be updated
---> Package python-perf.x86_64 0:3.10.0-1160.95.1.el7 will be an update
---> Package sudo.x86_64 0:1.8.23-10.el7_9.2 will be updated
---> Package sudo.x86_64 0:1.8.23-10.el7_9.3 will be an update
---> Package tzdata.noarch 0:2022g-1.el7 will be updated
---> Package tzdata.noarch 0:2023c-1.el7 will be an update
---> Package tzdata-java.noarch 0:2022g-1.el7 will be updated
---> Package tzdata-java.noarch 0:2023c-1.el7 will be an update
---> Package zlib.x86_64 0:1.2.7-20.el7_9 will be updated
---> Package zlib.x86_64 0:1.2.7-21.el7_9 will be an update
Postgres version from SELECT version(); command:
PostgreSQL 15.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (R
ed Hat 4.8.5-44), 64-bit
We investigated internally and could reproduce this by forcing the system to be artificially slow.
It looks like your issue is related to the lack of performance of the system running SonarQube.
Could you make sure the machine running SonarQube has enough memory / CPU available to run the processes? Ideally, a dedicated machine should be used to avoid conflicts with other processes.
If this does not work and - as the problem happened after the upgrade of Postgres - it might also be that the performance of the machine hosting your Postgres DB are not good enough.
Thank you for the hints. We had 4 cores and 4 GB of memory on the VM. We tried to increase to 8 GB and then to 8 cores and 16 GB. It did not help. We did the same on our production SonarQube server, which was NOT updated with “yum update”. The only thing done to it was power off, change CPU count/memory and power on. It now has the same problem as I reported for the dev server. We also see memory consumption go to 12 GB on startup. I guess that this is when SonarQube attempts to start.
AUTOVACUUM is on for the db. Both servers have about 5 GB of free disk space out of 11 GB total.
Each server is a separate VM that hosts PSQL database (only used by SonarQube) and SonarQube itself. Nothing else is on these servers. These are both virtual machines on the same host. There are 15 other VMs on the same host, but we do not exceed 50% of CPU usage and there’s about 50 GB of free memory on the host. I don’t think we are choking the host with performance requirements.
Do I understand correctly that your production server is now unable to start?
We did some improvement on memory usage at startup with version 10.2. You could try this one on your non-production instance. This could help to unlock the situation.
Thank you for that suggestion! I upgraded our test instance and it started fine. I then ran “yum update” and upgraded SonarQube on the production instance. It started without problems. So everything is resolved now. It’s strange how we got into that error state, but I’m happy that everything is working after the upgrade. Thanks again.
