Sonarqube instantly stops after starting

Reno_Chung · February 26, 2021, 8:30am

Environment :
centos-release-7-6.1810.2.el7.centos.x86_64

Sonarqube version :
I do not know how to check the version
web.xml says this:

<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://java.sun.com/xml/ns/javaee"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         id="SonarQube"
         version="3.0"
         metadata-complete="true">

Symptom:

Background:
Sonarqube was running before I joined this company and was never stopped/used before I did today.

web.log

2021.02.26 16:19:40 WARN web[o.a.c.l.WebappClassLoaderBase] The web application [sonar] appears to have started a thread named [elasticsearch[client][transport_worker][T#7]] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
[java.base@11.0.9.1]/sun.nio.ch.EPoll.wait(Native Method)
[java.base@11.0.9.1]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
[java.base@11.0.9.1]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
[java.base@11.0.9.1]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
app//io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:765)
app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:413)
app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
[java.base@11.0.9.1]/java.lang.Thread.run(Thread.java:834)
2021.02.26 16:19:40 WARN web[o.a.c.l.WebappClassLoaderBase] The web application [sonar] appears to have started a thread named [elasticsearch[client][transport_worker][T#8]] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
[java.base]@11.0.9.1/sun.nio.ch.EPoll.wait(Native Method)
[java.base]@11.0.9.1/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
[java.base]@11.0.9.1/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
[java.base]@11.0.9.1/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
app//io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:765)
app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:413)
app//io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
[java.base@11.0.9.1]/java.lang.Thread.run(Thread.java:834)
2021.02.26 16:19:40 INFO web[o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2021.02.26 16:19:40 INFO web[o.s.p.ProcessEntryPoint] Hard stopping process

es.log

2021.02.26 15:23:33 INFO  es[][o.e.n.Node] closed
2021.02.26 16:19:26 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [104gb], net total_space [492gb], types [rootfs]
2021.02.26 16:19:26 INFO  es[][o.e.e.NodeEnvironment] heap size [494.9mb], compressed ordinary object pointers [true]
2021.02.26 16:19:26 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [2NHf4TSERHGrR2s-napywQ]
2021.02.26 16:19:26 INFO  es[][o.e.n.Node] version[6.8.0], pid[32223], build[default/tar/65b6179/2019-05-15T20:06:13.172855Z], OS[Linux/3.10.0-1127.19.1.el7.x86_64/amd64], JVM[Red Hat, Inc./OpenJDK 64-Bit Server VM/11.0.9.1/11.0.9.1+1-LTS]
2021.02.26 16:19:26 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/opt/sonarqube/temp, -XX:ErrorFile=../logs/es_hs_err_pid%p.log, -Des.enforce.bootstrap.checks=true, -Xms512m, -Xmx512m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/elasticsearch, -Des.path.conf=/opt/sonarqube/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar]
2021.02.26 16:19:27 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2021.02.26 16:19:27 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2021.02.26 16:19:27 INFO  es[][o.e.p.PluginsService] loaded module [mapper-extras]
2021.02.26 16:19:27 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2021.02.26 16:19:27 INFO  es[][o.e.p.PluginsService] loaded module [percolator]
2021.02.26 16:19:27 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2021.02.26 16:19:27 INFO  es[][o.e.p.PluginsService] loaded module [repository-url]
2021.02.26 16:19:27 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2021.02.26 16:19:27 INFO  es[][o.e.p.PluginsService] no plugins loaded
2021.02.26 16:19:29 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2021.02.26 16:19:31 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and host providers [settings]
2021.02.26 16:19:31 INFO  es[][o.e.n.Node] initialized
2021.02.26 16:19:31 INFO  es[][o.e.n.Node] starting ...
2021.02.26 16:19:31 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2021.02.26 16:19:31 INFO  es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2021.02.26 16:19:34 INFO  es[][o.e.c.s.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {sonarqube}{2NHf4TSERHGrR2s-napywQ}{5uMj6GLrRKGSHcHvlGAeLA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}
2021.02.26 16:19:34 INFO  es[][o.e.c.s.ClusterApplierService] new_master {sonarqube}{2NHf4TSERHGrR2s-napywQ}{5uMj6GLrRKGSHcHvlGAeLA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: apply cluster state (from master [master {sonarqube}{2NHf4TSERHGrR2s-napywQ}{5uMj6GLrRKGSHcHvlGAeLA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2021.02.26 16:19:34 INFO  es[][o.e.n.Node] started
2021.02.26 16:19:35 INFO  es[][o.e.g.GatewayService] recovered [7] indices into cluster_state
2021.02.26 16:19:37 INFO  es[][o.e.c.r.a.AllocationService] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[components][1], [metadatas][0]] ...]).
2021.02.26 16:19:41 INFO  es[][o.e.n.Node] stopping ...
2021.02.26 16:19:41 INFO  es[][o.e.n.Node] stopped
2021.02.26 16:19:41 INFO  es[][o.e.n.Node] closing ...
2021.02.26 16:19:41 INFO  es[][o.e.n.Node] closed

sonar.log

--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3)
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

2021.02.26 16:19:22 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2021.02.26 16:19:23 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2021.02.26 16:19:23 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
2021.02.26 16:19:23 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2021.02.26 16:19:23 INFO  app[][o.e.p.PluginsService] no modules loaded
2021.02.26 16:19:23 INFO  app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2021.02.26 16:19:37 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2021.02.26 16:19:37 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube]: /usr/lib/jvm/java-11-openjdk-11.0.9.11-2.el7_9.x86_64/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/temp --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/common/*:/opt/sonarqube/lib/jdbc/postgresql/postgresql-42.2.5.jar org.sonar.server.app.WebServer /opt/sonarqube/temp/sq-process2351702109199737634properties
2021.02.26 16:19:41 INFO  app[][o.s.a.SchedulerImpl] Process[web] is stopped
2021.02.26 16:19:41 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 143
2021.02.26 16:19:41 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
2021.02.26 16:19:41 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped

Tobias_Trabelsi · February 26, 2021, 9:25am

Hello @Reno_Chung and welcome to the community

can you please post the complete logs (specially the web.log) so we can have a look at what went actually wrong?
one of the ways to check which version of sonarqube is installed, would be to have a look at lib/sonar-application-<VERSION>.jar.

The java version that is installed on the server could also be interesting to know.

For my personal interest; how long is that and for how long is the server running?

Reno_Chung · March 2, 2021, 1:42am

Thank you @Tobias_Trabelsi.

The system’s Sonar application version is 7.9.1
and JAVA openjdk version 1.8.0_275

I don’t know exactly when Sonar was running but I assume it’s been running over a year without anyone actually using it.

I can’t upload the full log since it’s logrotating, but these are the logs of the day Sonar stopped working:
reno_chung_sonarqube_logs.zip (2.1 MB)

Thank you for your help.

Tobias_Trabelsi · March 5, 2021, 8:17am

Hi @Reno_Chung ,

Error querying database. Cause: org.postgresql.util.PSQLException: Connection to localhost:5432 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.

looks like this sonarqube instance was configured to run against a postgresql database that is installed on the same server, but is currently stopped. check that the postgresql server is up and running and the configuration in conf/sonar.properties matches the database server configuration (jdbc url, username, password and schema if not default). This should hopefully get the system back up.

you are a bit behind on our LTS release. The current LTS is 7.9.5 and requires java 11. when sonarqube is planned to be actively used again, you should consider an update

hope that helps

system · March 9, 2021, 7:08am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.