Sonarqube - Process exited with exit value [es]: 143

Sonarqube version: 7.7 Developer edition
Our Sonarqube server was running fine all upto a month ago. All of a sudden, we started getting 500 internal server error. Looking through the log files, it looks some issue with startup of Elastic search but its not evident from the sonar.log messages.

Based on the error, I have tried setting the limits for max file descriptors but it doesnt seam to consider the new values. Please advise if I m going into the right direction.

sonar.log

2019.11.18 16:40:30 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
        <-- Wrapper Stopped
        --> Wrapper Started as Daemon
        Launching a JVM...
        Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
          Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.
     
    2019.11.18 16:40:33 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/sonarqube-7.7/temp
    2019.11.18 16:40:33 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
    2019.11.18 16:40:33 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/sonarqube-7.7/elasticsearch]: /opt/sonarqube/sonarqube-7.7/elasticsearch/bin/elasticsearch
    2019.11.18 16:40:33 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
    2019.11.18 16:40:34 INFO  app[][o.e.p.PluginsService] no modules loaded
    2019.11.18 16:40:34 INFO  app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
    2019.11.18 16:40:49 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
    2019.11.18 16:40:49 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube/sonarqube-7.7]: /usr/java/jdk1.8.0_131/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/sonarqube-7.7/temp -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -cp ./lib/common/*:/opt/sonarqube/sonarqube-7.7/lib/jdbc/mysql/mysql-connector-java-5.1.46.jar org.sonar.server.app.WebServer /opt/sonarqube/sonarqube-7.7/temp/sq-process3263024257452122355properties
    2019.11.18 16:40:51 INFO  app[][o.s.a.SchedulerImpl] Process [web] is stopped
    2019.11.18 16:40:51 INFO  app[][o.s.a.SchedulerImpl] Process [es] is stopped
    2019.11.18 16:40:51 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
    2019.11.18 16:40:51 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143

Es.log

2019.11.18 16:39:50 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [24.1gb], net total_space [31.4gb], types [rootfs]

2019.11.18 16:39:50 INFO  es[][o.e.e.NodeEnvironment] heap size [495.3mb], compressed ordinary object pointers [true]

2019.11.18 16:39:50 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [z7Pe-l47TImzr3sVp3mcuQ]

2019.11.18 16:39:50 INFO  es[][o.e.n.Node] version[6.6.2], pid[53700], build[default/tar/3bd3e59/2019-03-06T15:16:26.864148Z], OS[Linux/3.10.0-957.12.2.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]

2019.11.18 16:39:50 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/opt/sonarqube/sonarqube-7.7/temp/es6, -XX:ErrorFile=/opt/sonarqube/sonarqube-7.7/logs/es_hs_err_pid%p.log, -Xms512m, -Xmx512m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/sonarqube-7.7/elasticsearch, -Des.path.conf=/opt/sonarqube/sonarqube-7.7/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar]

2019.11.18 16:39:51 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]

2019.11.18 16:39:51 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]

2019.11.18 16:39:51 INFO  es[][o.e.p.PluginsService] loaded module [mapper-extras]

2019.11.18 16:39:51 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]

2019.11.18 16:39:51 INFO  es[][o.e.p.PluginsService] loaded module [percolator]

2019.11.18 16:39:51 INFO  es[][o.e.p.PluginsService] loaded module [reindex]

2019.11.18 16:39:51 INFO  es[][o.e.p.PluginsService] loaded module [repository-url]

2019.11.18 16:39:51 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]

2019.11.18 16:39:51 INFO  es[][o.e.p.PluginsService] no plugins loaded

2019.11.18 16:39:57 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.

2019.11.18 16:39:58 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and host providers [settings]

2019.11.18 16:39:59 INFO  es[][o.e.n.Node] initialized

2019.11.18 16:39:59 INFO  es[][o.e.n.Node] starting ...

2019.11.18 16:39:59 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}

2019.11.18 16:39:59 WARN  es[][o.e.b.BootstrapChecks] max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]

2019.11.18 16:40:02 INFO  es[][o.e.c.s.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {sonarqube}{z7Pe-l47TImzr3sVp3mcuQ}{4xhIYqQJTgCIwMEWmHsfmA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}

2019.11.18 16:40:02 INFO  es[][o.e.c.s.ClusterApplierService] new_master {sonarqube}{z7Pe-l47TImzr3sVp3mcuQ}{4xhIYqQJTgCIwMEWmHsfmA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: apply cluster state (from master [master {sonarqube}{z7Pe-l47TImzr3sVp3mcuQ}{4xhIYqQJTgCIwMEWmHsfmA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])

2019.11.18 16:40:02 INFO  es[][o.e.n.Node] started

2019.11.18 16:40:02 INFO  es[][o.e.g.GatewayService] recovered [0] indices into cluster_state

2019.11.18 16:40:05 INFO  es[][o.e.n.Node] stopping ...

2019.11.18 16:40:05 INFO  es[][o.e.n.Node] stopped

2019.11.18 16:40:05 INFO  es[][o.e.n.Node] closing ...

2019.11.18 16:40:05 INFO  es[][o.e.n.Node] closed

web.log

2019.11.19 09:06:58 WARN web[][o.s.p.ProcessEntryPoint] Fail to start web

java.lang.RuntimeException: org.apache.catalina.LifecycleException: Failed to initialize component [StandardServer[-1]]

at com.google.common.base.Throwables.propagate(Throwables.java:160)

at org.sonar.server.app.EmbeddedTomcat.start(EmbeddedTomcat.java:65)

at org.sonar.server.app.WebServer.start(WebServer.java:51)

at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:113)

at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:100)

at org.sonar.server.app.WebServer.main(WebServer.java:91)

Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [StandardServer[-1]]

at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)

at org.apache.catalina.startup.Tomcat.start(Tomcat.java:427)

at org.sonar.server.app.EmbeddedTomcat.start(EmbeddedTomcat.java:61)

... 4 common frames omitted

Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [StandardService[Tomcat]]

at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)

at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:875)

at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)

... 7 common frames omitted

Caused by: org.apache.catalina.LifecycleException: Failed to initialize connector [Connector[HTTP/1.1-9000]]

at org.apache.catalina.core.StandardService.initInternal(StandardService.java:559)

at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)

... 9 common frames omitted

Welcome to the community. Did you find something interesting in the web.log file ?

1 Like

Added the web.log error in the description above. Generic error again.