Sonarqube version: 7.7 Developer edition
Our Sonarqube server was running fine all upto a month ago. All of a sudden, we started getting 500 internal server error. Looking through the log files, it looks some issue with startup of Elastic search but its not evident from the sonar.log messages.
Based on the error, I have tried setting the limits for max file descriptors but it doesnt seam to consider the new values. Please advise if I m going into the right direction.
sonar.log
2019.11.18 16:40:30 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
<-- Wrapper Stopped
--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
2019.11.18 16:40:33 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/sonarqube-7.7/temp
2019.11.18 16:40:33 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.11.18 16:40:33 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/sonarqube-7.7/elasticsearch]: /opt/sonarqube/sonarqube-7.7/elasticsearch/bin/elasticsearch
2019.11.18 16:40:33 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2019.11.18 16:40:34 INFO app[][o.e.p.PluginsService] no modules loaded
2019.11.18 16:40:34 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.11.18 16:40:49 INFO app[][o.s.a.SchedulerImpl] Process[es] is up
2019.11.18 16:40:49 INFO app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube/sonarqube-7.7]: /usr/java/jdk1.8.0_131/jre/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/sonarqube-7.7/temp -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -cp ./lib/common/*:/opt/sonarqube/sonarqube-7.7/lib/jdbc/mysql/mysql-connector-java-5.1.46.jar org.sonar.server.app.WebServer /opt/sonarqube/sonarqube-7.7/temp/sq-process3263024257452122355properties
2019.11.18 16:40:51 INFO app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.11.18 16:40:51 INFO app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.11.18 16:40:51 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
2019.11.18 16:40:51 WARN app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 143
Es.log
2019.11.18 16:39:50 INFO es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [24.1gb], net total_space [31.4gb], types [rootfs]
2019.11.18 16:39:50 INFO es[][o.e.e.NodeEnvironment] heap size [495.3mb], compressed ordinary object pointers [true]
2019.11.18 16:39:50 INFO es[][o.e.n.Node] node name [sonarqube], node ID [z7Pe-l47TImzr3sVp3mcuQ]
2019.11.18 16:39:50 INFO es[][o.e.n.Node] version[6.6.2], pid[53700], build[default/tar/3bd3e59/2019-03-06T15:16:26.864148Z], OS[Linux/3.10.0-957.12.2.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_131/25.131-b11]
2019.11.18 16:39:50 INFO es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/opt/sonarqube/sonarqube-7.7/temp/es6, -XX:ErrorFile=/opt/sonarqube/sonarqube-7.7/logs/es_hs_err_pid%p.log, -Xms512m, -Xmx512m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/sonarqube-7.7/elasticsearch, -Des.path.conf=/opt/sonarqube/sonarqube-7.7/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar]
2019.11.18 16:39:51 INFO es[][o.e.p.PluginsService] loaded module [analysis-common]
2019.11.18 16:39:51 INFO es[][o.e.p.PluginsService] loaded module [lang-painless]
2019.11.18 16:39:51 INFO es[][o.e.p.PluginsService] loaded module [mapper-extras]
2019.11.18 16:39:51 INFO es[][o.e.p.PluginsService] loaded module [parent-join]
2019.11.18 16:39:51 INFO es[][o.e.p.PluginsService] loaded module [percolator]
2019.11.18 16:39:51 INFO es[][o.e.p.PluginsService] loaded module [reindex]
2019.11.18 16:39:51 INFO es[][o.e.p.PluginsService] loaded module [repository-url]
2019.11.18 16:39:51 INFO es[][o.e.p.PluginsService] loaded module [transport-netty4]
2019.11.18 16:39:51 INFO es[][o.e.p.PluginsService] no plugins loaded
2019.11.18 16:39:57 WARN es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2019.11.18 16:39:58 INFO es[][o.e.d.DiscoveryModule] using discovery type [zen] and host providers [settings]
2019.11.18 16:39:59 INFO es[][o.e.n.Node] initialized
2019.11.18 16:39:59 INFO es[][o.e.n.Node] starting ...
2019.11.18 16:39:59 INFO es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2019.11.18 16:39:59 WARN es[][o.e.b.BootstrapChecks] max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
2019.11.18 16:40:02 INFO es[][o.e.c.s.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {sonarqube}{z7Pe-l47TImzr3sVp3mcuQ}{4xhIYqQJTgCIwMEWmHsfmA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}
2019.11.18 16:40:02 INFO es[][o.e.c.s.ClusterApplierService] new_master {sonarqube}{z7Pe-l47TImzr3sVp3mcuQ}{4xhIYqQJTgCIwMEWmHsfmA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: apply cluster state (from master [master {sonarqube}{z7Pe-l47TImzr3sVp3mcuQ}{4xhIYqQJTgCIwMEWmHsfmA}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2019.11.18 16:40:02 INFO es[][o.e.n.Node] started
2019.11.18 16:40:02 INFO es[][o.e.g.GatewayService] recovered [0] indices into cluster_state
2019.11.18 16:40:05 INFO es[][o.e.n.Node] stopping ...
2019.11.18 16:40:05 INFO es[][o.e.n.Node] stopped
2019.11.18 16:40:05 INFO es[][o.e.n.Node] closing ...
2019.11.18 16:40:05 INFO es[][o.e.n.Node] closed
web.log
2019.11.19 09:06:58 WARN web[][o.s.p.ProcessEntryPoint] Fail to start web
java.lang.RuntimeException: org.apache.catalina.LifecycleException: Failed to initialize component [StandardServer[-1]]
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.sonar.server.app.EmbeddedTomcat.start(EmbeddedTomcat.java:65)
at org.sonar.server.app.WebServer.start(WebServer.java:51)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:113)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:100)
at org.sonar.server.app.WebServer.main(WebServer.java:91)
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [StandardServer[-1]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:140)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:427)
at org.sonar.server.app.EmbeddedTomcat.start(EmbeddedTomcat.java:61)
... 4 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to initialize component [StandardService[Tomcat]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:875)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
... 7 common frames omitted
Caused by: org.apache.catalina.LifecycleException: Failed to initialize connector [Connector[HTTP/1.1-9000]]
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:559)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
... 9 common frames omitted