Sonarqube keeps stopping

Info about the problem:

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    using one of the latest sonarqube images from the dockerhub registry: sonarqube:8.4-developer (*also tried sonarqube:8.2-community, but get the same issue)

  • what are you trying to achieve
    run the sonarqube service in aws ecs

  • what have you tried so far to achieve this
    The container starts, connects to DB, sonar, web, and es logs show the app booting up, then I see a log showing sonarqube gracefully shutting down with no clear issue or error. Any help or insight would be greatly appreciated!!
    After successfully deploying sonarqube onto aws ecs with connectivity to an rds instance containing scan data from a previous install, the service stops after it appears to start successfully. I have been investigating for errors (as sonar failures are usually in one of the log files), but there are no more clear indications as to why the service keeps stopping.

Here are the logs:
es.log

2020.08.25 15:37:27 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/ (overlay)]], net usable_space [20.8gb], net total_space [29.4gb], types [overlay]
2020.08.25 15:37:27 INFO  es[][o.e.e.NodeEnvironment] heap size [495.3mb], compressed ordinary object pointers [true]
2020.08.25 15:37:27 INFO  es[][o.e.n.Node] node name [sonarqube], node ID [x-Ipc9PsR3ifVuGsOEzj0Q]
2020.08.25 15:37:27 INFO  es[][o.e.n.Node] version[6.8.0], pid[36], build[default/tar/65b6179/2019-05-15T20:06:13.172855Z], OS[Linux/4.14.186-146.268.amzn2.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/11.0.6/11.0.6+10]
2020.08.25 15:37:27 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/opt/sonarqube/temp, -XX:ErrorFile=../logs/es_hs_err_pid%p.log, -Des.enforce.bootstrap.checks=true, -Xms512m, -Xmx512m, -XX:+HeapDumpOnOutOfMemoryError, -Dnode.store.allow_mmap=false, -Des.path.home=/opt/sonarqube/elasticsearch, -Des.path.conf=/opt/sonarqube/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar]
2020.08.25 15:37:28 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2020.08.25 15:37:28 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2020.08.25 15:37:28 INFO  es[][o.e.p.PluginsService] loaded module [mapper-extras]
2020.08.25 15:37:28 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2020.08.25 15:37:28 INFO  es[][o.e.p.PluginsService] loaded module [percolator]
2020.08.25 15:37:28 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2020.08.25 15:37:28 INFO  es[][o.e.p.PluginsService] loaded module [repository-url]
2020.08.25 15:37:28 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2020.08.25 15:37:28 INFO  es[][o.e.p.PluginsService] no plugins loaded
2020.08.25 15:37:30 WARN  es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
2020.08.25 15:37:31 INFO  es[][o.e.d.DiscoveryModule] using discovery type [zen] and host providers [settings]
2020.08.25 15:37:31 INFO  es[][o.e.n.Node] initialized
2020.08.25 15:37:31 INFO  es[][o.e.n.Node] starting ...
2020.08.25 15:37:31 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2020.08.25 15:37:31 INFO  es[][o.e.b.BootstrapChecks] explicitly enforcing bootstrap checks
2020.08.25 15:37:34 INFO  es[][o.e.c.s.MasterService] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {sonarqube}{x-Ipc9PsR3ifVuGsOEzj0Q}{CNqc1KWVQ4usuvRbxsIxeQ}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}
2020.08.25 15:37:34 INFO  es[][o.e.c.s.ClusterApplierService] new_master {sonarqube}{x-Ipc9PsR3ifVuGsOEzj0Q}{CNqc1KWVQ4usuvRbxsIxeQ}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: apply cluster state (from master [master {sonarqube}{x-Ipc9PsR3ifVuGsOEzj0Q}{CNqc1KWVQ4usuvRbxsIxeQ}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
2020.08.25 15:37:34 INFO  es[][o.e.n.Node] started
2020.08.25 15:37:34 INFO  es[][o.e.g.GatewayService] recovered [0] indices into cluster_state

sonar.log

2020.08.25 15:39:23 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2020.08.25 15:39:23 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2020.08.25 15:39:23 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
2020.08.25 15:39:23 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2020.08.25 15:39:23 INFO  app[][o.e.p.PluginsService] no modules loaded
2020.08.25 15:39:23 INFO  app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2020.08.25 15:39:35 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2020.08.25 15:39:35 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/opt/sonarqube]: /opt/java/openjdk/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/temp --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED -Xmx512m -Xms128m -XX:+HeapDumpOnOutOfMemoryError -Dhttp.nonProxyHosts=localhost|127.*|[::1] -cp ./lib/common/*:/opt/sonarqube/lib/jdbc/postgresql/postgresql-42.2.5.jar org.sonar.server.app.WebServer /opt/sonarqube/temp/sq-process16626680548950323970properties

web.log

2020.08.25 16:40:15 INFO  web[][o.s.p.ProcessEntryPoint] Starting web
2020.08.25 16:40:16 INFO  web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read
2020.08.25 16:40:16 INFO  web[][o.s.c.e.CoreExtensionsLoader] Loaded core extensions: developer-edition, developer-scanner, developer-server, license
2020.08.25 16:40:17 INFO  web[][o.e.p.PluginsService] no modules loaded
2020.08.25 16:40:17 INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2020.08.25 16:40:17 INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2020.08.25 16:40:17 INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2020.08.25 16:40:18 INFO  web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9001]
2020.08.25 16:40:18 INFO  web[][o.s.s.p.LogServerVersion] SonarQube Server / 7.9.0.26994 / 687d8007c0f62bfaaa44a890d93397de0a588119
2020.08.25 16:40:18 INFO  web[][o.sonar.db.Database] Create JDBC data source for jdbc:postgresql://sonarqubedb.cihviyekmiqy.us-east-1.rds.amazonaws.com:5432/sonarqube
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /opt/sonarqube
2020.08.25 16:40:20 INFO  web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled
2020.08.25 16:40:20 WARN  web[][o.s.s.p.DatabaseServerCompatibility] The database must be manually upgraded. Please backup the database and browse /setup. For more information: https://docs.sonarqube.org/latest/setup/upgrading
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Git / 1.8.0.1574 / aec3dc8f5228aabd218e1cd31ac6e6515a43715d
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin GitHub Authentication for SonarQube / 1.5.0.870 / 153f7c7af7a264adb0fcbe5fee87bdd140a6a3a1
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin JaCoCo / 1.0.2.475 / b79a4724f3a9af1051266b4f8ca0460977295ead
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin LDAP / 2.2.0.608 / 79dc3fa4393a29667673c70182f3016288b548b7
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SAML 2.0 Authentication for SonarQube / 1.1.0.181 / 16fcb5be2d3bcfee8920b29cd758d8b957a18acb
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarABAP / 3.8.0.2034 / 2904652c6ef4863eaace62c40fec0512724430e4
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarC# / 7.15.0.8572 / e0ad49e38a28a8fc333ba746fc998e48678f6a8b
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCFamily / 6.3.0.11371 / 87798a096368f2cee97709b9d41b9c135ac2600c
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCSS / 1.1.1.1010 / 365e21fd0cb9035669fc59f6fec7c8fd28a7303b
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarFlex / 2.5.1.1831 / a0c44437f6abb0feec76edd073f91fec64db2a6c
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarGo / 1.1.1.2000 / 40d55921c7a63b67386a053490d17b6723a46cd5
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarHTML / 3.1.0.1615 / 4181edb5eff5605bec82dc0aa15ecd70eaa5857f
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJS / 5.2.1.7778 / 49f34eaeaad59868d4353d89b1fc5c02bbe51976
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJava / 5.13.1.18282 / 568f8ed2349f48e250a9329895b9a870100dfbeb
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarKotlin / 1.5.0.315 / 4ff3a145a58f3f84f1b39846a205a129d742e993
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPHP / 3.2.0.4868 / ec66bd5f8490677eb0ebae82aa17c2a5d9c0e5e7
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPLSQL / 3.4.1.2576 / a27e6384088454d7160dd39ea5ba54a9929c15f4
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPython / 1.14.0.3086 / a6ac95b01b57bfb5f4ca604b4234595bb9f04b7e
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarRuby / 1.5.0.315 / 4ff3a145a58f3f84f1b39846a205a129d742e993
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarScala / 1.5.0.315 / 4ff3a145a58f3f84f1b39846a205a129d742e993
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarSwift / 4.1.0.3087 / e71ead60ade025c4f22a8c29fc32ab7c90606ec9
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTS / 1.9.0.3766 / 4a4080b78001a78d758d1d0fa0190fb9496b6f57
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTSQL / 1.4.0.3334 / 7b4dc9eeb6301765e09583a3d570b5941223e0b4
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarVB / 7.15.0.8572 / e0ad49e38a28a8fc333ba746fc998e48678f6a8b
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarXML / 2.0.1.2020 / c5b84004face582d56f110e24c29bf9c6a679e69
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Svn / 1.9.0.1295 / 942e075773975354e32691a60bfd968065703e04
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Vulnerability Analysis / 7.9.0.5105 / 90f56794ddd2e4e4c4265743c44e508dfff03c32
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Vulnerability Rules for C# / 7.9.0.5105 / 90f56794ddd2e4e4c4265743c44e508dfff03c32
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Vulnerability Rules for Java / 7.9.0.5105 / 90f56794ddd2e4e4c4265743c44e508dfff03c32
2020.08.25 16:40:20 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Vulnerability Rules for PHP / 7.9.0.5105 / 90f56794ddd2e4e4c4265743c44e508dfff03c32
2020.08.25 16:40:21 INFO  web[][o.s.s.p.d.m.c.PostgresCharsetHandler] Verify that database charset supports UTF8
2020.08.25 16:40:21 INFO  web[][o.s.s.p.Platform] Database needs to be migrated. Please refer to https://docs.sonarqube.org/latest/setup/upgrading
2020.08.25 16:40:21 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@4d8c0a69 [pattern=UrlPattern{inclusions=[/api/system/migrate_db.*, ...], exclusions=[/api/properties*, ...]}]
2020.08.25 16:40:21 INFO  web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2020.08.25 16:41:41 INFO  web[][o.s.p.ProcessEntryPoint] Gracefully stopping process
2020.08.25 16:41:42 INFO  web[][o.s.s.app.WebServer] WebServer stopped

Hi,

Welcome to the community!

I suspect that part of at least your es.log is missing; it doesn’t show shutdown. Generally, you’re going to find abnormal shutdown causes logged either in web.log (usually DB connection) or es.log.

Also, I notice this in web.log:
Database needs to be migrated. Please refer to https://docs.sonarqube.org/latest/setup/upgrading

Once you’ve got your server up, just follow those^ instructions.

 
Ann

Hi Ann! Thank you for your quick response!

Here was the rest of the es.log right before the container is shutdown to be restarted.
2020.08.25 18:19:20 INFO es[o.e.n.Node] stopping …
2020.08.25 18:19:20 INFO es[o.e.n.Node] stopped
2020.08.25 18:19:20 INFO es[o.e.n.Node] closing …
2020.08.25 18:19:20 INFO es[o.e.n.Node] closed

Speaking to the DB migration warning: my research and previous experience led me to the idea that migrating the postgresql DB could be done by going to the /setup page once the web server was up, and clicking “migrate”. If this is not the case, then which instructions would apply? I only see the db maintenance instructions for postgresql calling to execute a VACUUM FULL on the db after migration. The web server doesn’t appear to stay up long enough for me to be able to perform the /setup.

Hi,

If there are no errors in any of your server logs, then I have to conclude that the problem is external to SonarQube. Maybe there’s a sysadmin who doesn’t recognize the process or something “helpful” on your box?

Regarding migration, yes just go to that URL & hit the button.

 
:woman_shrugging:
Ann

adding these details from the cloudwatch logs here as well:
noting the “Process exited with exit value [es]: 143” WARN log

2020.08.25 19:33:06 INFO ce[o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9001]
2020.08.25 19:33:06 INFO ce[o.sonar.db.Database] Create JDBC data source for jdbc:postgresql://sonarqubedb.cihviyekmiqy.us-east-1.rds.amazonaws.com:5432/sonarqube
2020.08.25 19:33:07 INFO ce[o.s.s.p.ServerFileSystemImpl] SonarQube home: /opt/sonarqube
2020.08.25 19:33:07 INFO ce[o.s.c.c.CePluginRepository] Load plugins
2020.08.25 19:33:09 INFO ce[o.s.c.c.ComputeEngineContainerImpl] Running Developer edition
2020.08.25 19:33:09 INFO ce[o.s.ce.app.CeServer] Compute Engine is operational
2020.08.25 19:33:09 INFO app[o.s.a.SchedulerImpl] Process[ce] is up
2020.08.25 19:33:09 INFO app[o.s.a.SchedulerImpl] SonarQube is up
2020.08.25 19:33:12 INFO app[o.s.a.SchedulerImpl] Stopping SonarQube
2020.08.25 19:33:12 INFO ce[o.s.p.ProcessEntryPoint] Gracefully stopping process
2020.08.25 19:33:12 INFO ce[o.s.ce.app.CeServer] Compute Engine is stopping…
2020.08.25 19:33:12 INFO ce[o.s.c.t.CeProcessingSchedulerImpl] Gracefully stopping workers…
2020.08.25 19:33:12 INFO ce[o.s.ce.app.CeServer] Compute Engine is stopped
2020.08.25 19:33:12 INFO app[o.s.a.SchedulerImpl] Process[ce] is stopped
2020.08.25 19:33:13 INFO web[o.s.p.ProcessEntryPoint] Gracefully stopping process
2020.08.25 19:33:13 INFO web[o.s.s.n.NotificationDaemon] Notification service stopped
2020.08.25 19:33:13 INFO web[o.s.s.app.WebServer] WebServer stopped
2020.08.25 19:33:13 INFO app[o.s.a.SchedulerImpl] Process[web] is stopped
2020.08.25 19:33:13 WARN app[o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 143
2020.08.25 19:33:13 INFO app[o.s.a.SchedulerImpl] Process[es] is stopped
2020.08.25 19:33:13 INFO app[o.s.a.SchedulerImpl] SonarQube is stopped

Hi,

Sorry, but I’m really out of ideas here.

 
:confused:
Ann

The DB migration warnings have been resolved. Sonar was running long enough for me to hit the “migrate” button on the /setup page, then the app stopped. On the next boot I was able to see the homepage and the list of projects shortly before it the app shutdown again. I think the issue is resource related, but I don’t see any additional logs (aside from the [WARN app[o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 143]) that tell me which process is running into a problem that causes sonar to stop itself about a minute after successfully starting. I have run the commands to set the kernel mem settings appropriately as recommended in the docs, or else I would get sonar bootstrap check errors during start.

Log sources I am monitoring on DEBUG mode are sonar.log, es.log, and web.log, as well as the cloudwatch logs. Nothing in particular is standing out aside from the es process exit error, which is vague and research points to possible misconfiguration and/or invalid plugins. I have not installed any additional plugins; just launching using the versions from sonarqube’s public dockerhub. I’ve been doing additional research on the configurations as well, but currently am stumped at what could be causing this.

Looking for anyone in the community has further debug experience that can provide any further assistance in resolving this issue and getting sonar back in action for us.

Thank you

Hi @creynolds ,

the exit code 143 is coming from elasticsearch and indicates that the process did not terminate in time and it was stopped with SIGTERM. SIGTERM (kill -15) + FATAL_EXIT (128) = 143. But this is nothing unexpected as even elastic said this can happen from time to time with docker.

Let’s step back a little and summarize.

  • You are running Sonarqube on AWS ECS
  • The Container starts and gets terminated shortly after startup is complete
  • The Logs contain no Error messages

i would suggest to pick the infrastructure apart one by one. can you create a ec2 instance with docker installed on it and connect it to the same database? Maybe the Data source is giving us more information.
If this is working okay, let’s switch back to ECS and launch a custom image derived from the official sonarqube image that will allow us to connect to the running container (as in not start sonarqube but a dummy process as entrypoint), and debug from there.

Hope this will give us more intel of what’s going on here