Sonarqube 7.4 crashes after a short time


(Zowie Haugaard) #1

Sonarqube 7.4 crashes after a short period of time running on Ubuntu 16.04 LTS. No indication in log files as to direct cause, elasticsearch exits with code 129. Please find logs below

sonar.log

--> Wrapper Started as Console
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

2019.01.07 06:42:21 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /home/sonarqube/sonarqube-7.4/temp
2019.01.07 06:42:21 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.01.07 06:42:21 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/home/sonarqube/sonarqube-7.4/elasticsearch]: /home/sonarqube/sonarqube-7.4/elasticsearch/bin/elasticsearch -$
2019.01.07 06:42:21 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2019.01.07 06:42:22 INFO  app[][o.e.p.PluginsService] no modules loaded
2019.01.07 06:42:22 INFO  app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.01.07 06:42:33 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2019.01.07 06:42:33 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/home/sonarqube/sonarqube-7.4]: /usr/lib/jvm/java-x86_64-80/jre/bin/java -Djava.awt.headless=true -Dfile.en$
2019.01.07 06:42:55 INFO  app[][o.s.a.SchedulerImpl] Process[web] is up
2019.01.07 06:42:55 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/home/sonarqube/sonarqube-7.4]: /usr/lib/jvm/java-x86_64-80/jre/bin/java -Djava.awt.headless=true -Dfile.enco$
2019.01.07 06:43:03 INFO  app[][o.s.a.SchedulerImpl] Process[ce] is up
2019.01.07 06:43:03 INFO  app[][o.s.a.SchedulerImpl] SonarQube is up
2019.01.07 08:43:03 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 129
2019.01.07 08:43:03 INFO  app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.01.07 08:43:06 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [web]: 129
2019.01.07 08:43:06 INFO  app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.01.07 08:43:06 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [ce]: 129
2019.01.07 08:43:06 INFO  app[][o.s.a.SchedulerImpl] Process [ce] is stopped
2019.01.07 08:43:06 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
2019.01.07 08:43:06 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [ce]: 129
<-- Wrapper Stopped

es.log

2019.01.07 06:42:26 INFO  es[][o.e.n.Node] starting ...
2019.01.07 06:42:26 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2019.01.07 06:42:29 INFO  es[][o.e.c.s.ClusterService] new_master {sonarqube}{L76HRy-aTwG3mjnWX3a0JA}{wpD14ugQRmGLoHBkUbhMkw}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: zen-disco-elected-as-master ([0] nodes joined)
2019.01.07 06:42:29 INFO  es[][o.e.n.Node] started
2019.01.07 06:42:30 INFO  es[][o.e.g.GatewayService] recovered [8] indices into cluster_state
2019.01.07 06:42:31 INFO  es[][o.e.c.r.a.AllocationService] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[metadatas][0]] ...]).
2019.01.07 08:43:03 INFO  es[][o.e.n.Node] stopping ...
2019.01.07 08:43:03 INFO  es[][o.e.n.Node] stopped
2019.01.07 08:43:03 INFO  es[][o.e.n.Node] closing ...
2019.01.07 08:43:03 INFO  es[][o.e.n.Node] closed

(Julien Lancelot) #2

Hi,

Could you please also share the web.log content ?

Regards,
Julien Lancelot


(S├ębastien Lesaint) #3

Hello,

This sound highly similar to https://stackoverflow.com/a/41507249 (SQ configured to use more memory than available physical one and being killed by Linux OOM Killer).


(Zowie Haugaard) #4

web.log as follows:

2019.01.07 06:42:35 INFO  web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read
2019.01.07 06:42:36 INFO  web[][o.s.c.e.CoreExtensionsLoader] Loaded core extensions:
2019.01.07 06:42:36 INFO  web[][o.e.p.PluginsService] no modules loaded
2019.01.07 06:42:36 INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2019.01.07 06:42:36 INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2019.01.07 06:42:36 INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.01.07 06:42:39 INFO  web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9001]
2019.01.07 06:42:39 INFO  web[][o.s.s.p.LogServerVersion] SonarQube Server / 7.4.0.18908 / a4d0983e8057356b19996995343296311fbcc2ed
2019.01.07 06:42:39 INFO  web[][o.sonar.db.Database] Create JDBC data source for jdbc:postgresql://sonar-commons2.fyre.ibm.com/sonarqube
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /home/sonarqube/sonarqube-7.4
2019.01.07 06:42:42 INFO  web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Git / 1.6.0.1349 / 5324aafc7d16337ca792fec35903d0d313c9e2c7
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin GitHub Authentication for SonarQube / 1.4.0.695 / 80db5ad36b0108fb8e24e4308a19de4aeea1ac37
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin JaCoCo / 1.0.1.143 / 65c55e41e32c114704fd597ccea91fc55becb4ec
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin LDAP / 2.2.0.608 / 79dc3fa4393a29667673c70182f3016288b548b7
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SVG Badges / 3.0.1
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarC# / 7.7.0.7192 / c46d44cdae6e75538626f380fe3fa7553e6cf93d
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCSS / 1.0.2.611 / 8db5e31b2e21ff683da69668ec577c7a026acf23
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarFlex / 2.4.0.1222 / 68d9cb3b7daccbc4869c4f2b89d09218d26a0829
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarGo / 1.1.0.1612 / 1809c7be6d88396c0a1b8d597b1e9e0ed8d29730
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJS / 5.0.0.6962 / ac2fe7567f3dfeb3295e2e710afd8c06b7760e9d
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJava / 5.8.0.15699 / 34186ed1499e6166d3ef8670086d05ab6f12f6f9
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarKotlin / 1.2.1.2009 / 3dd3ade901a10bc88a7158c014eba9f5a568951c
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPHP / 2.14.0.3569 / 59a202b8d1c5276ce32da2a0bc63c94cae4cdeb2
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPython / 1.10.0.2131 / a00e0ddc83bbeb01a9ee7475fab9eafe3f4096ac
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarRuby / 1.2.1.2009 / 3dd3ade901a10bc88a7158c014eba9f5a568951c
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTS / 1.8.0.3332 / af8e41ab1a72ba3cc28b0e3e9e065a0ccb47d31d
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarVB / 7.7.0.7192 / c46d44cdae6e75538626f380fe3fa7553e6cf93d
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarXML / 1.5.1.1452 / eadc1dcd0e24a3c77068e3c1d41af0327d6b16ea
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Svn / 1.9.0.1295 / 942e075773975354e32691a60bfd968065703e04
2019.01.07 06:42:44 INFO  web[][o.s.s.p.d.m.c.PostgresCharsetHandler] Verify that database charset supports UTF8
2019.01.07 06:42:44 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@496f26ce [pattern=UrlPattern{inclusions=[/api/system/migrate_db.*, ...], exclusions=[/api/properties*, ...]}]
2019.01.07 06:42:44 INFO  web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2019.01.07 06:42:45 WARN  web[][o.s.c.p.ComponentKeys] Bad component key: org.sonar.java.externalreport.ExternalRulesDefinition@5bf4802. Please implement toString() method on class org.sonar.java.externalreport.ExternalRulesDefinition
2019.01.07 06:42:45 WARN  web[][o.s.c.p.ComponentKeys] Bad component key: org.sonar.java.externalreport.ExternalRulesDefinition@a052f22d. Please implement toString() method on class org.sonar.java.externalreport.ExternalRulesDefinition
2019.01.07 06:42:45 WARN  web[][o.s.c.p.ComponentKeys] Bad component key: org.sonar.java.externalreport.ExternalRulesDefinition@ab6ff172. Please implement toString() method on class org.sonar.java.externalreport.ExternalRulesDefinition
2019.01.07 06:42:45 INFO  web[][o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties (no proxy)
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.f.PreferredFontProvider] SVGImageGenerator will be using font 'Verdana' in order to compute SVG badges width.
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.w.SVGImageGenerator] SVGImageGenerator is now ready.
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.w.g.QualityGateBadgeGenerator] QualityGateBadgeGenerator is now ready.
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.w.m.MeasureBadgeGenerator] MeasureBadgeGenerator is now ready.
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.w.c.CeActivityBadgeGenerator] CeActivityBadgeGenerator is now ready.
2019.01.07 06:42:46 INFO  web[][o.s.s.s.LogServerId] Server ID: 46F795E3-AWPvO4Qnt3RfrRhT0UaY
2019.01.07 06:42:46 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/plugins/download
2019.01.07 06:42:46 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/permissions/search_templates
2019.01.07 06:42:46 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/security_reports/show
2019.01.07 06:42:46 INFO  web[][o.s.s.t.TelemetryDaemon] Sharing of SonarQube statistics is enabled.
2019.01.07 06:42:46 INFO  web[][o.s.s.n.NotificationDaemon] Notification service started (delay 60 sec.)
2019.01.07 06:42:47 INFO  web[][o.s.s.s.GeneratePluginIndex] Generate scanner plugin index
2019.01.07 06:42:47 INFO  web[][o.s.s.s.RegisterPlugins] Register plugins
2019.01.07 06:42:47 INFO  web[][o.s.s.s.RegisterMetrics] Register metrics
2019.01.07 06:42:47 INFO  web[][o.s.s.r.RegisterRules] Register rules
2019.01.07 06:42:53 INFO  web[][o.s.s.q.BuiltInQProfileRepositoryImpl] Load quality profiles
2019.01.07 06:42:53 INFO  web[][o.s.s.q.RegisterQualityProfiles] Register quality profiles
2019.01.07 06:42:53 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile css/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile go/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile kotlin/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way Recommended
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile py/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ruby/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cs/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile java/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile flex/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile xml/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/PSR-2
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Drupal
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile vbnet/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way recommended
2019.01.07 06:42:54 INFO  web[][o.s.s.s.RegisterPermissionTemplates] Register permission templates
2019.01.07 06:42:54 INFO  web[][o.s.s.s.RenameDeprecatedPropertyKeys] Rename deprecated property keys
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@76754892 [pattern=UrlPattern{inclusions=[/api/issues/delete_comment.*, ...], exclusions=[/api/properties*, ..$
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.DeprecatedPropertiesWsFilter@5ff2de60 [pattern=UrlPattern{inclusions=[/api/properties/*], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceReroutingFilter@85748b83 [pattern=UrlPattern{inclusions=[/api/components/bulk_update_key, ...], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.InitFilter@221d0c4b [pattern=UrlPattern{inclusions=[/sessions/init/*], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.OAuth2CallbackFilter@16265707 [pattern=UrlPattern{inclusions=[/oauth2/callback/*], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LoginAction@8d8fdcb3 [pattern=UrlPattern{inclusions=[/api/authentication/login], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LogoutAction@fad6c63d [pattern=UrlPattern{inclusions=[/api/authentication/logout], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.ValidateAction@c53409c [pattern=UrlPattern{inclusions=[/api/authentication/validate], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.p.PlatformLevelStartup] Running Community Edition
2019.01.07 06:42:54 INFO  web[][o.s.s.p.Platform] WebServer is operational

(Zowie Haugaard) #5

Web and compute are configured with xmx at 2gb, elasticsearch is set to max of 6gb. free reports 14gb available. Seems to me I should still be within bounds, yes?


(S├ębastien Lesaint) #6

If thoses are the only programs running on the box, probably.
You can google where OOMKiller logs and confirm for sure it is not the cause of the sudden SQ shutdown.

Also, you may have some cron or script ran by another user that kills SQ. Double check your setup.