Sonarqube 7.4 crashes after a short time


(Zowie Haugaard) #1

Sonarqube 7.4 crashes after a short period of time running on Ubuntu 16.04 LTS. No indication in log files as to direct cause, elasticsearch exits with code 129. Please find logs below

sonar.log

--> Wrapper Started as Console
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
  Copyright 1999-2006 Tanuki Software, Inc.  All Rights Reserved.

2019.01.07 06:42:21 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /home/sonarqube/sonarqube-7.4/temp
2019.01.07 06:42:21 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2019.01.07 06:42:21 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/home/sonarqube/sonarqube-7.4/elasticsearch]: /home/sonarqube/sonarqube-7.4/elasticsearch/bin/elasticsearch -$
2019.01.07 06:42:21 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2019.01.07 06:42:22 INFO  app[][o.e.p.PluginsService] no modules loaded
2019.01.07 06:42:22 INFO  app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.01.07 06:42:33 INFO  app[][o.s.a.SchedulerImpl] Process[es] is up
2019.01.07 06:42:33 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='web', ipcIndex=2, logFilenamePrefix=web]] from [/home/sonarqube/sonarqube-7.4]: /usr/lib/jvm/java-x86_64-80/jre/bin/java -Djava.awt.headless=true -Dfile.en$
2019.01.07 06:42:55 INFO  app[][o.s.a.SchedulerImpl] Process[web] is up
2019.01.07 06:42:55 INFO  app[][o.s.a.p.ProcessLauncherImpl] Launch process[[key='ce', ipcIndex=3, logFilenamePrefix=ce]] from [/home/sonarqube/sonarqube-7.4]: /usr/lib/jvm/java-x86_64-80/jre/bin/java -Djava.awt.headless=true -Dfile.enco$
2019.01.07 06:43:03 INFO  app[][o.s.a.SchedulerImpl] Process[ce] is up
2019.01.07 06:43:03 INFO  app[][o.s.a.SchedulerImpl] SonarQube is up
2019.01.07 08:43:03 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [es]: 129
2019.01.07 08:43:03 INFO  app[][o.s.a.SchedulerImpl] Process [es] is stopped
2019.01.07 08:43:06 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [web]: 129
2019.01.07 08:43:06 INFO  app[][o.s.a.SchedulerImpl] Process [web] is stopped
2019.01.07 08:43:06 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [ce]: 129
2019.01.07 08:43:06 INFO  app[][o.s.a.SchedulerImpl] Process [ce] is stopped
2019.01.07 08:43:06 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
2019.01.07 08:43:06 WARN  app[][o.s.a.p.AbstractProcessMonitor] Process exited with exit value [ce]: 129
<-- Wrapper Stopped

es.log

2019.01.07 06:42:26 INFO  es[][o.e.n.Node] starting ...
2019.01.07 06:42:26 INFO  es[][o.e.t.TransportService] publish_address {127.0.0.1:9001}, bound_addresses {127.0.0.1:9001}
2019.01.07 06:42:29 INFO  es[][o.e.c.s.ClusterService] new_master {sonarqube}{L76HRy-aTwG3mjnWX3a0JA}{wpD14ugQRmGLoHBkUbhMkw}{127.0.0.1}{127.0.0.1:9001}{rack_id=sonarqube}, reason: zen-disco-elected-as-master ([0] nodes joined)
2019.01.07 06:42:29 INFO  es[][o.e.n.Node] started
2019.01.07 06:42:30 INFO  es[][o.e.g.GatewayService] recovered [8] indices into cluster_state
2019.01.07 06:42:31 INFO  es[][o.e.c.r.a.AllocationService] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[metadatas][0]] ...]).
2019.01.07 08:43:03 INFO  es[][o.e.n.Node] stopping ...
2019.01.07 08:43:03 INFO  es[][o.e.n.Node] stopped
2019.01.07 08:43:03 INFO  es[][o.e.n.Node] closing ...
2019.01.07 08:43:03 INFO  es[][o.e.n.Node] closed

(Julien Lancelot) #2

Hi,

Could you please also share the web.log content ?

Regards,
Julien Lancelot


(S├ębastien Lesaint) #3

Hello,

This sound highly similar to https://stackoverflow.com/a/41507249 (SQ configured to use more memory than available physical one and being killed by Linux OOM Killer).


(Zowie Haugaard) #4

web.log as follows:

2019.01.07 06:42:35 INFO  web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read
2019.01.07 06:42:36 INFO  web[][o.s.c.e.CoreExtensionsLoader] Loaded core extensions:
2019.01.07 06:42:36 INFO  web[][o.e.p.PluginsService] no modules loaded
2019.01.07 06:42:36 INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
2019.01.07 06:42:36 INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
2019.01.07 06:42:36 INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019.01.07 06:42:39 INFO  web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9001]
2019.01.07 06:42:39 INFO  web[][o.s.s.p.LogServerVersion] SonarQube Server / 7.4.0.18908 / a4d0983e8057356b19996995343296311fbcc2ed
2019.01.07 06:42:39 INFO  web[][o.sonar.db.Database] Create JDBC data source for jdbc:postgresql://sonar-commons2.fyre.ibm.com/sonarqube
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /home/sonarqube/sonarqube-7.4
2019.01.07 06:42:42 INFO  web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Git / 1.6.0.1349 / 5324aafc7d16337ca792fec35903d0d313c9e2c7
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin GitHub Authentication for SonarQube / 1.4.0.695 / 80db5ad36b0108fb8e24e4308a19de4aeea1ac37
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin JaCoCo / 1.0.1.143 / 65c55e41e32c114704fd597ccea91fc55becb4ec
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin LDAP / 2.2.0.608 / 79dc3fa4393a29667673c70182f3016288b548b7
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SVG Badges / 3.0.1
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarC# / 7.7.0.7192 / c46d44cdae6e75538626f380fe3fa7553e6cf93d
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCSS / 1.0.2.611 / 8db5e31b2e21ff683da69668ec577c7a026acf23
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarFlex / 2.4.0.1222 / 68d9cb3b7daccbc4869c4f2b89d09218d26a0829
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarGo / 1.1.0.1612 / 1809c7be6d88396c0a1b8d597b1e9e0ed8d29730
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJS / 5.0.0.6962 / ac2fe7567f3dfeb3295e2e710afd8c06b7760e9d
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJava / 5.8.0.15699 / 34186ed1499e6166d3ef8670086d05ab6f12f6f9
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarKotlin / 1.2.1.2009 / 3dd3ade901a10bc88a7158c014eba9f5a568951c
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPHP / 2.14.0.3569 / 59a202b8d1c5276ce32da2a0bc63c94cae4cdeb2
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPython / 1.10.0.2131 / a00e0ddc83bbeb01a9ee7475fab9eafe3f4096ac
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarRuby / 1.2.1.2009 / 3dd3ade901a10bc88a7158c014eba9f5a568951c
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTS / 1.8.0.3332 / af8e41ab1a72ba3cc28b0e3e9e065a0ccb47d31d
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarVB / 7.7.0.7192 / c46d44cdae6e75538626f380fe3fa7553e6cf93d
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarXML / 1.5.1.1452 / eadc1dcd0e24a3c77068e3c1d41af0327d6b16ea
2019.01.07 06:42:42 INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Svn / 1.9.0.1295 / 942e075773975354e32691a60bfd968065703e04
2019.01.07 06:42:44 INFO  web[][o.s.s.p.d.m.c.PostgresCharsetHandler] Verify that database charset supports UTF8
2019.01.07 06:42:44 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@496f26ce [pattern=UrlPattern{inclusions=[/api/system/migrate_db.*, ...], exclusions=[/api/properties*, ...]}]
2019.01.07 06:42:44 INFO  web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2019.01.07 06:42:45 WARN  web[][o.s.c.p.ComponentKeys] Bad component key: org.sonar.java.externalreport.ExternalRulesDefinition@5bf4802. Please implement toString() method on class org.sonar.java.externalreport.ExternalRulesDefinition
2019.01.07 06:42:45 WARN  web[][o.s.c.p.ComponentKeys] Bad component key: org.sonar.java.externalreport.ExternalRulesDefinition@a052f22d. Please implement toString() method on class org.sonar.java.externalreport.ExternalRulesDefinition
2019.01.07 06:42:45 WARN  web[][o.s.c.p.ComponentKeys] Bad component key: org.sonar.java.externalreport.ExternalRulesDefinition@ab6ff172. Please implement toString() method on class org.sonar.java.externalreport.ExternalRulesDefinition
2019.01.07 06:42:45 INFO  web[][o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties (no proxy)
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.f.PreferredFontProvider] SVGImageGenerator will be using font 'Verdana' in order to compute SVG badges width.
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.w.SVGImageGenerator] SVGImageGenerator is now ready.
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.w.g.QualityGateBadgeGenerator] QualityGateBadgeGenerator is now ready.
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.w.m.MeasureBadgeGenerator] MeasureBadgeGenerator is now ready.
2019.01.07 06:42:46 INFO  web[][c.q.p.s.b.w.c.CeActivityBadgeGenerator] CeActivityBadgeGenerator is now ready.
2019.01.07 06:42:46 INFO  web[][o.s.s.s.LogServerId] Server ID: 46F795E3-AWPvO4Qnt3RfrRhT0UaY
2019.01.07 06:42:46 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/plugins/download
2019.01.07 06:42:46 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/permissions/search_templates
2019.01.07 06:42:46 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/security_reports/show
2019.01.07 06:42:46 INFO  web[][o.s.s.t.TelemetryDaemon] Sharing of SonarQube statistics is enabled.
2019.01.07 06:42:46 INFO  web[][o.s.s.n.NotificationDaemon] Notification service started (delay 60 sec.)
2019.01.07 06:42:47 INFO  web[][o.s.s.s.GeneratePluginIndex] Generate scanner plugin index
2019.01.07 06:42:47 INFO  web[][o.s.s.s.RegisterPlugins] Register plugins
2019.01.07 06:42:47 INFO  web[][o.s.s.s.RegisterMetrics] Register metrics
2019.01.07 06:42:47 INFO  web[][o.s.s.r.RegisterRules] Register rules
2019.01.07 06:42:53 INFO  web[][o.s.s.q.BuiltInQProfileRepositoryImpl] Load quality profiles
2019.01.07 06:42:53 INFO  web[][o.s.s.q.RegisterQualityProfiles] Register quality profiles
2019.01.07 06:42:53 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile css/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile go/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile kotlin/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way Recommended
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile py/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ruby/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cs/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile java/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile flex/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile xml/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/PSR-2
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Drupal
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile vbnet/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way
2019.01.07 06:42:54 INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way recommended
2019.01.07 06:42:54 INFO  web[][o.s.s.s.RegisterPermissionTemplates] Register permission templates
2019.01.07 06:42:54 INFO  web[][o.s.s.s.RenameDeprecatedPropertyKeys] Rename deprecated property keys
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@76754892 [pattern=UrlPattern{inclusions=[/api/issues/delete_comment.*, ...], exclusions=[/api/properties*, ..$
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.DeprecatedPropertiesWsFilter@5ff2de60 [pattern=UrlPattern{inclusions=[/api/properties/*], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceReroutingFilter@85748b83 [pattern=UrlPattern{inclusions=[/api/components/bulk_update_key, ...], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.InitFilter@221d0c4b [pattern=UrlPattern{inclusions=[/sessions/init/*], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.OAuth2CallbackFilter@16265707 [pattern=UrlPattern{inclusions=[/oauth2/callback/*], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LoginAction@8d8fdcb3 [pattern=UrlPattern{inclusions=[/api/authentication/login], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LogoutAction@fad6c63d [pattern=UrlPattern{inclusions=[/api/authentication/logout], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.ValidateAction@c53409c [pattern=UrlPattern{inclusions=[/api/authentication/validate], exclusions=[]}]
2019.01.07 06:42:54 INFO  web[][o.s.s.p.p.PlatformLevelStartup] Running Community Edition
2019.01.07 06:42:54 INFO  web[][o.s.s.p.Platform] WebServer is operational

(Zowie Haugaard) #5

Web and compute are configured with xmx at 2gb, elasticsearch is set to max of 6gb. free reports 14gb available. Seems to me I should still be within bounds, yes?