Sonarqube pod is restarting in kubernetes

SonarQube Server / Community Build
1

SonarQube pods is restarting multiple times:

  • which versions are you using (SonarQube 9.9.0.65466)
  • how is SonarQube deployed: Helm,
  • I was trying to access the sonarqube instance from url
  • Below are the logs
*2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Kotlin Code Quality and Security / 2.12.0.1956 / a6df1ae252bd62d63f8673c28f87ad14258a7904
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy PHP Code Quality and Security / 3.27.1.9352 / 3ddc5a03e1a7e3729d41e7c1a30a37d5715958c7
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy PL/I Code Quality and Security / 1.12.0.3443 / e9d43136dec6eea6c0129c12d2a019d141fdcc39
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy PL/SQL Code Quality and Security / 3.8.0.4948 / ab46f4c11644e5c9b17f0a5a1612d4b7362e04f3
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Python Code Quality and Security / 3.24.0.10784 / 30aa413d983f6c37b4298eba00555c3050009b52
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy RPG Code Quality / 3.3.0.3147 / 9cb6a4566eb55cc6b874220ab95f0504a952b560
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Ruby Code Quality and Security / 1.11.0.3905 / e1f28bc000e04ca01881e84218d01d464a17a36f
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Scala Code Quality and Security / 1.11.0.3905 / e1f28bc000e04ca01881e84218d01d464a17a36f
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Swift Code Quality and Security / 4.8.0.5759 / 3d5b95aaf19a398736a7f5eec314c3c05f7193d2
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy T-SQL Code Quality and Security / 1.7.0.5449 / 110c82f888b1f40e1c2a0d3165fcffd413af36a5
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Text Code Quality and Security / 2.0.1.611 / 53bfe3a68ae3886520d6a5c1c21dfdb564821ea1
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy VB.NET Code Quality and Security / 8.51.0.59060 / e14c642f118958f22fd08841dc42f9aae480366a
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy VB6 Code Quality and Security / 2.9.0.3341 / 2066b6ff3d7c789fefa0b86c6f70e9c9eb08f852
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Vulnerability Analysis / 9.9.0.19083 / 6139a53ee01a475a60f6e2944fff40f3fc00dbe1
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Vulnerability Rules for C# / 9.9.0.19083 / 6139a53ee01a475a60f6e2944fff40f3fc00dbe1
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Vulnerability Rules for JS / 9.9.0.19083 / 6139a53ee01a475a60f6e2944fff40f3fc00dbe1
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Vulnerability Rules for Java / 9.9.0.19083 / 6139a53ee01a475a60f6e2944fff40f3fc00dbe1
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Vulnerability Rules for PHP / 9.9.0.19083 / 6139a53ee01a475a60f6e2944fff40f3fc00dbe1
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy Vulnerability Rules for Python / 9.9.0.19083 / 6139a53ee01a475a60f6e2944fff40f3fc00dbe1
2023.03.21 09:10:53 INFO  web[][o.s.s.p.ServerPluginManager] Deploy XML Code Quality and Security / 2.7.0.3820 / 656bccc1910d50c8984536bd2dfd917066b858e6
2023.03.21 09:10:56 INFO  web[][o.s.s.p.d.m.c.PostgresCharsetHandler] Verify that database charset supports UTF8
2023.03.21 09:10:56 INFO  web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [http://localhost:9001]
2023.03.21 09:10:57 WARN  web[][o.s.a.s.w.WebService$Action] Description is not set on action api/monitoring/metrics
2023.03.21 09:10:57 WARN  web[][o.s.a.s.w.WebService$Action] Since is not set on action api/monitoring/metrics
2023.03.21 09:10:57 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/monitoring/metrics
2023.03.21 09:10:57 WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/system/liveness
2023.03.21 09:10:57 INFO  web[][o.s.s.p.Platform] Database needs to be migrated. Please refer to https://docs.sonarqube.org/latest/setup/upgrading
2023.03.21 09:10:57 INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.platform.web.WebServiceFilter@48c5698 [pattern=UrlPattern{inclusions=[/api/system/migrate_db.*, ...], exclusions=[/api/components/update_key, ...]}]
2023.03.21 09:10:57 INFO  web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port 9000
2023.03.21 09:12:56 INFO  app[][o.s.a.SchedulerImpl] Stopping SonarQube
2023.03.21 09:12:56 INFO  app[][o.s.a.SchedulerImpl] Sonarqube has been requested to stop
2023.03.21 09:12:56 INFO  app[][o.s.a.SchedulerImpl] Stopping [Compute Engine] process...
2023.03.21 09:12:56 INFO  app[][o.s.a.SchedulerImpl] Stopping [Web Server] process...
2023.03.21 09:12:56 INFO  web[][o.s.p.ProcessEntryPoint] Gracefully stopping process
2023.03.21 09:12:56 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown initiated...
2023.03.21 09:12:56 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown completed.
2023.03.21 09:12:56 INFO  web[][o.s.s.app.WebServer] Web Server stopped
2023.03.21 09:12:57 INFO  app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.03.21 09:12:57 INFO  app[][o.s.a.SchedulerImpl] Stopping [ElasticSearch] process...
2023.03.21 09:12:57 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 143
2023.03.21 09:12:57 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.03.21 09:12:57 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
azureadmin@connect-aks:~$!**
2

Hi,

You should check your other server logs.

 
Ann

3

It is failing due to below errors

Liveness probe condition
    Liveness:  exec [sh -c #!/bin/bash
# A Sonarqube container is considered healthy if the health status is GREEN or YELLOW
host="$(hostname -i || echo '127.0.0.1')"
if wget --header="X-Sonar-Passcode: ${SONAR_WEB_SYSTEMPASSCODE}" -qO- http://${host}:9000/api/system/health | grep -q -e '"health":"GREEN"' -e '"health":"YELLOW"'; then
  exit 0
fi
exit 1


Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  3m5s                default-scheduler  Successfully assigned dev-sonarqube/sonarqube-0 to aks-devsonar-19249943-vmss00001k
  Normal   Pulled     3m5s                kubelet            Container image "busybox:1.32" already present on machine
  Normal   Created    3m5s                kubelet            Created container init-sysctl
  Normal   Started    3m5s                kubelet            Started container init-sysctl
  Normal   Pulled     3m5s                kubelet            Container image "curlimages/curl:7.76.1" already present on machine
  Normal   Created    3m5s                kubelet            Created container inject-prometheus-exporter
  Normal   Started    3m4s                kubelet            Started container inject-prometheus-exporter
  Normal   Pulled     35s (x2 over 3m4s)  kubelet            Container image "sonarqube:lts-enterprise" already present on machine
  Normal   Created    35s (x2 over 3m4s)  kubelet            Created container sonarqube-lts
  Normal   Started    35s (x2 over 3m3s)  kubelet            Started container sonarqube-lts
  Warning  Unhealthy  35s (x3 over 95s)   kubelet            **Liveness probe failed**:
  Normal   Killing    35s                 kubelet            Container sonarqube-lts failed liveness probe, will be restarted
  Warning  Unhealthy  4s                  kubelet            Startup probe failed: Get "http://10.29.1.12:9000/api/system/status": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
4

Hi,

Are you doing that immediately after starting it up?

 
Ann

5

No we are waiting

6

Hi,

How long are you waiting?

If I’m reading this right, it looks like you start it and deem it unhealthy in the same second?

 
Ann

7

Please check the below code we have mentioned initialDelaySeconds

	    livenessProbe:
      exec:
        command:
        - sh
        - -c
        - "#!/bin/bash\n# A Sonarqube container is considered healthy if the health
          status is GREEN or YELLOW\nhost=\"$(hostname -i || echo '127.0.0.1')\"\nif
          wget --header=\"X-Sonar-Passcode: ${SONAR_WEB_SYSTEMPASSCODE}\" -qO- http://${host}:9000/api/system/health
          | grep -q -e '\"health\":\"GREEN\"' -e '\"health\":\"YELLOW\"'; then\n\texit
          0\nfi\nexit 1\n"
      failureThreshold: 3
      initialDelaySeconds: 60
      periodSeconds: 30
      successThreshold: 1
      timeoutSeconds: 1
10

HI @balakrishnateeda,

it seems the liveness probe is failing because it cannot connect to http://10.29.1.12:9000/api/system/status from within the pod.
Can you check if you can access to this IP address from within the pod?

11

I was not able go inside the pod because pod is CRASHLOOPBACKOFF

12

Since it is the liveness probe that is failing, you can try to increase the livenessProbe.initialDelaySeconds value to give you time to access the pod.

13

With same configuration we deployed on production pod is running perfectly

14

Good to hear, so there might be a network configuration in that specific environment that prevents the liveness probe to work as expected

15

No , Previously it worked perfectly now suddenly pod is restarting .
Is there any way to debug this liveness probe

16

As I suggested, you can increase the livenessProbe.initialDelaySeconds so you have time to investigate.
Then connect to one of the pod, (kubectl exec -it {podname} -- bash) and test that you have access to the liveness endpoint
wget -O- http://$ip:9000/api/system/status
where $ip is hostname -i

From there, if SonarQube is accessible but the wget times out, it means that you have a network connectivity problem that you need to investigate (The IP should be accessible from within the pod)