Sonarqube deployment failing to start in kubernetes due to Liveness check failed

  • version : SonarQube Server / 9.6.1.59531
  • enterprise-edition

we were running this from quite some time, but it started failing with below error , we are not sure what is causing this. Any suggestion on this issue will be of great help

2022.09.16 13:06:57 ERROR web[][o.s.s.w.WebServiceEngine] Fail to process request http://10.1.9.26:9000/api/system/liveness
java.lang.IllegalStateException: Liveness check failed
        at org.sonar.server.platform.ws.LivenessActionSupport.checkliveness(LivenessActionSupport.java:55)
        at org.sonar.server.platform.ws.LivenessAction.handle(LivenessAction.java:51)
        at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:111)
        at org.sonar.server.platform.web.WebServiceFilter.doFilter(WebServiceFilter.java:84)
        at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:139)
        at org.sonar.server.platform.web.SonarLintConnectionFilter.doFilter(SonarLintConnectionFilter.java:66)
        at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:139)
        at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:108)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:81)
        at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:68)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:75)
        at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:47)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:57)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.RequestIdFilter.doFilter(RequestIdFilter.java:61)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:62)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:829)

Hi,

Welcome to the community!

According to the docs:

When persistence is disabled, SonarQube startup takes significantly longer as the Elasticsearch indexes need to be rebuilt. As this delay depends on the amount of data in your SonarQube instance, the values for the startup/readiness and liveness probes need to be adjusted to your environment.

 
HTH,
Ann

Thanks Ann, any document that i can use to adjust rediness and liveness probe. We have this running in kubernetes environment

Hi,

I don’t have numbers or ratios for you. I would just bump up the numbers until it works…?

 
Ann

Thanks Ann, tweaking numbers gave me optimum value for rediness and liveness probe.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.