Sonarqube deployment failing to start in kubernetes due to Liveness check failed

ifsg-sonar · September 20, 2022, 1:10pm

version : SonarQube Server / 9.6.1.59531
enterprise-edition

we were running this from quite some time, but it started failing with below error , we are not sure what is causing this. Any suggestion on this issue will be of great help

2022.09.16 13:06:57 ERROR web[][o.s.s.w.WebServiceEngine] Fail to process request http://10.1.9.26:9000/api/system/liveness
java.lang.IllegalStateException: Liveness check failed
        at org.sonar.server.platform.ws.LivenessActionSupport.checkliveness(LivenessActionSupport.java:55)
        at org.sonar.server.platform.ws.LivenessAction.handle(LivenessAction.java:51)
        at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:111)
        at org.sonar.server.platform.web.WebServiceFilter.doFilter(WebServiceFilter.java:84)
        at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:139)
        at org.sonar.server.platform.web.SonarLintConnectionFilter.doFilter(SonarLintConnectionFilter.java:66)
        at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:139)
        at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:108)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:81)
        at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:68)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:75)
        at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:47)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:57)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.RequestIdFilter.doFilter(RequestIdFilter.java:61)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:62)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
        at jdk.internal.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAsPrivileged(Subject.java:550)
        at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
        at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
        at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:829)

ganncamp · September 21, 2022, 3:00pm

Hi,

Welcome to the community!

According to the docs:

When persistence is disabled, SonarQube startup takes significantly longer as the Elasticsearch indexes need to be rebuilt. As this delay depends on the amount of data in your SonarQube instance, the values for the startup/readiness and liveness probes need to be adjusted to your environment.

HTH,
Ann

ifsg-sonar · September 21, 2022, 3:28pm

Thanks Ann, any document that i can use to adjust rediness and liveness probe. We have this running in kubernetes environment

ganncamp · September 21, 2022, 4:11pm

Hi,

I don’t have numbers or ratios for you. I would just bump up the numbers until it works…?

Ann

ifsg-sonar · September 22, 2022, 12:16pm

Thanks Ann, tweaking numbers gave me optimum value for rediness and liveness probe.

system · September 29, 2022, 12:17pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.