Stuck in process (BulkIndexer) indexing elasticsearch. Sonar version 9.9.0-community

NOTV · March 8, 2024, 1:01pm

Make sure to tell us:

What version are you upgrading from? → 9.9.0
System information (Operating system, Java version, Database provider/version)
I’m not sure, but I use this chart → Sonarsource Chart Repository
database → postgresql server version 13.13
What’s the issue you’re facing? → when the pod sonarqube is up, it always stuck at process like this

2024.03.08 12:33:31 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [users/user] done | time=670ms
2024.03.08 12:33:31 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [projectmeasures/auth/projectmeasure]...
2024.03.08 12:34:31 INFO  web[][o.s.s.es.BulkIndexer] 0 requests processed (0 items/sec)
2024.03.08 12:35:31 INFO  web[][o.s.s.es.BulkIndexer] 0 requests processed (0 items/sec)
2024.03.08 12:36:31 INFO  web[][o.s.s.es.BulkIndexer] 1 requests processed (0 items/sec)
2024.03.08 12:37:31 INFO  web[][o.s.s.es.BulkIndexer] 1 requests processed (0 items/sec)
2024.03.08 12:38:31 INFO  web[][o.s.s.es.BulkIndexer] 1 requests processed (0 items/sec)
2024.03.08 12:39:31 INFO  web[][o.s.s.es.BulkIndexer] 2 requests processed (0 items/sec)
2024.03.08 12:40:31 INFO  web[][o.s.s.es.BulkIndexer] 2 requests processed (0 items/sec)
2024.03.08 12:41:31 INFO  web[][o.s.s.es.BulkIndexer] 3 requests processed (0 items/sec)
2024.03.08 12:42:31 INFO  web[][o.s.s.es.BulkIndexer] 3 requests processed (0 items/sec)
2024.03.08 12:43:31 INFO  web[][o.s.s.es.BulkIndexer] 3 requests processed (0 items/sec)
2024.03.08 12:44:31 INFO  web[][o.s.s.es.BulkIndexer] 4 requests processed (0 items/sec)
2024.03.08 12:45:31 INFO  web[][o.s.s.es.BulkIndexer] 4 requests processed (0 items/sec)
2024.03.08 12:46:31 INFO  web[][o.s.s.es.BulkIndexer] 5 requests processed (0 items/sec)
2024.03.08 12:47:31 INFO  web[][o.s.s.es.BulkIndexer] 5 requests processed (0 items/sec)

after that it will be error, and repeate the process BulkIndexer:

2024.03.08 12:47:31 INFO  web[][o.s.s.es.BulkIndexer] 5 requests processed (0 items/sec)
2024.03.08 12:47:38 ERROR web[][o.s.s.w.WebServiceEngine] Fail to process request http://100.73.3.151:9000/api/system/liveness
java.lang.IllegalStateException: Liveness check failed
	at org.sonar.server.platform.ws.LivenessActionSupport.checkliveness(LivenessActionSupport.java:55)
	at org.sonar.server.platform.ws.LivenessAction.handle(LivenessAction.java:51)
	at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:111)
	at org.sonar.server.platform.web.WebServiceFilter.doFilter(WebServiceFilter.java:84)
	at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:153)
	at org.sonar.server.platform.web.SonarLintConnectionFilter.doFilter(SonarLintConnectionFilter.java:66)
	at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:153)
	at org.sonar.server.platform.web.MasterServletFilter.doFilter(MasterServletFilter.java:116)
	at jdk.internal.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
	at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
	at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:81)
	at org.sonar.server.platform.web.UserSessionFilter.doFilter(UserSessionFilter.java:68)
	at jdk.internal.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
	at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
	at org.sonar.server.platform.web.CspFilter.doFilter(CspFilter.java:63)
	at jdk.internal.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
	at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
	at org.sonar.server.platform.web.CacheControlFilter.doFilter(CacheControlFilter.java:76)
	at jdk.internal.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
	at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
	at org.sonar.server.platform.web.SecurityServletFilter.doHttpFilter(SecurityServletFilter.java:60)
	at org.sonar.server.platform.web.SecurityServletFilter.doFilter(SecurityServletFilter.java:47)
	at jdk.internal.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
	at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
	at org.sonar.server.platform.web.RedirectFilter.doFilter(RedirectFilter.java:57)
	at jdk.internal.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
	at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
	at org.sonar.server.platform.web.RequestIdFilter.doFilter(RequestIdFilter.java:61)
	at jdk.internal.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
	at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
	at org.sonar.server.platform.web.RootFilter.doFilter(RootFilter.java:64)
	at jdk.internal.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
	at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
	at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:109)
	at jdk.internal.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.lambda$execute$0(SecurityUtil.java:280)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAsPrivileged(Unknown Source)
	at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:311)
	at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:187)
	at org.apache.catalina.core.ApplicationFilterChain.lambda$doFilter$0(ApplicationFilterChain.java:145)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:143)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:256)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.sonar.server.app.SecureErrorReportValve.invoke(SecureErrorReportValve.java:37)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Unknown Source)
2024.03.08 12:48:31 INFO  web[][o.s.s.es.BulkIndexer] 5 requests processed (0 items/sec)
2024.03.08 12:49:31 INFO  web[][o.s.s.es.BulkIndexer] 6 requests processed (0 items/sec)
2024.03.08 12:50:31 INFO  web[][o.s.s.es.BulkIndexer] 6 requests processed (0 items/sec)
2024.03.08 12:51:31 INFO  web[][o.s.s.es.BulkIndexer] 7 requests processed (0 items/sec)
2024.03.08 12:52:31 INFO  web[][o.s.s.es.BulkIndexer] 7 requests processed (0 items/sec)
2024.03.08 12:52:38 ERROR web[][o.s.s.w.WebServiceEngine] Fail to process request http://100.73.3.151:9000/api/system/liveness
java.lang.IllegalStateException: Liveness check failed

Before this issue happen, I try to upgrade version sonarqube from 9.3.0-community to 9.9.0-community. And I think I already successfully upgrade my DB in postgre

I also read from post Sonarqube deployment failing to start in kubernetes due to Liveness check failed, that say it need to adjust the liveness. But I try adjust to:

readinessProbe:
  initialDelaySeconds: 600
  periodSeconds: 300
  failureThreshold: 60
  timeoutSeconds: 10
livenessProbe:
  initialDelaySeconds: 600
  periodSeconds: 300
  failureThreshold: 60
  timeoutSeconds: 10
startupProbe:
  initialDelaySeconds: 300
  periodSeconds: 100
  failureThreshold: 240
  timeoutSeconds: 10

it still same

How to debug and solve it? I just curious why from that logs it always says 0 items/sec

2024.03.08 12:53:31 INFO  web[][o.s.s.es.BulkIndexer] 7 requests processed (0 items/sec)
2024.03.08 12:54:31 INFO  web[][o.s.s.es.BulkIndexer] 8 requests processed (0 items/sec)
2024.03.08 12:55:31 INFO  web[][o.s.s.es.BulkIndexer] 8 requests processed (0 items/sec)
2024.03.08 12:56:31 INFO  web[][o.s.s.es.BulkIndexer] 9 requests processed (0 items/sec)
2024.03.08 12:57:31 INFO  web[][o.s.s.es.BulkIndexer] 9 requests processed (0 items/sec)

jeremy.cotineau · March 26, 2024, 9:09am

Hello @NOTV, thanks a lot for participating in the community.

Could you please confirm which version you are currently running and which one is the target?

For the upgrade path, you need to be sure you are running it from SonarQube 9.9 if you want to upgrade to 10.X, so having a stable 9.9 is a prerequisite.

Otherwise considering the issue state indexing problem, and if you have persistency enabled, I would suggest scaling down to 0 the statefulset, then deleting the PVC, and scaling back to X.

Let me know if this help and if we should troubleshoot forward

NOTV · March 28, 2024, 4:12am

Current version running (still in use) → 9.3.0-community
I still run PoC to upgrade to LTS version (9.9.0-community / 9.9.4-community)
Actually I don’t want to upgrade to 10.x (because its not LTS right now)

I also already run vacuum and reindexing based on this docs → Performing the post-upgrade steps | SonarQube Docs

But I try your suggestion to scalling down 0 the statefull and delete the pvc