Some tasks failed or cancelled causing some projects to remain unavailable

SonarQube completed the reload of project data. Some tasks failed or cancelled causing some projects to remain unavailable.

I upgraded my Sonarqube Enterprise to version 9.5 however after upgrading it’s telling me that it failed in Backgroiund Tasks “45 still failing”. I’m also waiting for support to provide me the necessary license for my new db, is this the reason why it’s failing in some tasks?

Appreciate for any help. Here is the output of my ce.log:

[root@pgm010b07 logs]# tail -f ce.log
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=54ms
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=11ms
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=0ms
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=7ms
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.s.ComputationStepExecutor] Initialize | status=FAILED | time=10ms
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Webhooks | globalWebhooks=3 | projectWebhooks=0 | status=SUCCESS | time=104ms
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Pull Request decoration | status=SUCCESS | time=0ms
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Report branch Quality Gate status to devops platforms | status=SUCCESS | time=0ms
2022.09.26 00:06:22 INFO ce[AYN4nkZ6SNLkt3uLkIHP][o.s.c.t.CeWorkerImpl] Executed task | project=rxrp-incentiveapp-ui | type=REPORT | id=AYN4nkZ6SNLkt3uLkIHP | submitter=admin | status=FAILED | time=258ms

Not sure if it has something to do with the error that I’m encountering but I tailed ce.log and encountered these status=FAILED :

2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=49ms
2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=11ms
2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=0ms
2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.s.ComputationStepExecutor] Generate analysis UUID | status=SUCCESS | time=0ms
2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.s.ComputationStepExecutor] Load analysis metadata | status=SUCCESS | time=13ms
2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.s.ComputationStepExecutor] Initialize | status=FAILED | time=12ms
2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Webhooks | globalWebhooks=3 | projectWebhooks=0 | status=SUCCESS | time=103ms
2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Pull Request decoration | status=SUCCESS | time=0ms
2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Report branch Quality Gate status to devops platforms | status=SUCCESS | time=0ms
2022.09.26 00:08:27 INFO ce[AYN4oCxbSNLkt3uLkIHR][o.s.c.t.CeWorkerImpl] Executed task | project=com.albertsons.memsp:memsp-experience-svc | type=REPORT | branch=fb-iimp-418-itemUpdate | branchType=BRANCH | id=AYN4oCxbSNLkt3uLkIHR | submitter=admin | status=FAILED | time=244ms

I checked my dev environment and there’s on failure. Right now there’s 50 failing in PROD not sure if this is just normal or not (screenshot has 41 failing when i took a screen cap).

I tried building and here is the error of my build:

java.lang.IllegalStateException: SonarQube analysis ‘AYN4vZC7SNLkt3uLkIHx’ failed: FAILED
at org.sonarsource.scanner.jenkins.pipeline.WaitForQualityGateStep$Execution.checkQualityGate(WaitForQualityGateStep.java:298)
at org.sonarsource.scanner.jenkins.pipeline.WaitForQualityGateStep$Execution.checkTaskCompleted(WaitForQualityGateStep.java:239)
at org.sonarsource.scanner.jenkins.pipeline.WaitForQualityGateStep$Execution.start(WaitForQualityGateStep.java:175)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:322)
at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:196)
at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:124)
at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1295)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1184)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:41)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
at albBuild.sonarQualityGate(albBuild.groovy:48)
at cps.transform(Native Method)
at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixName(FunctionCallBlock.java:78)
at sun.reflect.GeneratedMethodAccessor461.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
at com.cloudbees.groovy.cps.Next.step(Next.java:83)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:136)
at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:275)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:187)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:420)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:95)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:330)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:294)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Finished: FAILURE

Hey there.

I would suggest looking at your es.log file (and you probably want to view the full file, not tail it to get a full picture of what’s going on).

Hi Colin,

I attached my es.log in this thread. If you could please help me troubleshoot. Thanks.
es.2022-09-25.log (11.6 KB)
es.log (5.0 KB)

It looks like you were facing some disk-watermark issues that are now resolved.

2022.09.25 23:30:58 WARN  es[][o.e.c.r.a.DiskThresholdMonitor] flood stage disk watermark [95%] exceeded on [LcDcvnUqQjKvmI0zn4h7Kw][sonarqube][/opt/sonarqube-9.5.0.56709/data/es7/nodes/0] free: 353mb[3.4%], all indices on this node will be marked read-only
.....
2022.09.25 23:36:58 INFO  es[][o.e.c.r.a.DiskThresholdMonitor] low disk watermark [85%] no longer exceeded on [LcDcvnUqQjKvmI0zn4h7Kw][sonarqube][/opt/sonarqube-9.5.0.56709/data/es7/nodes/0] free: 1.6gb[16.8%]

You should make sure it stays this way by increasing the disk space on your server some.

And, to re-index the project data, I can suggest:

  1. Stop your SonarQube instance
  2. Delete the data/es7 folder
  3. Start your SonarQube instance

hi Colin, is this flood stage disk watermark, is this pointing out to the disk storage itself where sonarqube application is installed?

Will try the steps that you mentioned above.

Yes it is!

@Colin thanks again for sharing your expertise. I was able to upgrade my prod instance to 9.5!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.