Two nights in a row our project analyses for the master branch stopped working. The problem shows up in a timerange where a lot of project analyses are startet.
In the first minutes of heavy load everything normal.
Then a few ConcurrentModificationExceptions in the log, analyses still working.
Then the first NullPointerException shows up and the project analyses for master branches start to fail.
Every project analyses afterwards for the master branch fails and logs the same exception.
Every project analyses on any other branches always run successfully.
Every application and portfolio recomputations always run successfully.
Because of the ConcurrentModificationExceptions we set the number of background task workers to 1 (from 3). This didn’t resolve the NPE (but still might play its role at the beginning).
After restarting sonarqube all analyses run normal again.
We see the ConcurrentModificationExceptions since the upgrade from 10.0 to 10.1. It also breaks the analysis but occures only randomly.
2447 projects, 241 applications and 105 portfolios
sonar settings default (as set in the images), except “sonar.ce.javaOpts=-Xmx3g”
For every failed task we see In the log:
Failed to execute task <ID>
and
java.lang.NullPointerException: Cannot read field "next" because "this.next" is null
at java.base/java.util.LinkedList$ListItr.next(Unknown Source)
at java.base/java.lang.Iterable.forEach(Unknown Source)
at com.A.D.O.C.B$_A.G(Unknown Source)
at com.A.D.O.C.B$_A.D(Unknown Source)
at com.sonarsource.A.D.A(Unknown Source)
at com.A.D.O.C.B.A(Unknown Source)
at com.A.D.O.D.G.A(Unknown Source)
at com.A.D.O.D.G.A(Unknown Source)
at com.A.D.O.D.G.B(Unknown Source)
at com.A.D.O.D.G.A(Unknown Source)
at com.A.D.C.A.A.triggerFrom(Unknown Source)
at org.sonar.ce.task.projectanalysis.step.TriggerViewRefreshStep.execute(TriggerViewRefreshStep.java:63)
at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
It looks like the 10.1 version has introduced some concurrency issues?
We encountered the same NullPointerException error on project analysis background tasks last week. Project analysis tasks for pull requests remained successful but every project analysis task on any branch (not just the default branch) failed with this NPE error. I’ve included an example stacktrace below because it has some line numbers not present in Roland’s.
We were able to (temporarily) resolve the issue by restarting the SonarQube service, however we’re concerned about it occurring again and causing additional disruptions.
This is our SonarQube configuration:
SonarQube Enterprise Edition - Version 10.1 (build 73491)
Running on an AWS EC2 instance (Linux, AMD64, m6a.xlarge) using the Enterprise package download (Download | SonarQube)
Java 17
Compute Engine properties: sonar.ce.javaOpts=-Xmx4G -Xms128m -XX:+HeapDumpOnOutOfMemoryError (increased from default -Xmx2G to -Xmx4G a couple weeks ago due to an out of memory incident)
Number of Workers: 2
1903 projects, 164 portfolios
Error Details:
java.lang.NullPointerException: Cannot read field "next" because "this.next" is null
at java.base/java.util.LinkedList$ListItr.next(LinkedList.java:897)
at java.base/java.lang.Iterable.forEach(Iterable.java:74)
at com.A.D.O.C.B$_A.G(Unknown Source)
at com.A.D.O.C.B$_A.D(Unknown Source)
at com.sonarsource.A.D.A(Unknown Source)
at com.A.D.O.C.B.A(Unknown Source)
at com.A.D.O.D.G.A(Unknown Source)
at com.A.D.O.D.G.A(Unknown Source)
at com.A.D.O.D.G.B(Unknown Source)
at com.A.D.O.D.G.A(Unknown Source)
at com.A.D.C.A.A.triggerFrom(Unknown Source)
at org.sonar.ce.task.projectanalysis.step.TriggerViewRefreshStep.execute(TriggerViewRefreshStep.java:63)
at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
We “solved” it at the moment by setting the numbers of workers to 1 (without concurrency the LinkedList can’t get corrupted).
But this has the drawback that the pending tasks gets very high some times which causes much delay in the results.
Thank you all for reporting this issue.
It has been identified to be a concurrency issue with a stateful component shared between different compute engine tasks, while it should have a unique instance per task.
It can be tracked on this bug ticket SONAR-20215 and should be fixed for the next SQ release (10.2).