Background task fails for project analysis

Carl · December 1, 2022, 2:36pm

Must-share information (formatted with Markdown):

which versions are you using
- SonarQube 9.7.1
- SonarScanner 4.7.0
- Sonar.CXX plugin 2.1.0 (for cobertura coverage reports)
- Microsoft AAD plugin 1.3.2
what are you trying to achieve
Getting the project analysis working again
what have you tried so far to achieve this
- Removing data/es7 which has worked before after upgrading (we recently upgraded to 9.7.1, but I’m not sure this is the reason).
- Researching on SonarQube Community and StackOverflow if somebody has the same issue (a lot of similar ones at least, e.g. Background Task fails)

On the project analysis of the master branch we get a failure every time for 4 days now. The current failure is the following (slightly redacted).

org.sonar.ce.task.projectanalysis.component.VisitException: Visit of Component {key=<project>:<C++ header file in project>,uuid=AYBqK7eNHieUv3R8V52S,type=FILE} failed
	at org.sonar.ce.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:41)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:95)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:95)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:95)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:95)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:95)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:95)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	at org.sonar.ce.task.projectanalysis.source.PersistFileSourcesStep.execute(PersistFileSourcesStep.java:73)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Cannot persist sources of <project>:<C++ header file in project>
	at org.sonar.ce.task.projectanalysis.source.PersistFileSourcesStep$FileSourceVisitor.visitFile(PersistFileSourcesStep.java:99)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitNode(DepthTraversalTypeAwareCrawler.java:76)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:51)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	... 38 more
Caused by: java.lang.IllegalStateException: Failed to read '/opt/sonarqube/temp/ce/5377421360800459520/150952544466248886/hashes-252'
	at org.sonar.ce.task.projectanalysis.source.SourceLinesHashCache.load(SourceLinesHashCache.java:86)
	at org.sonar.ce.task.projectanalysis.source.SourceLinesHashCache.get(SourceLinesHashCache.java:61)
	at org.sonar.ce.task.projectanalysis.source.SourceLinesHashRepositoryImpl.getLineHashesComputerToPersist(SourceLinesHashRepositoryImpl.java:70)
	at org.sonar.ce.task.projectanalysis.source.FileSourceDataComputer.compute(FileSourceDataComputer.java:48)
	at org.sonar.ce.task.projectanalysis.source.PersistFileSourcesStep$FileSourceVisitor.visitFile(PersistFileSourcesStep.java:96)
	... 41 more
Caused by: java.nio.file.NoSuchFileException: /opt/sonarqube/temp/ce/5377421360800459520/150952544466248886/hashes-252
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
	at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
	at java.base/java.nio.file.Files.newByteChannel(Files.java:422)
	at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
	at java.base/java.nio.file.Files.newInputStream(Files.java:156)
	at java.base/java.nio.file.Files.newBufferedReader(Files.java:2839)
	at java.base/java.nio.file.Files.readAllLines(Files.java:3330)
	at org.sonar.ce.task.projectanalysis.source.SourceLinesHashCache.load(SourceLinesHashCache.java:84)
	... 45 more

ganncamp · December 2, 2022, 5:58pm

Hi,

Can you remove the Cxx plugin and see if this still happens?

Ann

Carl · December 5, 2022, 9:31am

Why do you suspect the CXX plugin? We need if for code coverage, because SonarQube still doesn’t support Cobertura format for C++, so I would rather not change that.

My guess would be if anything it’s the version upgrade to 9.7.1, which I’m assuming due to the DB upgrade to be one way. CXX doesn’t show in the stacktrace, and the compute engine is part of 9.7.1.

Carl · December 6, 2022, 2:25pm

@ganncamp I now tried disabling the CXX extension, doing a project analysis afterwards still failed for me in the same way. So it’s definitely a Bug in 9.7.1.

I also found this old Google Groups entry:
https://groups.google.com/g/sonarqube/c/_ABNeSrItdc

They say it could be corrupted entries in a database which could be resolved by redoing the analysis with the files filtered out and then again without the filter.

I tried that too, but it didn’t help. I’m still getting the same issue. Even when excluding the whole repo, doing an analysis that would remove all corrupted entries from the DB and redoing the analysis.

All the files mentioned are present in my workspace when running the analysis.

Carl · December 6, 2022, 2:50pm

I also tried downgrading to 9.5.0 but that fails to start. I’m guessing downgrades are not possible after DB migration?

Carl · December 6, 2022, 3:47pm

Maybe another interesting datapoint. Looking at the error message the file in question is usually changing. When I rerun the same analysis. I can check the database table “file_sources” and search for the md5 hash of the file as src_hash and I get multiple results, I’m guessing one less than the number of times this file was analyzed. Since if I look for the exact UUID it’s missing.

So the tool is internaly not handling these files correctly.

ganncamp · December 7, 2022, 12:48pm

Hi,

Thanks for the followup. I asked you to try again without Cxx because sometimes plugins do wacky stuff.

I’m out of my depth, so I’m going to flag this for more expert eyes.

(And yes, once you’ve upgraded the database, you need to restore a backup to downgrade.)

Ann

Carl · December 7, 2022, 2:12pm

@ganncamp Thank you. In the meantime I may have found the root cause. I’m saying may because the whole-project analysis now worked 4 times, and considering that it was only working 1/7 times before this seems quite unlikely to not have had an effect, but I’m not sure if it fixed all the issues.

A new failure popped up which lead me to this thread: Fail to execute es requestES refresh request on indices 'components' - #8 by shiva_sq

I tried upping the elasticsearch memory to 4GB, by using sonar.search.javaOpts=-Xmx4G -Xms4G -XX:+HeapDumpOnOutOfMemoryError.

Since I was now getting OOM on startup I scaled down the heap size for the compute engine and the web engine. They were both set to 32GB. Now I set:

jvmOpts: "-Xmx16G -Xms1G -XX:+HeapDumpOnOutOfMemoryError"
jvmCeOpts: "-Xmx16G -Xms1G -XX:+HeapDumpOnOutOfMemoryError"

Hint: If you’re reading this and not using Helm, I think these should be sonar.web.javaOpts and sonar.ce.javaOpts. In the Helm file there’s currently no setting for sonar.search.javaOpts so I had to add that to the normal settings.

system · December 16, 2022, 12:54pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.