Random Quality Gate Failures

Hello,
We are using SonarQube 8.9 LTS in OpenShift with a Postgres 13 Database and Jenkins as an Orchestrator. Since upgrading to the latest LTS version of SonarQube, we are seeing this error randomly. Generally works itself out if we run another pipeline or two, but it is still problematic. The error fails in the Quality Gate Stage, saying that background tasks failed. We would like to know what may be causing these random errors and what we may be able to do about it.
Thanks,
Jason Butler


Error Details
org.sonar.ce.task.projectanalysis.component.VisitException: Visit of Component {key=dotnet50-tester:contactsapi.tests/coverage.opencover.xml,uuid=AXq_RL0uyp7rhJ1ggY_L,type=FILE} failed
	at org.sonar.ce.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:41)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:95)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitChildren(DepthTraversalTypeAwareCrawler.java:95)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:54)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	at org.sonar.ce.task.projectanalysis.source.PersistFileSourcesStep.execute(PersistFileSourcesStep.java:72)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Cannot persist sources of dotnet50-tester:contactsapi.tests/coverage.opencover.xml
	at org.sonar.ce.task.projectanalysis.source.PersistFileSourcesStep$FileSourceVisitor.visitFile(PersistFileSourcesStep.java:105)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitNode(DepthTraversalTypeAwareCrawler.java:76)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visitImpl(DepthTraversalTypeAwareCrawler.java:51)
	at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCrawler.java:39)
	... 26 more
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: org.postgresql.util.PSQLException: ERROR: unexpected data beyond EOF in block 16816 of relation base/16385/16492
  Hint: This has been seen to occur with buggy kernels; consider updating your system.
### The error may exist in org.sonar.db.source.FileSourceMapper
### The error may involve org.sonar.db.source.FileSourceMapper.update-Inline
### The error occurred while setting parameters
### SQL: update       file_sources     set       updated_at = ?,       binary_data = ?,       line_hashes = ?,       line_hashes_version = ?,       line_count = ?,       data_hash = ?,       src_hash = ?,       revision = ?     where       uuid = ?
### Cause: org.postgresql.util.PSQLException: ERROR: unexpected data beyond EOF in block 16816 of relation base/16385/16492
  Hint: This has been seen to occur with buggy kernels; consider updating your system.
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:199)
	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:67)
	at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152)
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85)
	at com.sun.proxy.$Proxy39.update(Unknown Source)
	at org.sonar.db.source.FileSourceDao.update(FileSourceDao.java:122)
	at org.sonar.ce.task.projectanalysis.source.PersistFileSourcesStep$FileSourceVisitor.persistSource(PersistFileSourcesStep.java:150)
	at org.sonar.ce.task.projectanalysis.source.PersistFileSourcesStep$FileSourceVisitor.visitFile(PersistFileSourcesStep.java:103)
	... 29 more
Caused by: org.postgresql.util.PSQLException: ERROR: unexpected data beyond EOF in block 16816 of relation base/16385/16492
  Hint: This has been seen to occur with buggy kernels; consider updating your system.
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2285)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164)
	at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:153)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
	at org.apache.ibatis.executor.ReuseExecutor.doUpdate(ReuseExecutor.java:52)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197)
	... 36 more

Close

Hi,

I’m struck by this part of the log, which indicates the problem lies outside either SonarQube or your DB:

Have you considered applying system updates?

 
Ann

Hi Ann,
We have recently updated SonarQube, Postgres, and OpenShift. We are on the latest LTS version of SonarQube and Postgres, and the most recent version of OpenShift 3.11. We also saw similar errors when doing the upgrade SonarQube 8.9 LTS. Please see this topic for details: Sonarqube Upgrade 8.6 - #14 by pierreguillot.
Ultimately we had to upgrade in an empty directory as opposed to a PVC and the we could port the upgraded DB into an actual PVC.

Hi,

Thanks for the pointer to the other thread. It seems like your use of PVC is a persistent problem. :wink:

Jokes aside, this really seems to lie outside SonarQube, between the DB and the PVC.

 
:woman_shrugging:
Ann