Background Tasks Cancelled with database error in ce.log

  • Versions used (SonarQube, Scanner, Plugin, and any relevant extension)

SonarQube developer edition - 8.9.0.43852

PostgreSQL version 9.5.14 - Driver Version 42.2.19

Both SonarQube and PostgreSQL are running in our on-site openshift solution.

Container spec for PostgreSQL:

containers:
        - resources:
            limits:
              cpu: 250m
              memory: 256Mi
            requests:
              cpu: 100m
              memory: 100Mi
  • Error observed (wrap logs/code around triple quote ``` for proper formatting)

When analyzing a big project, 247k lines mostly javascript, the analysis log reports Execution sucess

INFO: Analysis report generated in 5263ms, dir size=50 MB
INFO: Analysis report compressed in 14719ms, zip size=14 MB
INFO: Analysis report uploaded in 9785ms
INFO: ANALYSIS SUCCESSFUL, you can browse https://sonarqube-bygg.apps.ocpdq02.norsk-tipping.no/dashboard?id=omniweb
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at https://sonarqube-bygg.apps.ocpdq02.norsk-tipping.no/api/ce/task?id=AXn6tgE_goKcK2s6Mn5F
INFO: Analysis total time: 5:50.239 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 5:53.297s
INFO: Final Memory: 273M/924M
INFO: ------------------------------------------------------------------------

After this the background task in SonarQube starts up. It ends up in a Cancelled state.

Scanner Context:

SonarQube plugins:
  - CSS Code Quality and Security 1.4.2.2002 (cssfamily)
  - PL/SQL Code Quality and Security 3.6.1.3873 (plsql)
  - Scala Code Quality and Security 1.8.3.2219 (sonarscala)
  - C# Code Quality and Security 8.22.0.31243 (csharp)
  - Vulnerability Analysis 8.9.0.11439 (security)
  - Java Code Quality and Security 6.15.1.26025 (java)
  - HTML Code Quality and Security 3.4.0.2754 (web)
  - Flex Code Quality and Security 2.6.1.2564 (flex)
  - XML Code Quality and Security 2.2.0.2973 (xml)
  - SonarTS 2.1.0.4359 (typescript)
  - VB.NET Code Quality and Security 8.22.0.31243 (vbnet)
  - Swift Code Quality and Security 4.3.1.4892 (swift)
  - Dependency-Check 2.0.8 (dependencycheck)
  - CFamily Code Quality and Security 6.20.0.31240 (cpp)
  - Python Code Quality and Security 3.4.1.8066 (python)
  - Go Code Quality and Security 1.8.3.2219 (go)
  - JaCoCo 1.1.1.1157 (jacoco)
  - Kotlin Code Quality and Security 1.8.3.2219 (kotlin)
  - T-SQL Code Quality and Security 1.5.1.4340 (tsql)
  - JavaScript/TypeScript Code Quality and Security 7.4.2.15501 (javascript)
  - Ruby Code Quality and Security 1.8.3.2219 (ruby)
  - Vulnerability Rules for C# 8.9.0.11439 (securitycsharpfrontend)
  - Vulnerability Rules for Java 8.9.0.11439 (securityjavafrontend)
  - License for SonarLint 8.9.0.43852 (license)
  - Vulnerability Rules for JS 8.9.0.11439 (securityjsfrontend)
  - Groovy 1.6 (groovy)
  - Vulnerability Rules for Python 8.9.0.11439 (securitypythonfrontend)
  - PHP Code Quality and Security 3.17.0.7439 (php)
  - ABAP Code Quality and Security 3.9.1.3127 (abap)
  - Vulnerability Rules for PHP 8.9.0.11439 (securityphpfrontend)
Global server settings:
  - email.from=noreply@norsk-tipping.no
  - email.smtp_host.secured=******
  - sonar.core.id=5E032CD4-AWijrASnRA_j8d-xhy4X
  - sonar.core.serverBaseURL=https://sonarqube-bygg.apps.ocpdq02.norsk-tipping.no
  - sonar.core.startTime=2021-06-07T09:06:24+0000
  - sonar.dbcleaner.branchesToKeepWhenInactive=master,develop,trunk,branch-.*,release-.*
  - sonar.dependencyCheck.htmlReportPath=dependency-check-report.html
  - sonar.dependencyCheck.xmlReportPath=dependency-check-report.xml
  - sonar.forceAuthentication=false
  - sonar.javascript.file.suffixes=.js,.jsx,.vue,.es6
  - sonar.javascript.lcov.reportPaths=coverage/lcov.info
  - sonar.license.notifications.remainingLocThreshold=100000
  - sonar.plugins.risk.consent=ACCEPTED
  - sonar.technicalDebt.ratingGrid=0.01,0.05,0.1,0.2
Project server settings:
Project scanner properties:
  - sonar.exclusions=packages/**/node_modules/**/*, packages/app-web/build/**/*, packages/app-web/test/**/*, node_modules/**/*
  - sonar.host.url=https://sonarqube-bygg.apps.ocpdq02.norsk-tipping.no
  - sonar.javascript.file.suffixes=.js,.jsx
  - sonar.javascript.lcov.reportPath=coverage/lcov.info
  - sonar.language=js
  - sonar.projectBaseDir=/Users/a01583/Git/omniweb
  - sonar.projectKey=omniweb
  - sonar.projectName=Omniweb
  - sonar.scanner.app=ScannerCli
  - sonar.scanner.appVersion=4.3.0.2102
  - sonar.sourceEncoding=UTF-8
  - sonar.sources=packages
  - sonar.test.inclusions=packages/**/*.test.jsx, packages/**/*.test.js
  - sonar.working.directory=/Users/a01583/Git/omniweb/.scannerwork

Furthermore, the ce.log issues these errors:

2021.06.11 10:33:09 INFO  ce[AXn6oaPfgoKcK2s6Mn5C][o.s.c.t.s.ComputationStepExecutor] Persist components | status=SUCCESS | time=160ms
2021.06.11 10:33:09 INFO  ce[AXn6oaPfgoKcK2s6Mn5C][o.s.c.t.s.ComputationStepExecutor] Persist analysis | status=SUCCESS | time=32ms
2021.06.11 10:33:09 INFO  ce[AXn6oaPfgoKcK2s6Mn5C][o.s.c.t.s.ComputationStepExecutor] Persist analysis properties | status=SUCCESS | time=42ms
2021.06.11 10:33:09 INFO  ce[AXn6oaPfgoKcK2s6Mn5C][o.s.c.t.s.ComputationStepExecutor] Persist measures | inserts=64 | status=SUCCESS | time=54ms
2021.06.11 10:33:11 WARN  ce[AXn6oaPfgoKcK2s6Mn5C][o.a.i.e.BaseExecutor] Unexpected exception on closing transaction.
  Cause: org.postgresql.util.PSQLException: This connection has been closed.
2021.06.11 10:33:11 INFO  ce[AXn6oaPfgoKcK2s6Mn5C][o.s.c.t.s.ComputationStepExecutor] Persist live measures | status=FAILED | time=1124ms
2021.06.11 10:33:11 ERROR ce[AXn6oaPfgoKcK2s6Mn5C][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Execution of task 
class org.sonar.ce.task.projectanalysis.webhook.WebhookPostTask failed
org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
### The error may exist in org.sonar.db.project.ProjectMapper
### The error may involve org.sonar.db.project.ProjectMapper.selectByUuid
### The error occurred while executing a query
### Cause: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
        at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:149)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:76)
        at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:87)
        at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:152)
        at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:85)
        at com.sun.proxy.$Proxy36.selectByUuid(Unknown Source)
        at org.sonar.db.project.ProjectDao.selectByUuid(ProjectDao.java:76)
        at org.sonar.server.webhook.WebHooksImpl.readWebHooksFrom(WebHooksImpl.java:71)
at org.sonar.server.webhook.WebHooksImpl.readWebHooksFrom(WebHooksImpl.java:71)
        at org.sonar.server.webhook.WebHooksImpl.sendProjectAnalysisUpdateImpl(WebHooksImpl.java:106)        at org.sonar.server.webhook.WebHooksImpl.sendProjectAnalysisUpdate(WebHooksImpl.java:101)
        at org.sonar.ce.task.projectanalysis.webhook.WebhookPostTask.finished(WebhookPostTask.java:59)
        at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.executeTask(PostProjectAna
lysisTasksExecutor.java:118)
        at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.finished(PostProjectAnalys
isTasksExecutor.java:109)
        at org.sonar.ce.task.step.ComputationStepExecutor.executeListener(ComputationStepExecutor.java:91)
        at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:63)
        at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterrupti
bly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:350)
        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)
        at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)
        at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)
        at org.postgresql.jdbc.PgConnection.execSQLUpdate(PgConnection.java:497)
        at org.postgresql.jdbc.PgConnection.setTransactionIsolation(PgConnection.java:952)
        at org.apache.commons.dbcp2.DelegatingConnection.setTransactionIsolation(DelegatingConnection.java:575)
        at org.apache.commons.dbcp2.DelegatingConnection.setTransactionIsolation(DelegatingConnection.java:575)
        at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:141)
        at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:61)
        at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:337)
        at org.apache.ibatis.executor.ReuseExecutor.prepareStatement(ReuseExecutor.java:88)
        at org.apache.ibatis.executor.ReuseExecutor.doQuery(ReuseExecutor.java:59)
        at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
        at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
        at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
        ... 31 common frames omitted
Caused by: java.net.SocketException: Connection reset
        at java.base/java.net.SocketInputStream.read(Unknown Source)
        at java.base/java.net.SocketInputStream.read(Unknown Source)
        at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:161)
        at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:128)
        at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:113)
        at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)

Followed by this:

2021.06.11 10:33:12 ERROR ce[AXn6oaPfgoKcK2s6Mn5C][o.s.c.t.CeWorkerImpl] Failed to execute task AXn6oaPfgoKcK2s6Mn5C
org.sonar.ce.task.projectanalysis.component.VisitException: Visit of Component {key=omniweb,uuid=AXnqzmXXgoKcK2s6MnuA
,type=PROJECT} failed
        at org.sonar.ce.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
        at org.sonar.ce.task.projectanalysis.component.DepthTraversalTypeAwareCrawler.visit(DepthTraversalTypeAwareCr
awler.java:41)
        at org.sonar.ce.task.projectanalysis.step.PersistLiveMeasuresStep.execute(PersistLiveMeasuresStep.java:86)
        at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:81)
        at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:72)
        at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:59)
        at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:81)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterrupti
bly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error committing transaction.  Cause: org.apache.ibatis.executor.BatchExecutorException: org.sonar.db.measure.Liv
eMeasureMapper.upsert (batch index #1) failed. Cause: java.sql.BatchUpdateException: Batch entry 0 insert into live_m
easures (
      uuid,
      component_uuid,
      project_uuid,
      metric_uuid,
      value,
      text_value,
      variation,
      measure_data,
      created_at,
      updated_at
    ) values (
      'AXn6oft44FNX_EGajoJy',
      'AXnqzmXXgoKcK2s6MnuA',
      'AXnqzmXXgoKcK2s6MnuA',

and this

Caused by: org.apache.ibatis.executor.BatchExecutorException: org.sonar.db.measure.LiveMeasureMapper.upsert (batch in
dex #1) failed. Cause: java.sql.BatchUpdateException: Batch entry 0 insert into live_measures (
      uuid,
      component_uuid,
      project_uuid,
      metric_uuid,
      value,
      text_value,
      variation,
      measure_data,
      created_at,
      updated_at
    ) values (
      'AXn6oft44FNX_EGajoJy',
      'AXnqzmXXgoKcK2s6MnuA',
      'AXnqzmXXgoKcK2s6MnuA',
      '50',
      NULL,
      NULL,
      0.0,
      NULL,
      1623407590264,
      1623407590264
    )
    on conflict(component_uuid, metric_uuid) do update set
      value = excluded.value,
      variation = excluded.variation,
      text_value = excluded.text_value,
      measure_data  = excluded.measure_data,
      updated_at = excluded.updated_at
    where
      live_measures.value is distinct from excluded.value or
      live_measures.variation is distinct from excluded.variation or
      live_measures.text_value is distinct from excluded.text_value or
      live_measures.measure_data is distinct from excluded.measure_data was aborted: An I/O error occurred while send
ing to the backend.  Call getNextException to see other errors in the batch.
        at org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:149)
        at org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:129)
        at org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:122)
        at org.apache.ibatis.executor.BaseExecutor.commit(BaseExecutor.java:242)
        at org.apache.ibatis.executor.CachingExecutor.commit(CachingExecutor.java:119)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.commit(DefaultSqlSession.java:223)

Followed by some more of these kind of messages.

And it ends up telling me that:

2021.06.11 10:33:12 ERROR ce[][o.s.c.t.CeWorkerImpl] Call to listener ReportAnalysisFailureNotificationExecutionListe
ner failed.
org.apache.ibatis.exceptions.PersistenceException: 
### Error getting a new connection.  Cause: org.postgresql.util.PSQLException: FATAL: the database system is in recov
ery mode
### Cause: org.postgresql.util.PSQLException: FATAL: the database system is in recovery mode
        at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
        at org.apache.ibatis.session.defaults.DefaultSqlSession.getConnection(DefaultSqlSession.java:299)
        at org.sonar.db.DbSessionImpl.getConnection(DbSessionImpl.java:192)
        at org.sonar.db.property.PropertiesDao.hasProjectNotificationSubscribersForDispatchers(PropertiesDao.java:104
)
        at org.sonar.server.notification.NotificationService.hasProjectSubscribersForTypes(NotificationService.java:1
66)
        at org.sonar.ce.notification.ReportAnalysisFailureNotificationExecutionListener.onEnd(ReportAnalysisFailureNo
tificationExecutionListener.java:74)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.lambda$finalizeTask$1(CeWorkerImpl.java:255)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.lambda$callListeners$2(CeWorkerImpl.java:262)
        at java.base/java.util.Arrays$ArrayList.forEach(Unknown Source)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.callListeners(CeWorkerImpl.java:260)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.finalizeTask(CeWorkerImpl.java:255)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.afterExecute(CeWorkerImpl.java:239)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.close(CeWorkerImpl.java:199)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:161)
        at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
        at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
        at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterrupti
bly(TrustedListenableFutureTask.java:125)
        at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)
        at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.postgresql.util.PSQLException: FATAL: the database system is in recovery mode
        at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:613)
        at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:161)
        at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:213)
        at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
        at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223)
        at org.postgresql.Driver.makeConnection(Driver.java:465)
        at org.postgresql.Driver.connect(Driver.java:264)
        at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:55)
        at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355)
        at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:889)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:424)
        at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:349)
        at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:134)
        at org.apache.commons.dbcp2.BasicDataSource$PaGetConnection.run(BasicDataSource.java:73)

Everything else in SonarQube works ok. I can browse other projects and when another, smaller project is analysed later on it works without further issues. It only fails when the background taks is running for this, my biggest, project.

How do I solve this? Is there a memory issue I can fix? Anything else?

  • Steps to reproduce

This isn’t very easy to reproduce for me other than running the analysis for this exact big project.

  • Potential workaround

I have found no workaround

  • Scanner command used when applicable (private details masked)

sonar-scanner

Hello @haugsrud ,

welcome to the community :wave:

your limits to the database container seem rather slim, do you have monitoring on your database? does the resource limit get hit often?
my assumption is that the database container is not able to provide the resources necessary to perform the requested action resulting in a rollback

could you validate this?

Hi and thank you for your response. I can try setting the limits on the database higher.

By validate, do you mean validate that my database is in recovery mode? Looks like it from the postgre-logs below:

LOG:  server process (PID 326291) was terminated by signal 9: Killed
DETAIL:  Failed process was running: insert into live_measures (
              uuid,
              component_uuid,
              project_uuid,
              metric_uuid,
              value,
              text_value,
              variation,
              measure_data,
              created_at,
              updated_at
            ) values (
              $1,
              $2,
              $3,
              $4,
              $5,
              $6,
              $7,
              $8,
              $9,
              $10
            )
            on conflict(component_uuid, metric_uuid) do update set
              value = excluded.value,
              variation = excluded.variation,
              text_value = excluded.text_value,
              measure_data  = excluded.measure_data,
              updated_at = excluded.updated_at
            where
              live_measures.value is distinct from excluded.value or
              live_measures.variation is distinct from excluded.variation or
              live_measures.text_value is distinct from excluded.text_value or
              live_measures.measure_data is distinct from excluded.measure_data
LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
FATAL:  the database system is in recovery mode
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2021-06-11 10:51:55 UTC
FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
FATAL:  the database system is in recovery mode
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 6/99D3D638
LOG:  invalid record length at 6/9ACC33D0
LOG:  redo done at 6/9ACC33A8
LOG:  last completed transaction was at log time 2021-06-11 10:55:19.899648+00
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started
LOG:  incomplete startup packet

Yep and it looks it was hard stopped (kill -9 as in SIGKILL).

so something is killing your database and i assume it’s docker trying to enforce the limits.

That is plausible. I’m gonna try to increase the limits after the busiest workhours, will respond when it’s done.

Additional memory fixed the issue. Thank you.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.