V 9.9.3.79811 Tasks runs after 28 hours into time out with error

SonarQube Server / Community Build
1

Database

|Database|PostgreSQL|
|Database Version|13.12|
|Driver|PostgreSQL JDBC Driver|
|Driver Version|42.5.1|

Last succeeded job

Status: SUCCESS
Project: generator
ID: AY48o3me3jKvchfZpsKV
Submitter Anonymous
Submitted: March 14, 2024 12:05:30 PM
Started: 12:05:31 PM
Finished: 12:06:05 PM
Duration: 34s

Still Running Task

States: Running
Project: generator
ID: AZBuwIMFDic8adWd1FcY
Submitter : Anonymous
Submitted: July 1, 2024 4:43:49 PM
Started: (+1d)9:27:05 PM
Duration 20h 34min

Failed Task

States : FAILED
Project: generator
ID : AZBuRMeIDic8adWd1FcX
Submitter : Anonymous
Submitted :July 1, 2024 2:28:40 PM
Started: 5:03:42 PM
Finished: (+1d)9:27:05 PM
Duration: 28h 23min

Error Details

org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
### The error may exist in org.sonar.db.issue.IssueMapper
### The error may involve org.sonar.db.issue.IssueMapper.selectByKeysIfNotUpdatedAt-Inline
### The error occurred while setting parameters
### SQL: select             i.kee as kee,     i.rule_uuid as ruleUuid,     i.severity as severity,     i.manual_severity as manualSeverity,     i.message as message,     i.message_formattings as messageFormattings,     i.line as line,     i.locations as locations,     i.gap as gap,     i.effort as effort,     i.status as status,     i.resolution as resolution,     i.checksum as checksum,     i.assignee as assigneeUuid,     i.author_login as authorLogin,     i.tags as tagsString,     i.rule_description_context_key as ruleDescriptionContextKey,     i.issue_creation_date as issueCreationTime,     i.issue_update_date as issueUpdateTime,     i.issue_close_date as issueCloseTime,     i.created_at as createdAt,     i.updated_at as updatedAt,     r.is_external as "isExternal",     r.plugin_rule_key as ruleKey,     r.plugin_name as ruleRepo,     r.language as language,     r.security_standards as securityStandards,     p.kee as componentKey,     i.component_uuid as componentUuid,     p.module_uuid_path as moduleUuidPath,     p.path as filePath,     root.kee as projectKey,     i.project_uuid as projectUuid,     i.issue_type as type,     i.quick_fix_available as quickFixAvailable,           n.uuid is not null as isNewCodeReferenceIssue             from issues i     inner join rules r on r.uuid=i.rule_uuid     inner join components p on p.uuid=i.component_uuid     inner join components root on root.uuid=i.project_uuid     left join new_code_reference_issues n on i.kee = n.issue_key     where       i.kee in          (             ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          ,            ?          )        and i.updated_at <> ?
### Cause: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:153)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:145)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
	at org.sonar.db.DbSessionImpl.selectList(DbSessionImpl.java:87)

Any idea why the tasks failed?

2

Hey there.

Do you see any more information in your ce.log?

3

Hello Colin,

the last run ended with a more detailed error

Thx in advance
Wilfried

org.apache.ibatis.exceptions.PersistenceException:
### Error querying database.  Cause: org.apache.ibatis.executor.BatchExecutorException: org.sonar.db.issue.IssueMapper.updateIfBeforeSelectedDate (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: Batch entry 0 update issues set
    severity='CRITICAL',
    manual_severity='FALSE',
    message='Rename this constant name to match the regular expression ''^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$''.',
    message_formattings=NULL,
    line=7,
    locations=?,
    gap=NULL,
    effort=2,
    status='OPEN',
    resolution=NULL,
    checksum='927fcb4136cbc66292b0685ad18a0672',
    assignee=NULL,
    author_login=NULL,
    tags='convention',
    rule_description_context_key=NULL,
    component_uuid='AW6yhR15szWOnvDWVWhG',
    project_uuid='AWwaQdTDL_IbnQq9C7ej',
    issue_creation_date=1563808552000,
    issue_update_date=1574953000000,
    issue_close_date=NULL,
    updated_at=1720033650193,
    issue_type=1
    where kee = 'AWwaQfN10T5hvLrkQiBc' and updated_at <= 1720033625949 was aborted: An I/O error occurred while sending to the backend.  Call getNextException to see other errors in the batch.
### The error may exist in org.sonar.db.issue.IssueMapper
### The error may involve org.sonar.db.issue.IssueMapper.selectByKeysIfNotUpdatedAt
### The error occurred while executing a query
### Cause: org.apache.ibatis.executor.BatchExecutorException: org.sonar.db.issue.IssueMapper.updateIfBeforeSelectedDate (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: Batch entry 0 update issues set
    severity='CRITICAL',
    manual_severity='FALSE',
    message='Rename this constant name to match the regular expression ''^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$''.',
    message_formattings=NULL,
    line=7,
    locations=?,
    gap=NULL,
    effort=2,
    status='OPEN',
    resolution=NULL,
    checksum='927fcb4136cbc66292b0685ad18a0672',
    assignee=NULL,
    author_login=NULL,
    tags='convention',
    rule_description_context_key=NULL,
    component_uuid='AW6yhR15szWOnvDWVWhG',
    project_uuid='AWwaQdTDL_IbnQq9C7ej',
    issue_creation_date=1563808552000,
    issue_update_date=1574953000000,
    issue_close_date=NULL,
    updated_at=1720033650193,
    issue_type=1
    where kee = 'AWwaQfN10T5hvLrkQiBc' and updated_at <= 1720033625949 was aborted: An I/O error occurred while sending to the backend.  Call getNextException to see other errors in the batch.
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:153)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:145)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:140)
	at org.sonar.db.DbSessionImpl.selectList(DbSessionImpl.java:87)
	at org.sonar.db.BatchSession.selectList(BatchSession.java:82)
	at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
	at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
	at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:145)
	at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
	at jdk.proxy2/jdk.proxy2.$Proxy68.selectByKeysIfNotUpdatedAt(Unknown Source)
	at org.sonar.ce.task.projectanalysis.step.PersistIssuesStep.persistUpdatedIssues(PersistIssuesStep.java:159)
	at org.sonar.ce.task.projectanalysis.step.PersistIssuesStep.execute(PersistIssuesStep.java:117)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.ibatis.executor.BatchExecutorException: org.sonar.db.issue.IssueMapper.updateIfBeforeSelectedDate (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: Batch entry 0 update issues set
    severity='CRITICAL',
    manual_severity='FALSE',
    message='Rename this constant name to match the regular expression ''^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$''.',
    message_formattings=NULL,
    line=7,
    locations=?,
    gap=NULL,
    effort=2,
    status='OPEN',
    resolution=NULL,
    checksum='927fcb4136cbc66292b0685ad18a0672',
    assignee=NULL,
    author_login=NULL,
    tags='convention',
    rule_description_context_key=NULL,
    component_uuid='AW6yhR15szWOnvDWVWhG',
    project_uuid='AWwaQdTDL_IbnQq9C7ej',
    issue_creation_date=1563808552000,
    issue_update_date=1574953000000,
    issue_close_date=NULL,
    updated_at=1720033650193,
    issue_type=1
    where kee = 'AWwaQfN10T5hvLrkQiBc' and updated_at <= 1720033625949 was aborted: An I/O error occurred while sending to the backend.  Call getNextException to see other errors in the batch.
	at org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:149)
	at org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:129)
	at org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:122)
	at org.apache.ibatis.executor.BatchExecutor.doQuery(BatchExecutor.java:86)
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:325)
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:156)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:109)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:89)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:151)
	... 30 more
Caused by: java.sql.BatchUpdateException: Batch entry 0 update issues set
    severity='CRITICAL',
    manual_severity='FALSE',
    message='Rename this constant name to match the regular expression ''^[A-Z][A-Z0-9]*(_[A-Z0-9]+)*$''.',
    message_formattings=NULL,
    line=7,
    locations=?,
    gap=NULL,
    effort=2,
    status='OPEN',
    resolution=NULL,
    checksum='927fcb4136cbc66292b0685ad18a0672',
    assignee=NULL,
    author_login=NULL,
    tags='convention',
    rule_description_context_key=NULL,
    component_uuid='AW6yhR15szWOnvDWVWhG',
    project_uuid='AWwaQdTDL_IbnQq9C7ej',
    issue_creation_date=1563808552000,
    issue_update_date=1574953000000,
    issue_close_date=NULL,
    updated_at=1720033650193,
    issue_type=1
    where kee = 'AWwaQfN10T5hvLrkQiBc' and updated_at <= 1720033625949 was aborted: An I/O error occurred while sending to the backend.  Call getNextException to see other errors in the batch.
	at org.postgresql.jdbc.BatchResultHandler.handleError(BatchResultHandler.java:165)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:565)
	at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:893)
	at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:916)
	at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1684)
	at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
	at org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:123)
	... 38 more
Caused by: org.postgresql.util.PSQLException: An I/O error occurred while sending to the backend.
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:566)
	... 44 more
Caused by: java.io.EOFException
	at org.postgresql.core.PGStream.receiveChar(PGStream.java:455)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2120)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:560)
	... 44 more
4

We’ve seen some reports of this An I/O error occurred while sending to the backend. error when the database is under heavy load (and has run out of memory, for example). Can you take a peek at the resource utilization of your database?

I’d also like to know a few more things about your SonarQube setup:

  • Which edition are you using? Is more than 1 CE Worker enabled?
  • How have you configured memory settings for the Compute Engine? You can find the details in your global Administration > System > Compute Engine. It would be great if you could share all the figures here.

image
image1815×989 44.8 KB

5

Informations about our System and Compute Engine

System

Edition `Community`

Compute Engine

#### Compute Engine Tasks

|Pending|`0`|
| --- | --- |
|Longest Time Pending (ms)|`0`|
|In Progress|`0`|
|Processed With Error|`1`|
|Processed With Success|`128`|
|Processing Time (ms)|`90835537`|
|Worker Count|`1`|
|Max Worker Count|`1`|
|Workers Paused|No|

#### Compute Engine JVM State

|Max Memory (MB)|`512`|
| --- | --- |
|Free Memory (MB)|`116`|
|Heap Committed (MB)|`261`|
|Heap Init (MB)|`128`|
|Heap Max (MB)|`512`|
|Heap Used (MB)|`143`|
|Non Heap Committed (MB)|`124`|
|Non Heap Init (MB)|`7`|
|Non Heap Used (MB)|`122`|
|System Load Average|`156.5% (last minute)`|
|Threads|`25`|

#### Compute Engine Database Connection

|Pool Total Connections|`10`|
| --- | --- |
|Pool Active Connections|`0`|
|Pool Idle Connections|`10`|
|Pool Max Connections|`60`|
|Pool Min Idle Connections|`10`|
|Pool Max Wait (ms)|`8000`|
|Pool Max Lifetime (ms)|`1800000`|

#### Compute Engine Logging

|Logs Level|`INFO`|
| --- | --- |
|Logs Dir|`/opt/sonarqube/logs`|

#### Compute Engine JVM Properties

|com.redhat.fips|`false`|
| --- | --- |
|com.zaxxer.hikari.pool_number|`1`|
|file.encoding|`UTF-8`|
|file.separator|`/`|
|http.nonProxyHosts|`localhost|127.*|[::1]`|
|java.awt.headless|`true`|
|java.class.path|`./lib/sonar-application-9.9.3.79811.jar:/opt/sonarqube/lib/jdbc/postgresql/postgresql-42.5.1.jar`|
|java.class.version|`61.0`|
|java.home|`/opt/java/openjdk`|
|java.io.tmpdir|`/opt/sonarqube/temp`|
|java.library.path|`/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib`|
|java.runtime.name|`OpenJDK Runtime Environment`|
|java.runtime.version|`17.0.10+7`|
|java.specification.name|`Java Platform API Specification`|
|java.specification.vendor|`Oracle Corporation`|
|java.specification.version|`17`|
|java.vendor|`Eclipse Adoptium`|
|java.vendor.url|`https://adoptium.net/`|
|java.vendor.url.bug|`https://github.com/adoptium/adoptium-support/issues`|
|java.vendor.version|`Temurin-17.0.10+7`|
|java.version|`17.0.10`|
|java.version.date|`2024-01-16`|
|java.vm.compressedOopsMode|`32-bit`|
|java.vm.info|`mixed mode, sharing`|
|java.vm.name|`OpenJDK 64-Bit Server VM`|
|java.vm.specification.name|`Java Virtual Machine Specification`|
|java.vm.specification.vendor|`Oracle Corporation`|
|java.vm.specification.version|`17`|
|java.vm.vendor|`Eclipse Adoptium`|
|java.vm.version|`17.0.10+7`|
|jdk.debug|`release`|
|line.separator|``|
|native.encoding|`UTF-8`|
|os.arch|`amd64`|
|os.name|`Linux`|
|os.version|`5.10.209-198.858.amzn2.x86_64`|
|path.separator|`:`|
|sun.arch.data.model|`64`|
|sun.boot.library.path|`/opt/java/openjdk/lib`|
|sun.cpu.endian|`little`|
|sun.io.unicode.encoding|`UnicodeLittle`|
|sun.java.command|`org.sonar.ce.app.CeServer /opt/sonarqube/temp/sq-process16237009785228250342properties`|
|sun.java.launcher|`SUN_STANDARD`|
|sun.jnu.encoding|`UTF-8`|
|sun.management.compiler|`HotSpot 64-Bit Tiered Compilers`|
|user.country|`US`|
|user.dir|`/opt/sonarqube`|
|user.home|`/home/sonarqube`|
|user.language|`en`|
|user.name|`sonarqube`|
|user.timezone|`Etc/UTC`|`
6

I suggest bumping up the maximum heap size of your Compute Engine. If you have a particularly large project, this can be necessary.

In your conf/sonar.properties file, you can adjust sonar.ce.javaOpts. I suggest a value of 2GB, assuming you have sufficient memory on the server.

sonar.ce.javaOpts=-Xmx2G -Xms128m -XX:+HeapDumpOnOutOfMemoryError

You’ll need to reboot the service for the changes to take affect (./sonar.sh restart)

And I still recommend these points:

We’ve seen some reports of this An I/O error occurred while sending to the backend. error when the database is under heavy load (and has run out of memory, for example). Can you take a peek at the resource utilization of your database?