Issue Summary
After upgrading from SonarQube 9.9.4-enterprise to v2025.1.1 (104738) Enterprise, we are experiencing issues with background tasks for analysis (Compute Engine).
Environment Details
- Platform: OpenShift
- Database: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.24.0.0.0
- OJDBC Driver: ojdbc11-23.7.0.25.01.jar (previously ojdbc11 23.3.0.23.09)
- Resources Allocated:
- CPU: 4 cores
- RAM: 8GB
Observations
The database migration during the upgrade was successful. However, errors began appearing when rule registration was initiated:
2025.03.27 19:51:32 INFO web[][o.s.s.r.r.RulesRegistrant] Register rules
We observed that a specific query is responsible for 40% of the database load. The query results in approximately 36,000 rows (~50 MB of data).
Error Details
During background task execution, we encountered the following error:
2025.04.01 04:43:14 ERROR ce[25a6a34e-e6cf-4f4a-b56d-08b61bdbf919][o.s.c.t.CeWorkerImpl] Failed to execute task 25a6a34e-e6cf-4f4a-b56d-08b61bdbf919
org.apache.ibatis.exceptions.PersistenceException:
### Error querying database. Cause: java.sql.SQLRecoverableException: ORA-17002: I/O error: Checksum fail
https://docs.oracle.com/error-help/db/ora-17002/
### The error may exist in org.sonar.db.rule.RuleMapper
### The error may involve org.sonar.db.rule.RuleMapper.selectAll
### The error occurred while handling results
### SQL: select rdi.rule_uuid as "rdi_ruleUuid", rdi.software_quality as "rdi_softwareQuality", rdi.severity as "rdi_severity", CASE WHEN rt.is_system_tag = 1 THEN rt.value ELSE NULL END as rt_systemTags, CASE WHEN rt.is_system_tag = 0 THEN rt.value ELSE NULL END as rt_tags, rt.is_system_tag as "rt_isSystemTag", rt.value as "rt_value", rds.content as "rds_content", rds.uuid as "rds_uuid", rds.kee as "rds_kee", rds.context_key as "rds_contextKey", rds.context_display_name as "rds_contextDisplayName", r.uuid as "r_uuid", r.plugin_rule_key as "ruleKey", r.plugin_name as "repositoryKey", r.description_format as "descriptionFormat", r.status, r.name, r.plugin_config_key as "configKey", r.priority as "severity", r.is_template as "isTemplate", r.is_external as "isExternal", r.is_ad_hoc as "isAdHoc", r.language as "language", r.template_uuid as "templateUuid", r.def_remediation_function as "defRemediationFunction", r.def_remediation_gap_mult as "defRemediationGapMultiplier", r.def_remediation_base_effort as "defRemediationBaseEffort", r.gap_description as "gapDescription", r.security_standards as "securityStandardsField", r.rule_type as "type", r.plugin_key as "pluginKey", r.scope, r.created_at as "createdAt", r.updated_at as "updatedAt", r.note_data as "noteData", r.note_user_uuid as "noteUserUuid", r.note_created_at as "noteCreatedAt", r.note_updated_at as "noteUpdatedAt", r.remediation_function as "remediationFunction", r.remediation_gap_mult as "remediationGapMultiplier", r.remediation_base_effort as "remediationBaseEffort", r.ad_hoc_name as "adHocName", r.ad_hoc_description as "adHocDescription", r.ad_hoc_severity as "adHocSeverity", r.ad_hoc_type as "adHocType", r.education_principles as "educationPrinciplesField", r.clean_code_attribute as "cleanCodeAttribute" from rules r left outer join rule_desc_sections rds on rds.rule_uuid = r.uuid left outer join rules_default_impacts rdi on rdi.rule_uuid = r.uuid left outer join rule_tags rt on rt.rule_uuid = r.uuid order by r.uuid
### Cause: java.sql.SQLRecoverableException: ORA-17002: I/O error: Checksum fail
https://docs.oracle.com/error-help/db/ora-17002/
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:156)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:147)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:142)
at org.apache.ibatis.binding.MapperMethod.executeForMany(MapperMethod.java:147)
at org.apache.ibatis.binding.MapperMethod.execute(MapperMethod.java:80)
at org.apache.ibatis.binding.MapperProxy$PlainMethodInvoker.invoke(MapperProxy.java:141)
at org.apache.ibatis.binding.MapperProxy.invoke(MapperProxy.java:86)
at jdk.proxy2/jdk.proxy2.$Proxy71.selectAll(Unknown Source)
at org.sonar.db.rule.RuleDao.selectAll(RuleDao.java:92)
at org.sonar.ce.task.projectanalysis.issue.RuleRepositoryImpl.loadRulesFromDb(RuleRepositoryImpl.java:140)
at org.sonar.ce.task.projectanalysis.issue.RuleRepositoryImpl.ensureInitialized(RuleRepositoryImpl.java:130)
at org.sonar.ce.task.projectanalysis.issue.RuleRepositoryImpl.findByKey(RuleRepositoryImpl.java:102)
at org.sonar.ce.task.projectanalysis.step.LoadQualityProfilesStep.execute(LoadQualityProfilesStep.java:60)
at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.sql.SQLRecoverableException: ORA-17002: I/O error: Checksum fail
https://docs.oracle.com/error-help/db/ora-17002/
at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1775)
at oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:4383)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchMoreRows(InsensitiveScrollableResultSet.java:951)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchNextRows(InsensitiveScrollableResultSet.java:858)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.absoluteInternal(InsensitiveScrollableResultSet.java:830)
at oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:529)
at com.zaxxer.hikari.pool.HikariProxyResultSet.next(HikariProxyResultSet.java)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForNestedResultMap(DefaultResultSetHandler.java:1015)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:335)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:310)
at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:202)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:66)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:80)
at org.apache.ibatis.executor.ReuseExecutor.doQuery(ReuseExecutor.java:62)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:336)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:158)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:110)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:90)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:154)
... 31 common frames omitted
Caused by: java.io.IOException: Checksum fail
at oracle.net.ano.CryptoNIONSDataChannel.decryptAndChecksum(CryptoNIONSDataChannel.java:315)
at oracle.net.ano.CryptoNIONSDataChannel.readDataFromSocketChannel(CryptoNIONSDataChannel.java:119)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForUnmarshall(T4CMAREngineNIO.java:932)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:466)
at oracle.jdbc.driver.DynamicByteArray.unmarshalCLR(DynamicByteArray.java:221)
at oracle.jdbc.driver.T4CNumberAccessor.unmarshalBytes(T4CNumberAccessor.java:188)
at oracle.jdbc.driver.T4CNumberAccessor.unmarshalOneRow(T4CNumberAccessor.java:170)
at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1691)
at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1406)
at oracle.jdbc.driver.T4C8Oall.readRXD(T4C8Oall.java:1118)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:879)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:237)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:524)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:298)
at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1770)
... 49 common frames omitted
The SQL query causing the issue:
SELECT rdi.rule_uuid AS "rdi_ruleUuid",
rdi.software_quality AS "rdi_softwareQuality",
rdi.severity AS "rdi_severity",
CASE WHEN rt.is_system_tag = 1 THEN rt.value ELSE NULL END AS rt_systemTags,
CASE WHEN rt.is_system_tag = 0 THEN rt.value ELSE NULL END AS rt_tags,
rt.is_system_tag AS "rt_isSystemTag",
rt.value AS "rt_value",
rds.content AS "rds_content",
rds.uuid AS "rds_uuid",
rds.kee AS "rds_kee",
rds.context_key AS "rds_contextKey",
rds.context_display_name AS "rds_contextDisplayName",
r.uuid AS "r_uuid",
r.plugin_rule_key AS "ruleKey",
r.plugin_name AS "repositoryKey",
r.description_format AS "descriptionFormat",
r.status,
r.name,
r.plugin_config_key AS "configKey",
r.priority AS "severity",
r.is_template AS "isTemplate",
r.is_external AS "isExternal",
r.is_ad_hoc AS "isAdHoc",
r.language AS "language",
r.template_uuid AS "templateUuid",
r.def_remediation_function AS "defRemediationFunction",
r.def_remediation_gap_mult AS "defRemediationGapMultiplier",
r.def_remediation_base_effort AS "defRemediationBaseEffort",
r.gap_description AS "gapDescription",
r.security_standards AS "securityStandardsField",
r.rule_type AS "type",
r.plugin_key AS "pluginKey",
r.scope,
r.created_at AS "createdAt",
r.updated_at AS "updatedAt",
r.note_data AS "noteData",
r.note_user_uuid AS "noteUserUuid",
r.note_created_at AS "noteCreatedAt",
r.note_updated_at AS "noteUpdatedAt",
r.remediation_function AS "remediationFunction",
r.remediation_gap_mult AS "remediationGapMultiplier",
r.remediation_base_effort AS "remediationBaseEffort",
r.ad_hoc_name AS "adHocName",
r.ad_hoc_description AS "adHocDescription",
r.ad_hoc_severity AS "adHocSeverity",
r.ad_hoc_type AS "adHocType",
r.education_principles AS "educationPrinciplesField",
r.clean_code_attribute AS "cleanCodeAttribute"
FROM rules r
LEFT OUTER JOIN rule_desc_sections rds
ON rds.rule_uuid = r.uuid
LEFT OUTER JOIN rules_default_impacts rdi
ON rdi.rule_uuid = r.uuid
LEFT OUTER JOIN rule_tags rt
ON rt.rule_uuid = r.uuid
ORDER BY r.uuid;
Additional Information
- This query is executed on every background task and 50% of our analysis fail on this query randomly
- The error ORA-17008: I/O error: Checksum fail is documented by Oracle: https://docs.oracle.com/error-help/db/ora-17008/
- We suspect this may be due to:
- Inefficient query execution causing high database load.
- Potential compatibility issues between SonarQube v2025.1.1 and Oracle 19c.
- OJDBC driver behavior under heavy load.
Request for Support
- Has anyone else experienced similar issues post-upgrade?
- Are there any optimizations or workarounds to mitigate this?
- Is this a known issue in SonarQube v2025.1.1?
Any insights or recommendations would be greatly appreciated!
Thank you for your support!