SonarQube 9.6 / 9.6.1 - Oracle ORA-01000 error on startup

  • SonarQube version: 9.6, 9.6.1
  • Environment: Linux, OpenJDK 11, Oracle 19C database, ojdbc8 driver
  • Plugins: Crowd 2.2 (but have tested without plugin and error still occurs)

Upgrading from 9.2.4 to 9.6/9.6.1, startup after upgrade throws the following error

2022.09.05 22:13:13 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [rules/rule/activeRule]...
2022.09.05 22:13:13 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [rules/rule/activeRule] done | time=621ms
2022.09.05 22:13:13 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [rules/rule]...
2022.09.05 22:13:19 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [rules/rule] done | time=5282ms
2022.09.05 22:13:19 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [users/user]...
2022.09.05 22:13:19 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [users/user] done | time=409ms
2022.09.05 22:13:19 INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [projectmeasures/auth/projectmeasure]...
2022.09.05 22:13:24 ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
java.lang.IllegalStateException: Fail to execute request to select measures of project AYLatv66gsPavjZzbDot
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectMeasures(ProjectMeasuresIndexerIterator.java:219)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.doNext(ProjectMeasuresIndexerIterator.java:189)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.doNext(ProjectMeasuresIndexerIterator.java:52)
	at org.sonar.core.util.CloseableIterator.bufferNext(CloseableIterator.java:96)
	at org.sonar.core.util.CloseableIterator.hasNext(CloseableIterator.java:87)
	at org.sonar.server.measure.index.ProjectMeasuresIndexer.doIndex(ProjectMeasuresIndexer.java:158)
	at org.sonar.server.measure.index.ProjectMeasuresIndexer.indexOnStartup(ProjectMeasuresIndexer.java:73)
	at org.sonar.server.es.IndexerStartupTask.synchronousIndexing(IndexerStartupTask.java:83)
	at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:68)
	at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
	at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:53)
	at java.base/java.util.Optional.ifPresent(Optional.java:183)
	at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:128)
	at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:45)
	at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:124)
	at org.sonar.server.platform.PlatformImpl.executeStartupTasks(PlatformImpl.java:183)
	at org.sonar.server.platform.PlatformImpl$1.lambda$doRun$1(PlatformImpl.java:107)
	at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.runIfNotAborted(PlatformImpl.java:344)
	at org.sonar.server.platform.PlatformImpl$1.doRun(PlatformImpl.java:107)
	at org.sonar.server.platform.PlatformImpl$AutoStarterRunnable.run(PlatformImpl.java:328)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Fail to execute request to select the project biggest branch
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectProjectBiggestBranch(ProjectMeasuresIndexerIterator.java:310)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectMeasures(ProjectMeasuresIndexerIterator.java:208)
	... 21 common frames omitted
Caused by: java.sql.SQLException: ORA-01000: maximum open cursors exceeded

	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:494)
	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:446)
	at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1054)
	at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:623)
	at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
	at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
	at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
	at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:59)
	at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:747)
	at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:904)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1082)
	at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
	at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1343)
	at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3822)
	at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1165)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
	at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectProjectBiggestBranch(ProjectMeasuresIndexerIterator.java:303)
	... 22 common frames omitted
Caused by: oracle.jdbc.OracleDatabaseException: ORA-01000: maximum open cursors exceeded

	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:498)

We do have a large number of projects (2000+) but this was fine prior to upgrade.

Thanks

This is the first report we’ve had of any issues with cursors after upgrading to 9.6. Since this is purely an oracle error I would suggest you reach out to your DBAs to understand the number of open cursors available to your Oracle database and consider raising them to see if it alleviates the issue.

Thanks Colin

Are there any recommendations re the expected number of cursors the new version of SonarQube requires?

Ours DBAs were the first port of call when we initially encountered the issue and we verified that the Development DB (where we’re testing this upgrade) is aligned with our Live DB at a setting of 2000 max open cursors, which has been sufficient for a database housing multiple DevOps products.

Hello
We’re experiencing the same issue, and during upgrade to Sonarqube Enterprise v9.6.1 from Enterprise v9.5.0, the cursor are getting exhausted and causing the application to fail to startup.

Caused by: java.lang.IllegalStateException: Fail to execute request to select the project biggest branch
        at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectProjectBiggestBranch(ProjectMeasuresIndexerIterator.java:310)
        at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectMeasures(ProjectMeasuresIndexerIterator.java:208)
        ... 21 common frames omitted
Caused by: java.sql.SQLException: ORA-01000: maximum open cursors exceeded

        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:509)
        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:461)
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1104)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:553)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:655)
        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:270)
        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:91)
        at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:807)
        at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:983)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1168)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3666)
        at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1426)
        at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3713)
        at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1167)
        at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
        at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:122)
        at org.sonar.db.measure.ProjectMeasuresIndexerIterator.selectProjectBiggestBranch(ProjectMeasuresIndexerIterator.java:303)
        ... 22 common frames omitted
Caused by: oracle.jdbc.OracleDatabaseException: ORA-01000: maximum open cursors exceeded
        at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:513)

Hey there.

I’ve flagged this thread for some expert attention.

1 Like

Thanks Colin,

We did some testing on the number of open cursors we could configure to proceed with the upgrade, and managed to complete the upgrade (in Development only).

Factoring in other activity on the database, it looks like versions 9.6/9.6.1 needed 3x the number of cursors that our previous upgrade did. It does just appear to be a requirement at post-upgrade startup as a stop/start after that doesn’t kick off the same indexing.

However, without knowing more about why there’s a sudden jump in cursors, and whether similar might occur elsewhere, we’re pausing on a Production upgrade.

Hello
Just to highlight, we also tried increasing the cursor count value greater than 3x than we had in previous upgrade, but still it didn’t work.
We didn’t experience the same in test environment, but in production. Hence we rolled back the change.
Hence, requesting to please continue checking the cause and possible solution for the same.

Thanks for reporting the error.
We’ll fix it soon: [SONAR-17326] - Jira

2 Likes

The cause of the issue will be fixed in the next release.
As a workaround, I suggest you try to increase it further to higher value.

Hi
Thanks for the update. Seems [SONAR-17326|[SONAR-17326] - Jira] has been resolved. By when could we expect the enterprise fix version available on docker-hub?
Will plan for the upgrade accordingly.

Hi,

You can expect the SonarQube 9.7 Docker version available around mid-October (~Oct 17 or so). Make sure to subscribe for updates in the Releases category:

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.