Upgrade 9.9.0 Fail

Make sure to tell us:

  • What version are you upgrading from?
    8.9.9 lts (enterprise edition) to 9.9.0 lts (enterprise edition)

  • System information (Operating system, Java version, Database provider/version)
    Offical docker image, sqlserver

  • What’s the issue you’re facing?
    I have run several upgrade, and show different errors:

2023.03.22 12:41:39 ERROR web[DbMigrations] Executed DB migrations: failure | time=56220ms
2023.03.22 12:41:39 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] Container restart failed | time=56253ms
2023.03.22 12:41:39 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] Container restart failed
java.lang.IllegalStateException: Unable to load component class org.sonar.server.platform.db.migration.version.v95.CreateIndexForRuleDescSections

2023.03.22 12:26:22 ERROR web[DbMigrations] Executed DB migrations: failure | time=55824ms
2023.03.22 12:26:22 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] Container restart failed | time=55848ms
2023.03.22 12:26:22 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] Container restart failed
java.lang.IllegalStateException: Unable to load component class org.sonar.server.platform.db.migration.version.v97.DropIndexForComponentsProjectUuid

2023.03.22 12:55:41 ERROR web[DbMigrations] Executed DB migrations: failure | time=50788ms
2023.03.22 12:55:41 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] Container restart failed | time=50822ms
2023.03.22 12:55:41 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] Container restart failed
java.lang.IllegalStateException: Unable to load component class org.sonar.server.platform.db.migration.version.v91.RemoveSVNPropertiesData

Hi,

Welcome to the community!

We’re going to need the failure details, i.e. what lead up to those error lines.

 
Ann

I just start the 9.9.0 docker image with the same db connection of 8.9.9, and trigger the link “https://sonar-ip/setup” to do the db upgrade. but fail.

more error message:

2023.03.22 12:55:24 INFO  web[][DbMigrations] #6016 'Create unique index for 'portfolio_projects''...
2023.03.22 12:55:24 INFO  web[][DbMigrations] #6016 'Create unique index for 'portfolio_projects'': success | time=20ms
2023.03.22 12:55:24 INFO  web[][DbMigrations] #6017 'Migrate portfolios to new tables'...
2023.03.22 12:55:24 INFO  web[][o.s.s.p.d.m.s.MassUpdate] 1 rows processed (0 items/sec)
2023.03.22 12:55:24 INFO  web[][DbMigrations] #6017 'Migrate portfolios to new tables': success | time=388ms
2023.03.22 12:55:24 INFO  web[][DbMigrations] #6018 'Create index for 'issue_changes' on 'issue_key' and 'change_type''...
2023.03.22 12:55:40 INFO  app[][o.s.a.SchedulerImpl] Stopping SonarQube
2023.03.22 12:55:40 INFO  app[][o.s.a.SchedulerImpl] Sonarqube has been requested to stop
2023.03.22 12:55:40 INFO  app[][o.s.a.SchedulerImpl] Stopping [Compute Engine] process...
2023.03.22 12:55:40 INFO  app[][o.s.a.SchedulerImpl] Stopping [Web Server] process...
2023.03.22 12:55:40 INFO  web[][o.s.p.ProcessEntryPoint] Gracefully stopping process
2023.03.22 12:55:41 INFO  web[][DbMigrations] #6018 'Create index for 'issue_changes' on 'issue_key' and 'change_type'': success | time=16669ms
2023.03.22 12:55:41 ERROR web[][DbMigrations] Executed DB migrations: failure | time=50788ms
2023.03.22 12:55:41 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] Container restart failed | time=50822ms
2023.03.22 12:55:41 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] Container restart failed
java.lang.IllegalStateException: Unable to load component class org.sonar.server.platform.db.migration.version.v91.RemoveSVNPropertiesData
        at org.sonar.core.platform.SpringComponentContainer.getComponentByType(SpringComponentContainer.java:139)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:64)
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
        at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
        at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:66)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
        at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalStateException: org.springframework.context.annotation.AnnotationConfigApplicationContext@382fdcb9 has been closed already
        at org.springframework.context.support.AbstractApplicationContext.assertBeanFactoryActive(AbstractApplicationContext.java:1138)
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1171)
        at org.sonar.core.platform.SpringComponentContainer.getComponentByType(SpringComponentContainer.java:137)
        ... 9 common frames omitted
2023.03.22 12:55:41 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown initiated...
2023.03.22 12:55:41 INFO  web[][c.z.h.HikariDataSource] HikariPool-1 - Shutdown completed.
2023.03.22 12:55:41 INFO  web[][o.s.s.app.WebServer] Web Server stopped
2023.03.22 12:55:42 INFO  app[][o.s.a.SchedulerImpl] Process[Web Server] is stopped
2023.03.22 12:55:42 INFO  app[][o.s.a.SchedulerImpl] Stopping [ElasticSearch] process...
2023.03.22 12:55:42 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 143
2023.03.22 12:55:42 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2023.03.22 12:55:42 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

Hi,

Thanks for a fuller log excerpt. I’m flagging this for team attention.

 
Ann

Hi Ann,

Is that memory issue?
Our sonarqube service has over 40G data.
And we start docker image with 6CPU/28G Memory (We use K8S, and the nodes have limit).

I’ve noticed those lines in the logs. Is it possible that k8s is stopping the pod because it is not ready? How do you check the liveness/readiness of the pod?

Thanks, Wojciech.

I am using the standard helm chart.
I think it has enough time to finish all the upgrade steps, and once we can visit “https://sonar-it/setup”, that mean the sonarqube is up.

      livenessProbe:
        httpGet:
          path: {{ .Values.livenessProbe.sonarWebContext }}api/system/status
          port: http
        initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
        periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
      readinessProbe:
        httpGet:
          path: {{ .Values.readinessProbe.sonarWebContext }}api/system/status
          port: http
        initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
        periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
        failureThreshold: {{ .Values.readinessProbe.failureThreshold }}

readinessProbe:
initialDelaySeconds: 60
periodSeconds: 30
failureThreshold: 6

livenessProbe:
initialDelaySeconds: 60
periodSeconds: 30
failureThreshold: 6

Hi @hudson,

Could you share a bigger portion of the logs to try to find a cause before Stopping SonarQube appears in the logs?

On top of that, could you share the k8s event log to see if perhaps k8s is killing the pod for some other reason?

We try several upgrades, and finally, it works.
But something are showing not correct.


Hi,

What plugins do you have installed? Every time we’ve seen this, it was caused by a plugin.

 
HTH,
Ann

Thanks for your reply.
We only install one plugin named “sonar-cnes-report”.

Hi,

Have you tried removing it?

BTW I believe CNES isn’t compatible with 9.9.

 
Ann

Thanks a lot. Remove the plugin and fix the issue.

1 Like

A post was merged into an existing topic: Upgraded SonarQube stopping automatically