Error upgrading SonarQube Developer from 9.7 to 9.8 getting JDBC settings error

SonarQube
1

Aim

I am running SonarQube 9.7 Developer using a Docker container hosted in an Azure WebApp with an Azure SQL DB (approx 3Gb in size),

I have tried to update to 9.8, but get an error after the Db migration Database connection cannot be established. Please check database status and JDBC settings..

Steps undertaken

  1. Updated the Docker Image from 9.7 to 9.8
  2. Started the container
  3. Get the expected site in maintenance mode message, loaded the /setup page and triggered the DB schema update
  4. I see load on the SQL instance for around 5 minutes (about what I expect from from previous updates), but when the load drops, the frontend shows the error message Database connection cannot be established. Please check database status and JDBC settings..

What I have tried

I have tried restarts of the container, assuming it had got confused as to where it was in the upgrade, but still get the same message.

I thought it might be related to the performance of the SQL instance, so I swapped the Azure SQL SKU from S1 (20DTU) to the S3 (100DTU) SKU, restored the DB and tried the upgrade again. I got the same error, but the migrate phase was quicker (as expected given the SQL instance was 5x the performanant)

What is the Next Step

So I have revert to 9.7 of now.

Note: I had to delete the contents of the Elastic Search data folder /es7/nodes/0 as I got the error on startup of 9.7 ‘Cannot downgrade a node from version [7.17.6] to version [7.17.5]’ as the ES Node data had been updated by the SonarQube 9.8 update. Deleting this node data caused the ES node cache to be regenerated

I do still have the failed upgraded DB from the failed SonarQube 9.8 upgrade, is there any query run to see what the issue is?

Or should I just abort this upgrade and wait for 9.9 LTS upgrade next month?

Or any other suggestions?

2

Hi,

What do your server logs say?

 
Ann

3

The server logs did not show an error, that is what I thought was strange

2023-01-13T16:50:28.178Z INFO - Started multi-container app
2023-01-13T16:50:28.183Z INFO - Initiating warmup request to container bmsonarqubeprodwebsite_sonarqube_1_0e939355 for site bmsonarqubeprodwebsite
2023-01-13T16:50:28.190Z INFO - Container bmsonarqubeprodwebsite_sonarqube_1_0e939355 for site bmsonarqubeprodwebsite initialized successfully and is ready to serve requests.
2023-01-13T17:35:19.503Z INFO - Starting multi-container app…
2023-01-13T17:35:19.920Z INFO - Pulling image: sonarqube:9.8-developer
2023-01-13T17:35:20.787Z INFO - 9.8-developer Pulling from library/sonarqube
2023-01-13T17:35:21.070Z INFO - Digest: sha256:a73ecba149df1ebb84f0dc483657ab6cdcf8ed5c6cb72964630fb12b043ccc9c
2023-01-13T17:35:21.072Z INFO - Status: Image is up to date for sonarqube:9.8-developer
2023-01-13T17:35:21.076Z INFO - Pull Image successful, Time taken: 0 Minutes and 1 Seconds
2023-01-13T17:35:21.109Z INFO - Starting container for site
2023-01-13T17:35:21.111Z INFO - docker run -d -p 9109:9000 --name bmsonarqubeprodwebsite_sonarqube_0_d151d2eb -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=sonarqubewebsite -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=onarqubewebsite.azurewebsites.net -e WEBSITE_INSTANCE_ID=df2788fb3e5f96d53e88638b51f78b399174bc066b03447dde3bac5ebbfa91f6 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False sonarqube:9.8-developer -Dsonar.search.javaAdditionalOpts=-Dnode.store.allow_mmap=false

When ever I have had issues before, I have seen an issue in this log, but not this time.

Looking at the server load, it looks as if the Docker contain just does not attempt to connect to the SQL instance after the schema has been upgraded.

I was wondering if there is any check I can make on the SQL tables to see how far the migration got?

4

I have done another complete end to end test,

  1. I restored my DB to the 9.7 version and used a 9.7 container images to make sure all is good before the upgrade
  2. I update the container images to 9.8
  3. I see the /maintainance message, got to the /setup page and trigger the DB update
  4. I get the error

image
image983×550 15.5 KB

  1. But the logs just show i.e no error message

2023-01-18T15:04:08.841Z INFO - Started multi-container app
2023-01-18T15:04:08.847Z INFO - Initiating warmup request to container bmsonarqubewebsite_sonarqube_0_a576289c for site bmsonarqubewebsite
2023-01-18T15:04:08.855Z INFO - Container bmsonarqubewebsite_sonarqube_0_a576289c for site bmsonarqubewebsite initialized successfully and is ready to serve requests.
2023-01-18T15:16:57.133Z INFO - Starting multi-container app…
2023-01-18T15:16:58.579Z INFO - Pulling image: sonarqube:9.8-developer
2023-01-18T15:16:59.488Z INFO - 9.8-developer Pulling from library/sonarqube
2023-01-18T15:16:59.812Z INFO - Digest: sha256:a73ecba149df1ebb84f0dc483657ab6cdcf8ed5c6cb72964630fb12b043ccc9c
2023-01-18T15:16:59.818Z INFO - Status: Image is up to date for sonarqube:9.8-developer
2023-01-18T15:16:59.826Z INFO - Pull Image successful, Time taken: 0 Minutes and 1 Seconds
2023-01-18T15:16:59.846Z INFO - Starting container for site
2023-01-18T15:16:59.847Z INFO - docker run -d -p 9326:9000 --name bmsonarqubewebsite_sonarqube_1_a61a5db2 -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=bmsonarqubewebsite -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=bmsonarqubewebsite.azurewebsites.net -e WEBSITE_INSTANCE_ID=18783e8b40071823193e08c99253daca8096048ae20cdbdb8e9016bc62023bf6 -e WEBSITE_USE_DIAGNOSTIC_SERVER=False sonarqube:9.8-developer -Dsonar.search.javaAdditionalOpts=-Dnode.store.allow_mmap=false

2023-01-18T15:16:59.848Z INFO - Logging is not enabled for this container.
Please use Enable diagnostics logging - Azure App Service | Microsoft Learn to enable logging to see container logs here.
2023-01-18T15:19:28.745Z INFO - Started multi-container app
2023-01-18T15:19:28.752Z INFO - Initiating warmup request to container bmsonarqubewebsite_sonarqube_1_a61a5db2 for site bmsonarqubewebsite
2023-01-18T15:19:28.763Z INFO - Container bmsonarqubewebsite_sonarqube_1_a61a5db2 for site bmsonarqubewebsite initialized successfully and is ready to serve requests.

As all the settings are read from Azure Web Apps application settings, they have not changed, so I am at the loss as to what the issue is.

If I revert my container to 9.7 I can restart the instance.

5

I reran the update again and this time watched the Azure LogStream from and actually saw the error message (for some reason I did not see this error when logging back at the logs as opposed to watching ‘live’)

2023-01-19T09:21:37.312570884Z: [INFO] 2023.01.19 09:21:37 INFO web[o.s.s.p.d.m.DatabaseMigrationImpl] Starting DB Migration and container restart
…
2023-01-19T09:24:12.599097580Z: [INFO] 2023.01.19 09:24:12 INFO web[DbMigrations] #6703 ‘Drop project measure variation column’…
2023-01-19T09:24:12.635942690Z: [INFO] 2023.01.19 09:24:12 ERROR web[DbMigrations] #6703 ‘Drop project measure variation column’: failure | time=40ms
2023-01-19T09:24:12.636717705Z: [INFO] 2023.01.19 09:24:12 ERROR web[DbMigrations] Executed DB migrations: failure | time=155226ms
2023-01-19T09:24:12.645227469Z: [INFO] 2023.01.19 09:24:12 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=155334ms
2023-01-19T09:24:12.653407627Z: [INFO] 2023.01.19 09:24:12 ERROR web[o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
2023-01-19T09:24:12.654044139Z: [INFO] org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #6703 ‘Drop project measure variation column’ failed

No I see the error message, I see this has already been reported Possible db migration error when updating to Sonarqube Enterprise 9.8.0 with MSSQL - SonarQube - Sonar Community (sonarsource.com), and it is supposedly fixed in 9.9

1 Like
6

Hi,

Did the fix in that other thread work for you?

 
Ann

7

I have decided to wait until 9.9 as it is supposedly fixed in that release

1 Like