Docker upgrade from 8.4.1 to 8.6.1 or higher does not work

We initially first reported the issue here

But it looks like there is not much traction there, hopefully someone here has a workaround.
Thanks

PS: We also reported this issue with documentation.

Main Issue:

We’re currently running sonar developer version 8.4.1 with Postgres 12.6
If we follow the upgrade steps, sonarqube fails to start with the following error

sonarqube          | 2021.03.29 17:30:51 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
sonarqube          | 2021.03.29 17:30:51 ERROR app[][o.s.a.p.ManagedProcessHandler] Fail to launch process [es]
sonarqube          | org.sonar.process.MessageException: Property 'node.store.allow_mmapfs' is no longer supported. Use 'node.store.allow_mmap' instead.
sonarqube          | 2021.03.29 17:30:51 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
sonarqube          | 2021.03.29 17:30:51 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
sonarqube          | 2021.03.29 17:30:51 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
sonarqube exited with code 0

The issue seems to stem from Postgres

Our current docker-compose file is as follows

version: "3"

services:
###################
# SONARQUBE
###################
  sonarqube:
    container_name: sonarqube
    build:
      context: ./sonar
      dockerfile: Dockerfile.sonar
    # image: sonarqube:8.8-developer
    # depends_on:
      # - db
    networks:
      - sonarnet
    environment:
      # - SONAR_JDBC_URL=jdbc:postgresql://db:5432/sonar
      # - SONAR_JDBC_USERNAME=sonar
      # - SONAR_JDBC_PASSWORD=sonar
      # - SONAR_SEARCH_JAVAADDITIONALOPTS=-Dnode.store.allow_mmapfs=false
      - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
      - sonar.jdbc.username=sonar
      - sonar.jdbc.password=sonar
      - sonar.search.javaAdditionalOpts=-Dnode.store.allow_mmapfs=false
    volumes:
      - sonarqube_data:/opt/sonarqube/data
      - sonarqube_extensions:/opt/sonarqube/extensions
      - sonarqube_logs:/opt/sonarqube/logs
    restart: always

###################
# DB
###################
  db:
    image: postgres:12.6
    container_name: postgress
    networks:
      - sonarnet
    environment:
      - POSTGRES_USER=sonar
      - POSTGRES_PASSWORD=sonar
    volumes:
      - postgresql:/var/lib/postgresql
      # This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52
      - postgresql_data:/var/lib/postgresql/data
    restart: always

###################
# NGINX
###################
  nginx:
    container_name: nginx
    build:
      context: ./sonar
      dockerfile: Dockerfile.nginx
    ports:
      - "80:80"
      - "443:443"
    networks:
      - sonarnet
    restart: always


###########################################################
###########################################################
# NETWORKS
###########################################################
###########################################################
networks:
  sonarnet:
    driver: bridge


###########################################################
###########################################################
# VOLUMES
###########################################################
###########################################################
volumes:
  sonarqube_data:
    external:
      name: sonarqube_data
  sonarqube_extensions:
    external:
      name: sonarqube_extensions_8_4_1
      # name: sonarqube_extensions
  sonarqube_logs:
    external:
      name: sonarqube_logs
  postgresql:
    external:
      name: postgresql
  postgresql_data:
    external:
      name: postgresql_data

Dockerfile.sonar contents

FROM sonarqube:8.4.1-developer
#FROM sonarqube:8.8-developer
COPY sonar.properties /opt/sonarqube/conf/sonar.properties

We’ve tried with 8.6.1, 8.7.1 and 8.8, all the same behavior
For the remainder of this discussion assume that we want to upgrade to 8.8

So we changed the sonarqube environment format to match 8.8

      - SONAR_JDBC_URL=jdbc:postgresql://db:5432/sonar
      - SONAR_JDBC_USERNAME=sonar
      - SONAR_JDBC_PASSWORD=sonar
      - SONAR_SEARCH_JAVAADDITIONALOPTS=-Dnode.store.allow_mmapfs=false

instead of

      - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
      - sonar.jdbc.username=sonar
      - sonar.jdbc.password=sonar
      - sonar.search.javaAdditionalOpts=-Dnode.store.allow_mmapfs=false

In any case neither format works

If we remove the dependency on the Postgres and let the image build/run with internal db, then sonarqube runs and does not report an error,
however if we keep the Postgres dependency which is where our data resides, it fails with the above error.

Originally, we were running Postgres 11.9 and ran into the same issue, checked the latest supported Postgres version and upgraded to 12.6, Sonarqube 8.4.1 is working fine with it, but we need to upgrade.

Have we missed anything obvious? or is this a bug?
We appreciate any guidance / workaround in getting this resolved as we need to move to 8.7+ version of Sonarqube to avoid other issues.

Thanks

Hi, the issue is with elasticsearch, not postgres. When you are not using any database, SonarQube starts with an embedded h2 database, for test/evaluation purpose. In that situation, checks on elasticsearch are disabled because it’s not meant to be a production instance.

When you use a real DB, SonarQube uses a different configuration for ElasticSearch, in a more “production” mode. And there are issue with docker container using the default mmapfs.

The property to disable mmapfs changed from node.store.allow_mmapfs to node.store.allow_mmap, so you need to reflect that change in your SONAR_SEARCH_JAVAADDITIONALOPTS property.

(or, alternatively, you don’t disable mmapfs, and you increase the limits on your docker host as if it was a non-docker SQ installation (increasing values for vm.max_map_count, fs.file-max, ulimit, as described here)

1 Like

Thanks Pierre for super quick response and guidance in the proper direction.
I will follow the documents and report back.

Thanks

Thanks Pierre,
Followed the steps, sonarqube started ok
However when I attempt /setup
It failed during the db upgrade process

docker logs shows this

sonarqube          | 2021.03.31 15:07:55 ERROR web[][DbMigrations] #4021 'Make 'message_type' column not nullable for `ce_task_message` table': failure | time=7ms
sonarqube          | 2021.03.31 15:07:55 ERROR web[][DbMigrations] Executed DB migrations: failure | time=3816ms
sonarqube          | 2021.03.31 15:07:55 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration failed | time=3892ms
sonarqube          | 2021.03.31 15:07:55 ERROR web[][o.s.s.p.d.m.DatabaseMigrationImpl] DB migration ended with an exception
sonarqube          | org.sonar.server.platform.db.migration.step.MigrationStepExecutionException: Execution of migration step #4021 'Make 'message_type' column not nullable for `ce_task_message` table' failed
sonarqube          |    at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:79)
sonarqube          |    at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:67)
sonarqube          |    at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:405)
sonarqube          |    at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:52)
sonarqube          |    at org.sonar.server.platform.db.migration.engine.MigrationEngineImpl.execute(MigrationEngineImpl.java:68)
sonarqube          |    at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doUpgradeDb(DatabaseMigrationImpl.java:105)
sonarqube          |    at org.sonar.server.platform.db.migration.DatabaseMigrationImpl.doDatabaseMigration(DatabaseMigrationImpl.java:80)
sonarqube          |    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
sonarqube          |    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
sonarqube          |    at java.base/java.lang.Thread.run(Unknown Source)
sonarqube          | Caused by: java.lang.IllegalStateException: Fail to execute ALTER TABLE ce_task_message ALTER COLUMN message_type TYPE VARCHAR (255), ALTER COLUMN message_type SET NOT NULL
sonarqube          |    at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:106)
sonarqube          |    at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:86)
sonarqube          |    at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:128)
sonarqube          |    at org.sonar.server.platform.db.migration.version.v85.MakeMessageTypeColumnNotNullableOnCeTaskMessageTable.execute(MakeMessageTypeColumnNotNullableOnCeTaskMessageTable.java:43)
sonarqube          |    at org.sonar.server.platform.db.migration.step.DdlChange.execute(DdlChange.java:45)
sonarqube          |    at org.sonar.server.platform.db.migration.step.MigrationStepsExecutorImpl.execute(MigrationStepsExecutorImpl.java:75)
sonarqube          |    ... 9 common frames omitted
sonarqube          | Caused by: org.postgresql.util.PSQLException: ERROR: column "message_type" contains null values
sonarqube          |    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553)
sonarqube          |    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2285)
sonarqube          |    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
sonarqube          |    at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
sonarqube          |    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
sonarqube          |    at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)
sonarqube          |    at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)
sonarqube          |    at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)
sonarqube          |    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:279)
sonarqube          |    at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
sonarqube          |    at org.apache.commons.dbcp2.DelegatingStatement.execute(DelegatingStatement.java:194)
sonarqube          |    at org.sonar.server.platform.db.migration.step.DdlChange$ContextImpl.execute(DdlChange.java:91)
sonarqube          |    ... 14 common frames omitted

And the UI displayed this (eventually)
image

Try again does not do much

Thanks

That’s a migration failing on your database, it seems that your ce_task_message contains unexpected data (null message_type).

Before proceeding to the fix, could you please run this query and post the result? It may help us to fix a potential bug in the database migration script:
select count(*) from ce_task_message where message_type is null

In this specific case it’s rather easy to fix, please run this update on your database:
update ce_task_message set message_type = 'GENERIC' where message_type is null

All message_type are generic at this stage of the migration.

You can then resume the migration (=restart SonarQube & browse /setup to trigger the upgrade).

I’m creating a ticket internally to track this issue, we are going to double-check the previous migration (supposed to populate this column, so it should not be empty, we populated the migration just before). Thanks for reporting this.

1 Like

Thanks Pierre, before I run the update, do you need any other data?
Below is the result of the query you requested
image

Million thanks Pierre, you’ve been most helpful.
The above details regarding Elasticsearch would be appreciated by others if it is added to the upgrade notes.

One thing I noticed with the latest docker, that sonarqube_extensions volume no longer has any files in plugins directory, is this by design or a problem?

With 8.4.1 we had it full of plugins.
Thanks

Thanks, that’s all I need :slight_smile:

I’m glad that helped :+1:

Well, actually, as far as i know we don’t currently advise to set SONAR_SEARCH_JAVAADDITIONALOPTS=-Dnode.store.allow_mmapfs=false. We recommend in docker hub in the Docker Host Requirements section to tune your host settings. The setting you have may come from an “old” docker-compose somewhere. Do you know where the docker-compose you use is from? Maybe we have some outdated content out there.

It’s by design: we moved bundled plugins (like analyzers) out of this folder. They are now in lib/extensions. That way, everything you have in /extensions is “yours”.

If you don’t use any custom plugins, you actually don’t need to map that folder.

Ticket is here for posterity.

Well, actually, as far as i know we don’t currently advise to set SONAR_SEARCH_JAVAADDITIONALOPTS=-Dnode.store.allow_mmapfs=false . We recommend in docker hub in the Docker Host Requirements section to tune your host settings. The setting you have may come from an “old” docker-compose somewhere. Do you know where the docker-compose you use is from? Maybe we have some outdated content out there.

Honestly I don’t remember where that setting came from, we’ve been using Sonarqube for quite some time now, probably from older requirements, because I noticed that we had updated the docker host /etc/sysctl.conf and /etc/security/limits.conf with lower limits than what is currently required.
Probably at the time it was the suggest approach.

As for the docker-compose file, we coded that ourselves at a time when developer edition of docker was not even available, and we had a Dockerfile that installed it from a zip of developer edition,
of course now that it exists, the Dockerfile.sonar is a two liner :slightly_smiling_face:

I have already made the necessary changes on the host, but also kept
SONAR_SEARCH_JAVAADDITIONALOPTS=-Dnode.store.allow_mmap=false
Just to be safe, it is not recommended, I’ll remove it.

Thanks for the extensions clarification, it makes sense to do it that way.

If your docker host actually allows you to customise the FS limit, remove this flag & go with the default mmapfs enabled. That’s the FS Store recommended by Elasticsearch. By disabling it, ES will fallback to fs , which is currently hybridfs, using either mmapfs or niofs. Since mmapfs is disabled, ES will pick niofs. We don’t have at SonarSource any benchmark on SQ performances on niofs. So it may work, but if you experience degraded performance with ES queries, we may not be able to help.

Thank you for the detailed response.
Done as you recommended and its working fine so far.

Again, many thanks Pierre.

Hi Pierre, just to inform you, there is a recommendation mismatch between what’s on Docker hub and what on Sonarqube documentation.


image

We initially had the Docker hub settings which were not working, only after making the changes you suggested which were from Sonarqube documentation, things worked.

Just thought to report this, so that docker hub notes can be updated.

Thanks

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.