Docker Container boot failed after update and rollback

Hi,

i wanted to upgrade my sonarqube developer edition docker container from 8-developer-beta to 8.2-developer. Before i updated the docker-compose file, i made backups of the volumes used.
Sadly the update wont be successfull cause of a postgres constraint error while migrating the database. (I dont saved the logs anymore). Anyway, i then wanted to spin up the old image with volumes again. But since this moment i get the following errors and the container stopped after that.

Error:

sonarqube_1                          | 2020.03.10 12:32:59 WARN  es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
sonarqube_1                          | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/opt/sonarqube/data/es6]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
sonarqube_1                          | 	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	... 6 more
sonarqube_1                          | Caused by: java.io.IOException: failed to obtain lock on /opt/sonarqube/data/es6/nodes/0
sonarqube_1                          | 	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:218) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:270) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	... 6 more
sonarqube_1                          | Caused by: java.nio.file.AccessDeniedException: /opt/sonarqube/data/es6/nodes/0/node.lock
sonarqube_1                          | 	at sun.nio.fs.UnixException.translateToIOException(Unknown Source) ~[?:?]
sonarqube_1                          | 	at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
sonarqube_1                          | 	at sun.nio.fs.UnixException.rethrowAsIOException(Unknown Source) ~[?:?]
sonarqube_1                          | 	at sun.nio.fs.UnixFileSystemProvider.newFileChannel(Unknown Source) ~[?:?]
sonarqube_1                          | 	at java.nio.channels.FileChannel.open(Unknown Source) ~[?:?]
sonarqube_1                          | 	at java.nio.channels.FileChannel.open(Unknown Source) ~[?:?]
sonarqube_1                          | 	at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:125) ~[lucene-core-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:30:25]
sonarqube_1                          | 	at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) ~[lucene-core-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:30:25]
sonarqube_1                          | 	at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) ~[lucene-core-7.7.2.jar:7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:30:25]
sonarqube_1                          | 	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:211) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:270) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.4.jar:6.8.4]
sonarqube_1                          | 	... 6 more
sonarqube_1                          | 2020.03.10 12:32:59 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 1
sonarqube_1                          | 2020.03.10 12:32:59 INFO  app[][o.s.a.SchedulerImpl] Process[es] is stopped
sonarqube_1                          | 2020.03.10 12:32:59 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

My docker-compose.yaml

version: "2"

services:
  nginx:
    restart: always
    image: nginx
    container_name: nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/etc/nginx/conf.d"
      - "/etc/nginx/vhost.d"
      - "/usr/share/nginx/html"
      - "./volumes/proxy/certs:/etc/nginx/certs:ro"

  nginx-gen:
    restart: always
    image: jwilder/docker-gen
    container_name: nginx-gen
    volumes:
      - "/var/run/docker.sock:/tmp/docker.sock:ro"
      - "./volumes/proxy/templates/nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro"
    volumes_from:
      - nginx
    entrypoint: /usr/local/bin/docker-gen -notify-sighup nginx -watch -wait 5s:30s /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf

  letsencrypt-nginx-proxy-companion:
    restart: always
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt-nginx-proxy-companion
    volumes_from:
      - nginx
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./volumes/proxy/certs:/etc/nginx/certs:rw"
    environment:
      - NGINX_DOCKER_GEN_CONTAINER=nginx-gen



  sonarqube:
    restart: always
    image: sonarqube:8-developer-beta
    environment:
      - SONARQUBE_JDBC_URL=jdbc:postgresql://db:5432/sonar
      - VIRTUAL_HOST=${SONAR_DOMAIN}
      - VIRTUAL_NETWORK=nginx-proxy
      - VIRTUAL_PORT=9000
      - LETSENCRYPT_HOST=${SONAR_DOMAIN}
      - LETSENCRYPT_EMAIL=${SONAR_DOMAIN_ADMIN}
      - sonar.jdbc.username=[REDACTED]
      - sonar.jdbc.password=[REDACTED]
      - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
    volumes:
      - sonarqube_conf:/opt/sonarqube/conf
      - sonarqube_data:/opt/sonarqube/data
      - sonarqube_extensions:/opt/sonarqube/extensions
      - sonarqube_bundled-plugins:/opt/sonarqube/lib/bundled-plugins

  db:
    restart: always
    image: postgres:11-alpine
    environment:
      - POSTGRES_USER=[REDACTED]
      - POSTGRES_PASSWORD=[REDACTED]
    volumes:
      - postgresql:/var/lib/postgresql
      - postgresql_data:/var/lib/postgresql/data

volumes:
  sonarqube_conf:
  sonarqube_data:
  sonarqube_extensions:
  sonarqube_bundled-plugins:
  postgresql:
  postgresql_data:

Can anyone tell me, why this error now appears?

Welcome to the community forum!

The logs indicate that the elasticsearch process does not have write access to the data folder. Since you mounted a volume sonarqube_data , you have to make sure that the sonarqube user of the docker image does have write permission to that volume.

Oh thanks for the fast response.
Indead there was a permission problem with the backup folder.
i did a chown -R 999:999 on the sonar volumes at and atleast the container is booting again but sadly now sonar wants to migrate the database (whyever) as well. Like in the 8.2 image.

He failed with

Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "rules_parameters_unique"
sonarqube_1                          |   Detail: Key (rule_id, name)=(456, minimumBranchCoverageRatio) already exists.
sonarqube_1                          | 	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2497)
sonarqube_1                          | 	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2233)
sonarqube_1                          | 	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:310)
sonarqube_1                          | 	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:446)
sonarqube_1                          | 	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:370)
sonarqube_1                          | 	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:149)
sonarqube_1                          | 	at org.postgresql.jdbc.PgPreparedStatement.execute(PgPreparedStatement.java:138)
sonarqube_1                          | 	at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
sonarqube_1                          | 	at org.apache.commons.dbcp2.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:94)
sonarqube_1                          | 	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:47)
sonarqube_1                          | 	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:74)
sonarqube_1                          | 	at org.apache.ibatis.executor.ReuseExecutor.doUpdate(ReuseExecutor.java:52)
sonarqube_1                          | 	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:117)
sonarqube_1                          | 	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:76)
sonarqube_1                          | 	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:197)
sonarqube_1                          | 	... 29 common frames omitted

and

db_1                                 | 2020-03-10 13:55:22.295 UTC [128] ERROR:  duplicate key value violates unique constraint "rules_parameters_unique"
db_1                                 | 2020-03-10 13:55:22.295 UTC [128] DETAIL:  Key (rule_id, name)=(456, minimumBranchCoverageRatio) already exists.
db_1                                 | 2020-03-10 13:55:22.295 UTC [128] STATEMENT:  insert into rules_parameters (
db_1                                 | 	      rule_id,
db_1                                 | 	      name,
db_1                                 | 	      param_type,
db_1                                 | 	      default_value,
db_1                                 | 	      description
db_1                                 | 	    )
db_1                                 | 	    values (
db_1                                 | 	      $1,
db_1                                 | 	      $2,
db_1                                 | 	      $3,
db_1                                 | 	      $4,
db_1                                 | 	      $5
db_1                                 | 	    )
db_1                                 | 	RETURNING "id"

Any ideas here?
Thanks

Is this error the same you described in your first post ?

It looks like the datamigration started, something got wrong somehow, and you are trying to redo it. Do you have a postgresql backup of your 8.0 version, untouched by the migration ? If so, please shutdown SonarQube, restore this backup, and start SonarQube 8.2 again and trigger the migration.

as i mentioned i backup all the volumes including the postgres ones as so the database as well before i touched anything.

So i though that restore the volumes from 8.0-beta and run 8.0-beta again will be good.

And yes. It’s the same error.

So i tried to follow your advice.

I shutdown, restore the volumes from 8.0 and start 8.2 again. Container was booting, wanted a migration, began this migration and died at the same column.

### Error updating database.  Cause: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "rules_parameters_unique"
sonarqube_1                          |   Detail: Key (rule_id, name)=(456, minimumBranchCoverageRatio) already exists.
sonarqube_1                          | ### The error may exist in org.sonar.db.rule.RuleMapper
sonarqube_1                          | ### The error may involve org.sonar.db.rule.RuleMapper.insertParameter-Inline
sonarqube_1                          | ### The error occurred while setting parameters
sonarqube_1                          | ### SQL: insert into rules_parameters (       rule_id,       name,       param_type,       default_value,       description     )     values (       ?,       ?,       ?,       ?,       ?     )
sonarqube_1                          | ### Cause: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "rules_parameters_unique"
sonarqube_1                          |   Detail: Key (rule_id, name)=(456, minimumBranchCoverageRatio) already exists.
sonarqube_1                          | 	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
sonarqube_1                          | 	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:199)

Oh that was a ride… Seems to be that, the postgres 11-alpine container was not the best idea. So i changed the image to postgres:11 and it works.

1 Like

thanks for the information!