Need assistance on Sonarqube upgrade from 5.6.6 to 6.7.6

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
    v5.6.6
  • what are you trying to achieve
    We are trying to upgrade to the Sonarqube version from v5.6.6 to v6.7.6.
  • what have you tried so far to achieve this
    Steps followed:
    We tried to upgrade sonarqube from version v5.6.6 --> v6.7.6 but ran into below mentioned error after 30 - 45 mins when I start the upgrade activity.

Sonar.log:
2019.04.23 23:02:07 DEBUG app[o.e.c.t.TransportClientNodesService] failed to connect to node [{#transport#-1}{OLD6OD2iSES5EnwzAsberA}{127.0.0.1}{127.0.0.1:9001}], ignoring…
org.elasticsearch.transport.ConnectTransportException: [127.0.0.1:9001] connect_timeout[30s]
at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:362)
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570)
at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:117)
at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:351)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:407)
at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:357)
at org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:198)
at org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:319)
at org.sonar.application.process.EsProcessMonitor.addHostToClient(EsProcessMonitor.java:186)
at org.sonar.application.process.EsProcessMonitor.buildTransportClient(EsProcessMonitor.java:177)
at org.sonar.application.process.EsProcessMonitor.getTransportClient(EsProcessMonitor.java:160)
at org.sonar.application.process.EsProcessMonitor.checkStatus(EsProcessMonitor.java:134)
at org.sonar.application.process.EsProcessMonitor.checkOperational(EsProcessMonitor.java:93)
at org.sonar.application.process.EsProcessMonitor.isOperational(EsProcessMonitor.java:78)
at org.sonar.application.process.SQProcess.refreshState(SQProcess.java:161)
at org.sonar.application.process.SQProcess$EventWatcher.run(SQProcess.java:220)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9001

Web.log:
2019.04.23 23:06:41 INFO web[DbMigrations] #1207 ‘Drop unused columns on PROJECT_MEASURES’…
2019.04.23 23:23:16 WARN web[o.s.p.ProcessEntryPoint] Fail to start web
java.lang.IllegalArgumentException: Unable to create shared memory :
at org.sonar.process.sharedmemoryfile.AllProcessesCommands.(AllProcessesCommands.java:100)
at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.(DefaultProcessCommands.java:34)
at org.sonar.process.sharedmemoryfile.DefaultProcessCommands.secondary(DefaultProcessCommands.java:52)
at org.sonar.server.app.WebServer.isOperational(WebServer.java:68)
at org.sonar.server.app.WebServer.getStatus(WebServer.java:60)
at org.sonar.process.ProcessEntryPoint.waitForOperational(ProcessEntryPoint.java:145)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:120)
at org.sonar.process.ProcessEntryPoint.launch(ProcessEntryPoint.java:100)
at org.sonar.server.app.WebServer.main(WebServer.java:91)
Caused by: java.io.FileNotFoundException: /data/sonarqube-6.7.6/temp/sharedmemory (Too many open files in system)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.(RandomAccessFile.java:243)
at org.sonar.process.sharedmemoryfile.AllProcessesCommands.(AllProcessesCommands.java:97)
… 8 common frames omitted

Our server configuration:
CPU : 2
RAM: 4 GB
sonar.jdbc.username=
sonar.jdbc.password=
sonar.jdbc.url=jdbc:mysql://schema_name?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance
sonar.jdbc.maxActive=100
sonar.jdbc.maxIdle=20
sonar.jdbc.minIdle=10
sonar.jdbc.maxWait=5000
sonar.jdbc.minEvictableIdleTimeMillis=600000
sonar.jdbc.timeBetweenEvictionRunsMillis=30000
sonar.web.javaOpts=-Xmx4096m -XX:MaxPermSize=2048m -XX:+HeapDumpOnOutOfMemoryError
sonar.web.host=
sonar.web.context=/sonar
sonar.web.port=1082
sonar.log.level=DEBUG
sonar.notifications.delay=60
sonar.security.localUsers=admin
sonar.authenticator.createUsers: true
sonar.security.updateUserAttributes: true

Server config:
sysctl -w vm.max_map_count=262144
sysctl -w fs.file-max=65536
ulimit -n 65536
ulimit -u 2048

Pleaser help me to resolve the issue

Hi,

Thanks for including both log errors in your first post! :smile:

Here’s your problem:

You should look at upping your file handle limit. The error in the ES log is in this case a red herring; the nodes keep up with each other, and when one (in this case the web node) shuts down the others follow to not leave you with a zombie instance.

 
HTH,
Ann

Hi Ann,

Appreciate your quick response,

Even we tried to set ulimit -u value as 4096 and unlimit but we ran with same error and please find the es.log detail as i dont see any errors

Worst case, we need to populate the sonar dump each time its getting failed otherwise if we tried to reinitiate we ran with db errors as “Caused by: java.lang.IllegalStateException: Fail to execute ALTER TABLE project_measures DROP COLUMN rules_category_id, DROP COLUMN tendency, DROP COLUMN measure_date, DROP COLUMN url, DROP COLUMN rule_priority, DROP COLUMN characteristic_id, DROP COLUMN rule_id” :frowning:

ES.log
2019.04.23 23:02:08 DEBUG es[o.e.b.SystemCallFilter] Linux seccomp filter installation successful, threads: [all]
2019.04.23 23:02:08 DEBUG es[o.e.b.JarHell] java.class.path: /data/sonarqube-6.7.6/elasticsearch/lib/HdrHistogram-2.1.9.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-spatial3d-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-misc-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/jts-1.13.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-highlighter-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-queryparser-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-memory-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/jackson-core-2.8.6.jar:/data/sonarqube-6.7.6/elasticsearch/lib/joda-time-2.9.5.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-sandbox-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-grouping-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-suggest-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/jackson-dataformat-cbor-2.8.6.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-join-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/jopt-simple-5.0.2.jar:/data/sonarqube-6.7.6/elasticsearch/lib/hppc-0.7.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/securesm-1.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/java-version-checker-5.6.3.jar:/data/sonarqube-6.7.6/elasticsearch/lib/jna-4.4.0-1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-queries-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/snakeyaml-1.15.jar:/data/sonarqube-6.7.6/elasticsearch/lib/spatial4j-0.6.jar:/data/sonarqube-6.7.6/elasticsearch/lib/log4j-core-2.9.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/log4j-1.2-api-2.9.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/jackson-dataformat-smile-2.8.6.jar:/data/sonarqube-6.7.6/elasticsearch/lib/elasticsearch-5.6.3.jar:/data/sonarqube-6.7.6/elasticsearch/lib/jackson-dataformat-yaml-2.8.6.jar:/data/sonarqube-6.7.6/elasticsearch/lib/log4j-api-2.9.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-analyzers-common-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-spatial-extras-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-spatial-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/t-digest-3.0.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-backward-codecs-6.6.1.jar:/data/sonarqube-6.7.6/elasticsearch/lib/plugin-cli-5.6.3.jar:/data/sonarqube-6.7.6/elasticsearch/lib/lucene-core-6.6.1.jar
2019.04.23 23:02:08 DEBUG es[o.e.b.JarHell] sun.boot.class.path: /opt/jdk1.8.0_141/jre/lib/resources.jar:/opt/jdk1.8.0_141/jre/lib/rt.jar:/opt/jdk1.8.0_141/jre/lib/sunrsasign.jar:/opt/jdk1.8.0_141/jre/lib/jsse.jar:/opt/jdk1.8.0_141/jre/lib/jce.jar:/opt/jdk1.8.0_141/jre/lib/charsets.jar:/opt/jdk1.8.0_141/jre/lib/jfr.jar:/opt/jdk1.8.0_141/jre/classes

Hi,

Take a look here: https://docs.sonarqube.org/latest/requirements/requirements/#header-5

 
Ann

Hi Ann,

Thanks a lot for your response! I have followed the same requirements before performing the upgrade activity.

I could see that “Not recommended for large instances” mentioned in requirement document, can you confirm if there is a limitation for MySQL server.

At the moment, we are effectively populated with the DB data of 49 GB.

Please confirm!

Hi,

Our experience was that there could be problems under load & the larger your instance the more often/likely you’re “under load”.

And for the next LTS, 7.9 (E.T.A. “soon”) we won’t support MySQL at all:

 
Ann

1 Like

Thank you Ann for your response! Just in curios of asking is there any chance of supporting Oracle database for community version post 6.7.6 version as MySQL is no longer supported post 7.9 version

Hi,

Oracle is still supported in all editions. No plans to stop supporting it.

 
Ann

Thanks Ann for the confirmation!