Sonarqube won't start - using Docker-compose with lts-community version since 12th August

baoun · September 1, 2022, 3:03pm

Must-share information (formatted with Markdown):

which versions are you using (SonarQube, LTS-COMMUNITY version)
what are you trying to achieve starting sonarqube with docker-compose
what have you tried so far to achieve this using differents version of sonarqube but sonar won’t start since the update done on the 12th August around 23h.

in fact I see that in the log, the last time sonar was up was on the 12 th August, and on the log folder, was ce.log, access.log and sonar.log, but since that date we have only sonar.log web.log and new es.log where I can see error on this last one as below :

2022.09.01 11:09:57 INFO  es[][o.e.n.Node] version[7.16.2], pid[38], build[default/tar/2b937c44140b6559905130a8650c64dbd0879cfb/2021-12-18T19:42:46.604893745Z], OS[Linux/3.10.0-1160.76.1.el7.x86_64/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/11.0.13/11.0.13+8]
2022.09.01 11:09:57 INFO  es[][o.e.n.Node] JVM home [/opt/java/openjdk]
2022.09.01 11:09:57 INFO  es[][o.e.n.Node] JVM arguments [-XX:+UseG1GC, -Djava.io.tmpdir=/opt/sonarqube/temp, -XX:ErrorFile=../logs/es_hs_err_pid%p.log, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djna.tmpdir=/opt/sonarqube/temp, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=COMPAT, -Des.enforce.bootstrap.checks=true, -Xmx512m, -Xms512m, -XX:MaxDirectMemorySize=256m, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/opt/sonarqube/elasticsearch, -Des.path.conf=/opt/sonarqube/temp/conf/es, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=false]
2022.09.01 11:10:05 INFO  es[][o.e.p.PluginsService] loaded module [analysis-common]
2022.09.01 11:10:05 INFO  es[][o.e.p.PluginsService] loaded module [lang-painless]
2022.09.01 11:10:05 INFO  es[][o.e.p.PluginsService] loaded module [parent-join]
2022.09.01 11:10:05 INFO  es[][o.e.p.PluginsService] loaded module [reindex]
2022.09.01 11:10:05 INFO  es[][o.e.p.PluginsService] loaded module [transport-netty4]
2022.09.01 11:10:05 INFO  es[][o.e.p.PluginsService] no plugins loaded
2022.09.01 11:10:06 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/opt/sonarqube/data (/dev/mapper/centos-data)]], net usable_space [94gb], net total_space [299.8gb], types [xfs]
2022.09.01 11:10:06 INFO  es[][o.e.e.NodeEnvironment] heap size [512mb], compressed ordinary object pointers [true]
2022.09.01 11:10:08 ERROR es[][o.e.b.Bootstrap] Exception
org.elasticsearch.ElasticsearchException: failed to bind service
        at org.elasticsearch.node.Node.<init>(Node.java:1090) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:77) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:112) [elasticsearch-cli-7.16.2.jar:7.16.2]
        at org.elasticsearch.cli.Command.main(Command.java:77) [elasticsearch-cli-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:122) [elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.2.jar:7.16.2]
 at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:80) [elasticsearch-7.16.2.jar:7.16.2]
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=9be65efb actual=64195efb (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/opt/sonarqube/data/es7/nodes/0/_state/_3w0q.cfs") [slice=_3w0q.fnm]))
        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:448) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:175) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:109) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:69) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:290) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:459) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:356) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.node.Node.<init>(Node.java:427) ~[elasticsearch-7.16.2.jar:7.16.2]
        ... 11 more
2022.09.01 11:10:08 ERROR es[][o.e.b.ElasticsearchUncaughtExceptionHandler] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=9be65efb actual=64195efb (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/opt/sonarqube/data/es7/nodes/0/_state/_3w0q.cfs") [slice=_3w0q.fnm]))];
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:157) ~[elasticsearch-7.16.2.jar:7.16.2]
.
.
.
.
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=9be65efb actual=64195efb (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/opt/sonarqube/data/es7/nodes/0/_state/_3w0q.cfs") [slice=_3w0q.fnm]))
        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:448) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:175) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:109) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:69) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:84) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:64) ~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 - mayyasharipova - 2021-10-12 15:13:05]
        at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:290) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:459) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:356) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.node.Node.<init>(Node.java:427) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.node.Node.<init>(Node.java:309) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:234) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:434) ~[elasticsearch-7.16.2.jar:7.16.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:166) ~[elasticsearch-7.16.2.jar:7.16.2]

Thanking you for any help

dimitris.kavvathas · October 12, 2022, 8:48am

After researching this, it seems that it’s connected to a version of zlib and similar issues have been reported by other users.

We were unable to reproduce it so far, but we suspect that it appears only on specific CPUs and environments, as reported here.

@baoun could you please share with us some more information about your environment and CPU, with lscpu for example?

baoun · October 12, 2022, 9:27am

Thanks Dimitris for your reply,

Please find below lscpu command output
Architecture : x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Boutisme : Little Endian
Processeur(s) : 8
Liste de processeur(s) en ligne : 0-7
Thread(s) par cœur : 1
Cœur(s) par socket : 8
Socket(s) : 1
Nœud(s) NUMA : 1
Identifiant constructeur : GenuineIntel
Famille de processeur : 15
Modèle : 6
Nom de modèle : Common KVM processor
Révision : 1
Vitesse du processeur en MHz : 2097.570
BogoMIPS : 4195.14
Constructeur d’hyperviseur : KVM
Type de virtualisation : complet
Cache L1d : 32K
Cache L1i : 32K
Cache L2 : 4096K
Nœud NUMA 0 de processeur(s) : 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology eagerfpu pni cx16 x2apic hypervisor lahf_lm

Regards,

BAOUN

dimitris.kavvathas · December 15, 2022, 11:46am

Hello @baoun,

Thanks for your patience. We have gathered information from you and a few more users and indeed the issue lies with zlib and appears only on specific CPUs.

Alpine seems to have backported a fix for zlib. Could you please try again to run SonarQube with docker and let us know if it works for you?

In case you still experience issues, please run the image with an interactive bash shell and send us the information of the zlib library version:

docker run --rm -it --entrypoint /bin/bash <sonarqube-image>
apk info zlib

Thank you,
Dimitris

baoun · December 15, 2022, 12:29pm

Thank you Dimitris for your answer, I do appreciate it.

as requested please find below the result of the command

docker exec -it sfactory_sonar bash

bash-5.1# apk info zlib

*WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/main: No such file or directory
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/community: No such file or directory

zlib-1.2.12-r3 description:

A compression/decompression Library

zlib-1.2.12-r3 webpage:
https://zlib.net/

zlib-1.2.12-r3 installed size:
108 KiB

Thanking you in advance
Bouchaib

dimitris.kavvathas · December 15, 2022, 1:08pm

Thank you, @baoun !

I assume since you sent me the command, you still get the same error with zlib-1.2.12-r3?

baoun · December 15, 2022, 1:15pm

You are right I am still waiting for anyone that can help.

dimitris.kavvathas · December 21, 2022, 4:21pm

Thanks for your quick response @baoun.

Seems that even the latest patched version of zlib in Alpine 3.13 doesn’t fix the issue for you.

Could you please try running the latest docker SonarQube version? (tag sonarqube:9.8.0-community)
We have updated this version with the latest Alpine image.

baoun · December 22, 2022, 9:01am

I do not know how to thank you Dimitris, the issue is solved with the version you proposed me.

an update of the DB is mandatory by adding /setup to the sonarqube URL, this took around 3 minutes, and all is fine.

Thank you again for your support.

BR
Baoun

dimitris.kavvathas · December 23, 2022, 11:14am

That’s great to hear @baoun! I’m glad we were able to finally help you!
Thanks for your patience and collaboration!

Best regards,
Dimitris

system · December 30, 2022, 11:16am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.