Hi community,
I’m using SonarQube Developer 9.3.0, installed via Docker - using docker-compose.yml according to the example in documentation . Then, we’re booting it up with docker-compose up -d
I’ve been having issues updating it. There is information about it in documentation , but it explains only manual stoping/removing the container, etc. As there are no additional notes about docker compose, I thought just swapping Docker image tags in docker-compose file and executing again docker-compose up -d
would be sufficient - according to the Docker Documentation , that would essentially do the same thing:
If there are existing containers for a service, and the service’s configuration or image was changed after the container’s creation, docker-compose up
picks up the changes by stopping and recreating the containers (preserving mounted volumes).
This method worked first few times without any issues, but yesterday I ended up with completely ruined / damaged database indexes and SonarQube was not able to start… Logs showed tons of error stack trace, but the main exception I guess is this:
failed to merge org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?)
Luckily we’ve created a snapshot earlier, so we could just roll back to the previous VM state and not even tried to fix the issue (as we already had once issue with db indexes being destroyed just by rebooting VM).
Can someone explain me what could be the problem in my approach or what should be the proper way to update SonarQube? This is already 2nd time in 4 months I’ve managed to destroy the db
Thanks,
Paula
1 Like
ganncamp
(G Ann Campbell)
April 1, 2022, 6:45pm
2
Hi Paula,
It sounds like a problem, not with the DB but with the Elasticsearch indexes. Can you delete them and try starting again?
Ann
Hi Ann,
Thank you for your answer.
Unfortunately, I cannot tell, as I used the VM snapshot to immediately reverse to the previous (working) state.
Could you please tell me if the way I update SonarQube is ok, or is there another preferred way to update?
Thanks in advance!
BR,
Paula
Hi, I have the same problem. I also use Docker-Compose to host Sonarqube and wanted to update the containers today. The Sonarqube container sonarqube:community
built 6 days ago doesn’t start anymore and gives the error: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?)
The error is also thrown by the ElasticSearch database as described here, so I already tried to delete the directory es7
in the data
volume of the container so that this is recreated on startup. Unfortunately this does not solve the problem. Are there any other things I can try? If you want, I can post the whole stacktrace here.
dm-ion
(David)
April 6, 2022, 4:33pm
5
Also getting this issue after SonarQube updated their docker to alpine 3.15.3.
If you build your own SonarQube docker (9.3.0 or 9.4.0 I’ve tested) on alpine 3.15.2, you should be unblocked.
1 Like
dm-ion
(David)
April 8, 2022, 6:49pm
6
FYI another thread about this issue here: Zlib 1.2.12 getting corruption errors - #4 by warkolm - Elasticsearch - Discuss the Elastic Stack
opened 12:50PM - 31 Mar 22 UTC
>bug
feedback_needed
:Core/Infra/Core
Team:Core/Infra
running ES 7.1.2 on archlinux, after zlib was upgraded from 1.2.11 to 1.2.12 the… service did not start
Taking this opportunity to upgrade to the latest elasticsearch I installed ES 8.1.0 single node with no existing data and got the same issue, after downgrading zlib both versions of ES work fine
sample stack trace
```
{"@timestamp":"2022-03-31T04:58:24.686Z", "log.level": "WARN", "message":"failing [elected-as-master ([1] nodes joined)[{gxdev1}{ntQC1xXORxaS-X7rjU0w-A}{Hye6nHtRT7iZ4qxd9FGUeg}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw} completing election,
_BECOME_MASTER_TASK_, _FINISH_ELECTION_]]: failed to commit cluster state version [79]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[gxdev1][masterService#upda
teTask][T#1]","log.logger":"org.elasticsearch.cluster.service.MasterService","elasticsearch.cluster.uuid":"55_PjKTLS5-yDT-K-pkh6w","elasticsearch.node.id":"ntQC1xXORxaS-X7rjU0w-A","elasticsearch.node.name":"gxdev1","elasticsearch.cluster.
name":"elasticsearch","error.type":"org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException","error.message":"publication failed","error.stack_trace":"org.elasticsearch.cluster.coordination.FailedToCommitClusterStateExc
eption: publication failed\n\tat org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1718)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(Listenabl
eFuture.java:115)\n\tat org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55)\n\tat org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1625)\n\
tat org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114)\n\tat org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:165)\n\tat org.elasticsearch.cluster.coord
ination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:376)\n\tat org.elasticsearch.cluster.coordination.Coordinator$4.onFailure(Coordinator.java:1371)\n\tat org.elasticsearch.cluster.coordination.Publicat
ionTransportHandler$PublicationContext$1.onFailure(PublicationTransportHandler.java:360)\n\tat org.elasticsearch.cluster.coordination.PublicationTransportHandler$PublicationContext.lambda$sendClusterStateDiff$7(PublicationTransportHandler
.java:438)\n\tat org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:192)\n\tat org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)\n\tat org.elasticsearch.action
.ActionListener$RunAfterActionListener.onFailure(ActionListener.java:350)\n\tat org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)\n\tat org.elasticsearch.action.ActionListener$RunAfterActionListener.onFa
ilure(ActionListener.java:350)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleExce
ption(TransportService.java:1349)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1458)\n\tat org.elasticsearch.transport.TransportService$DirectResponseChannel$2.run(Transpo
rtService.java:1437)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed e
xecution\n\tat org.elasticsearch.common.util.concurrent.FutureUtils.rethrowExecutionException(FutureUtils.java:80)\n\tat org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:72)\n\tat org.elasticsearch.common.util.con
current.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:112)\n\t... 21 more\nCaused by: java.util.concurrent.ExecutionException: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=2
e603023 actual=f0db10c0 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path=\"/mq_cluster/data/elasticsearch/_state/_9b.fdt\")))\n\tat org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)\n\tat org.
elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:231)\n\tat org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:53)\n\tat org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.jav
a:65)\n\t... 22 more\nCaused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=2e603023 actual=f0db10c0 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path=\"/mq_cluster/data/elasticse
arch/_state/_9b.fdt\")))\n\tat org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:440)\n\tat org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.writeCompoundFile(Lucene90CompoundFormat.java:123)\n\tat org.apache.lucene.co
decs.lucene90.Lucene90CompoundFormat.write(Lucene90CompoundFormat.java:98)\n\tat org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5563)\n\tat org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(Docum
entsWriterPerThread.java:537)\n\tat org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:468)\n\tat org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:497)\n\tat org.apache.lucene.index.Do
cumentsWriter.flushAllThreads(DocumentsWriter.java:676)\n\tat org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4014)\n\tat org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3988)\n\tat org.apache.lucene.index.IndexWri
ter.flush(IndexWriter.java:3967)\n\tat org.elasticsearch.gateway.PersistedClusterStateService$MetadataIndexWriter.flush(PersistedClusterStateService.java:692)\n\tat org.elasticsearch.gateway.PersistedClusterStateService$Writer.addMetadata
(PersistedClusterStateService.java:991)\n\tat org.elasticsearch.gateway.PersistedClusterStateService$Writer.overwriteMetadata(PersistedClusterStateService.java:975)\n\tat org.elasticsearch.gateway.PersistedClusterStateService$Writer.write
FullStateAndCommit(PersistedClusterStateService.java:788)\n\tat org.elasticsearch.gateway.GatewayMetaState$LucenePersistedState.setLastAcceptedState(GatewayMetaState.java:504)\n\tat org.elasticsearch.cluster.coordination.CoordinationState
.handlePublishRequest(CoordinationState.java:392)\n\tat org.elasticsearch.cluster.coordination.Coordinator.handlePublishRequest(Coordinator.java:418)\n\tat org.elasticsearch.cluster.coordination.PublicationTransportHandler.acceptState(Pub
licationTransportHandler.java:200)\n\tat org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:183)\n\tat org.elasticsearch.cluster.coordination.PublicationTranspo
rtHandler.lambda$new$0(PublicationTransportHandler.java:103)\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)\n\tat org.elasticsearch.transport.TransportService$6.doRun(Transp
ortService.java:917)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26
)\n\t... 3 more\n"}
```
```
org.elasticsearch.cluster.coordination.FailedToCommitClusterStateException: publication failed
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication$4.onFailure(Coordinator.java:1718)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:115)
at org.elasticsearch.common.util.concurrent.ListenableFuture.addListener(ListenableFuture.java:55)
at org.elasticsearch.cluster.coordination.Coordinator$CoordinatorPublication.onCompletion(Coordinator.java:1625)
at org.elasticsearch.cluster.coordination.Publication.onPossibleCompletion(Publication.java:114)
at org.elasticsearch.cluster.coordination.Publication.onPossibleCommitFailure(Publication.java:165)
at org.elasticsearch.cluster.coordination.Publication$PublicationTarget$PublishResponseHandler.onFailure(Publication.java:376)
at org.elasticsearch.cluster.coordination.Coordinator$4.onFailure(Coordinator.java:1371)
at org.elasticsearch.cluster.coordination.PublicationTransportHandler$PublicationContext$1.onFailure(PublicationTransportHandler.java:360)
at org.elasticsearch.cluster.coordination.PublicationTransportHandler$PublicationContext.lambda$sendClusterStateDiff$7(PublicationTransportHandler.java:438)
at org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:192)
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
at org.elasticsearch.action.ActionListener$RunAfterActionListener.onFailure(ActionListener.java:350)
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66)
at org.elasticsearch.action.ActionListener$RunAfterActionListener.onFailure(ActionListener.java:350)
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1349)
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1458)
at org.elasticsearch.transport.TransportService$DirectResponseChannel$2.run(TransportService.java:1437)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
atjava.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
at org.elasticsearch.common.util.concurrent.FutureUtils.rethrowExecutionException(FutureUtils.java:80)
at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:72)
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListenerDirectly(ListenableFuture.java:112)
... 21 more
Caused by: java.util.concurrent.ExecutionException: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=2e603023 actual=f0db10c0 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/mq_cluster/data/elasticsearch/_state/_9b.fdt")))
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:257)
at org.elasticsearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:231)
at org.elasticsearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:53)
at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:65)
... 22 more
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=2e603023 actual=f0db10c0 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/mq_cluster/data/elasticsearch/_state/_9b.fdt")))
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:440)
at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.writeCompoundFile(Lucene90CompoundFormat.java:123)
at org.apache.lucene.codecs.lucene90.Lucene90CompoundFormat.write(Lucene90CompoundFormat.java:98)
at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5563)
at org.apache.lucene.index.DocumentsWriterPerThread.sealFlushedSegment(DocumentsWriterPerThread.java:537)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:468)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:497)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:676)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4014)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3988)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3967)
at org.elasticsearch.gateway.PersistedClusterStateService$MetadataIndexWriter.flush(PersistedClusterStateService.java:692)
at org.elasticsearch.gateway.PersistedClusterStateService$Writer.addMetadata(PersistedClusterStateService.java:991)
at org.elasticsearch.gateway.PersistedClusterStateService$Writer.overwriteMetadata(PersistedClusterStateService.java:975)
at org.elasticsearch.gateway.PersistedClusterStateService$Writer.writeFullStateAndCommit(PersistedClusterStateService.java:788)
at org.elasticsearch.gateway.GatewayMetaState$LucenePersistedState.setLastAcceptedState(GatewayMetaState.java:504)
at org.elasticsearch.cluster.coordination.CoordinationState.handlePublishRequest(CoordinationState.java:392)
at org.elasticsearch.cluster.coordination.Coordinator.handlePublishRequest(Coordinator.java:418)
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.acceptState(PublicationTransportHandler.java:200)
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.handleIncomingPublishRequest(PublicationTransportHandler.java:183)
at org.elasticsearch.cluster.coordination.PublicationTransportHandler.lambda$new$0(PublicationTransportHandler.java:103)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:67)
at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:917)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:776)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26
... 3 more
```
elMachete
(Jan Macheta)
April 11, 2022, 3:29pm
7
I was also experiencing the same issue after upgrading to sonarqube:9.4-developer image.
The reason is probably the same, as in @dm-ion link - I run the container under VM on xcp-ng (PVHVM mode) but AFAIK I can’t change cpu type for the VM.
Building custom image with alpine 3.15.2 worked ok for me
dm-ion
(David)
April 11, 2022, 4:34pm
8
In my post I only referenced the latest alpine version. The SonarQube docker is on the 3.14 line, so for our docker of SonarQube I’ve been using 3.14.2 to be consistent with what SonarQube has been using. No reason to believe this would be any different then 3.15.2 for SonarQube, but going with what they’ve tested.
So any alpine version after (and including) these you may have some Zip CRC issues, or issues with elastic search checksums it seems:
3.12.11
3.13.9
3.14.5
3.15.3
So, looks like the root of the problem is updated zlib
library to v1.2.12 in the alpine
images, which changes are then reflected in rebuilt docker images of sonarqube
?
I am also running Sonarqube via Docker inside a VM on a Proxmox server - suggested solution is to change the CPU type to Haswell
- does anyone know what are consequences of this? I am also running some other services on the same VM at the moment… Is this indeed an intended behavior from zlib
or should this be fixed from their side for kvm64
CPU architecture type?
Thanks in advance!
dm-ion
(David)
April 28, 2022, 9:22pm
10
Hi looks like this will be fixed in a newer version of the SonarQube docker when they update alpine or if they force updating of zlib. The alpine folks have updated ZLib to include a patch for this issue and pushed to the 3.15 repo and I can confirm it works.
So to do this you simply copy paste the SonarQube Dockerfile, upgrade alpine to 3.15.4, then add this command:
RUN apk upgrade zlib
Correct incorrect inputs provided to the CRC functions. · madler/zlib@ec3df00 (github.com)
aports - Alpine packages build scripts (alpinelinux.org)
1 Like
Fast forward… It’s been now more then 2 months, here is even an issue on GitHub about this topic: No longer able to run sonarqube:9.3.0-community · Issue #544 · SonarSource/docker-sonarqube · GitHub
We know that solution to problem is updating zlib
library to 1.2.12-r1
. However, I’m not keen on writing my own Docker image just bcos of this. So, I ask here the same question as it was asked on GitHub (above): Is this one-line fix going to be implemented from Sonarsource (@ganncamp ), or we have to take care of it? Or I have to make a pull request on my own on GitHub?
Best regards,
Paula
Hi all,
Sorry for the late reply, and delay in dealing with this. This should have been fixed with 9.5. Furthermore, we were also waiting on Alpine to publish a new version of 3.15 which would include some other fixes as well. I’ll update our base image today. This will be part of the upcoming 9.6 version.
system
(system)
Closed
August 5, 2022, 1:29pm
15
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.