Sonarqube issue while startup

SonarQube Server / Community Build
1

Must-share information (formatted with Markdown):

  • which versions are you using (SonarQube Server / Community Build, Scanner, Plugin, and any relevant extension) - 9.9.8
  • how is SonarQube deployed: zip, Docker, Helm - docker container
  • what are you trying to achieve - trying to bring sonarqube up where it is failing with elastic search startup.
  • what have you tried so far to achieve this - I have stopped the sonarqube and deleted the contents inside es7 folder and started sonarqube again the sonarqube is getting started but in the sonarqube main page i see a banner saying. I can see few cancelled task.
    " SonarQube completed the reload of project data. Some [tasks failed or cancelled](https://sonarqube.xyz.com/admin/background_tasks?taskType=ISSUE_SYNC&status=FAILED) causing some projects to remain unavailable."

image
image1920×860 107 KB

I’m unable to view the issues tab too it says.

Issues page is temporarily unavailable

This page will be available after the data is reloaded. This might take a while depending on the amount of projects and issues in your SonarQube instance.

You can keep analyzing your projects during this process.

sonar.log (69.6 KB)
access.log (803.2 KB)
es.log (369.4 KB)
web.log (719.7 KB)

Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!

2

Hi,

Welcome to the community!

Your version is past EOL. You should upgrade to either the latest version or the current LTA (long-term active version) at your earliest convenience. Your upgrade path is:

9.9.8 → 2025.1

You may find these resources helpful:

If you have questions about upgrading, feel free to open a new thread for that here.

Regarding your question, the es.log tells the tale:

LockObtainFailedException[Lock held by another program: /opt/sonarqube/data/es7/nodes/0/_state/write.lock];

You’ve got something interfering with SonarQube’s files, likely something “helpful” like a virus scan.

 
HTH,
Ann

3

Can we check in the UI if scans are running or inside the pod?

4

Hi,

I don’t understand the question. But I can tell you that there’s no indication in the UI where analysis ran.

 
HTH,
Ann

5

[es.log|attachment]
Looking into the server processes I can see only the java processes related to sonarqube is only running. I have ran ps -ef command.

I see more of node unhealthy and node lock problems. I have attached the es.log file.

Please guide on how to fix this problem.
es.log (3.8 MB)

6

Hi,

What shows up in your other server logs?

 
Ann

7

access.log (694.1 KB)
ce.log (16.7 KB)
sonar.log (2.7 KB)
web.log (299.9 KB)

In ce.log I can see the below warnings. I have also attached all the logs for better understanding

	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9001], URI [/_bulk?timeout=1m], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[{"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/2/no master];"}],"type":"cluster_block_exception","reason":"blocked by: [SERVICE_UNAVAILABLE/2/no master];"},"status":503}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347)
		at org.elasticsearch.client.RestClient.access$1900(RestClient.java:108)
		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:397)
		... 16 common frames omitted
8

just checked the curl request and you can see the below respose.

curl -kv http://localhost:9001
*   Trying 127.0.0.1:9001...
* Connected to localhost (127.0.0.1) port 9001 (#0)
> GET / HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-elastic-product: Elasticsearch
< content-type: application/json; charset=UTF-8
< content-length: 536
< 
{
  "name" : "sonarqube",
  "cluster_name" : "sonarqube",
  "cluster_uuid" : "eR1PWGFuRYGF2n_cW1jkhA",
  "version" : {
    "number" : "7.17.15",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "0b8ecfb4378335f4689c4223d1f1115f16bef3ba",
    "build_date" : "2023-11-10T22:03:46.987399016Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
* Connection #0 to host localhost left intact

I can see a banner saying 12 tasks are failing and the error logs show

java.lang.IllegalStateException: Unrecoverable indexation failures: 1 errors among 1 requests. Check Elasticsearch logs for further details.
	at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:42)
	at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:131)
	at org.sonar.server.measure.index.ProjectMeasuresIndexer.doIndex(ProjectMeasuresIndexer.java:158)
	at org.sonar.server.measure.index.ProjectMeasuresIndexer.indexOnAnalysis(ProjectMeasuresIndexer.java:87)
	at org.sonar.ce.task.projectanalysis.step.IndexAnalysisStep.execute(IndexAnalysisStep.java:45)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
	at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
	at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
	at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
	at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
	at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
9

Hi,

Well, there’s nothing of interest in the other logs, so I went back to the original es.log. This is not something I’ve seen before

`2025.03.19 00:00:00 WARN es[o.e.c.c.ClusterFormationFailureHelper] this node is unhealthy: health check failed due to broken node lock

So I thought I’d see what the internet had to say about it.

It says this:

That means that Elasticsearch saw a change in its data directory for which it was not responsible, so it stops all write activity to protect your data. To fix it, remove any other process that might make such a change and then restart Elasticsearch.

To be clear, you can’t restart SonarQube Elasticsearch process granularly. You’ll need to restart SonarQube.

 
HTH,
Ann

10

I have tried to redeploy sonarqube. it is complaining the same issue. I have also tried to mount the same volume in another jumphost kind of pod and deleted the contents in es7 folder and started the sonarqube server. still in es.log i see the

2025.03.25 06:30:57 WARN  es[][o.e.c.c.ClusterFormationFailureHelper] this node is unhealthy: health check failed due to broken node lock
2025.03.25 06:31:07 WARN  es[][o.e.c.c.ClusterFormationFailureHelper] this node is unhealthy: health check failed due to broken node lock
2025.03.25 06:31:17 WARN  es[][o.e.c.c.ClusterFormationFailureHelper] this node is unhealthy: health check failed due to broken node lock
2025.03.25 06:31:27 WARN  es[][o.e.c.c.ClusterFormationFailureHelper] this node is unhealthy: health check failed due to broken node lock
2025.03.25 06:31:30 WARN  es[][r.suppressed] path: /_cluster/health, params: {master_timeout=30s, level=cluster, timeout=30s}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
	at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.17.15.jar:7.17.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.17.15.jar:7.17.15]
	at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.17.15.jar:7.17.15]
	at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.17.15.jar:7.17.15]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.15.jar:7.17.15]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
2025.03.25 06:31:37 WARN  es[][o.e.c.c.ClusterFormationFailureHelper] this node is unhealthy: health check failed due to broken node lock
2025.03.25 06:31:47 WARN  es[][o.e.c.c.ClusterFormationFailureHelper] this node is unhealthy: health check failed due to broken node lock
2025.03.25 06:31:57 WARN  es[][o.e.c.c.ClusterFormationFailureHelper] this node is unhealthy: health check failed due to broken node lock
11

Hi,

You tried to redeploy SonarQube, but did you look for and disable any processes that would interfere with its files?

 
Ann