Sonarqube and Elasticsearch services keeps restarting after upgrading to 10.0.0

SonarQube Server / Community Build
,
1

Below is the configuration used:
Server:

  • OS - Amazon Linux 2
  • Instance Type - r7i.xlarge
  • Ram - 32 GB

Sonarqube:

  • Sonarqube version - 10.0.0.68432 (Community)
  • sonarqube-community-branch-plugin - 1.15.0
  • sonar-cxx-plugin - 2.1.2.736
  • How is SonarQube deployed: zip (using chef cookbook)
  • RDS Engine - PostgreSQL
  • Engine version - 11.22
  • Java version - 17

What I am trying to achieve:

  • Trying to upgrade the Sonarqube version from 9.9.6 to 10.0.0

The error I have found in sonar.log

INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:43513]
INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/mnt/apps/sonarqube-10.0.0.68432/elasticsearch]: /usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/mnt/apps/sonarqube-10.0.0.68432/elasticsearch -Des.path.conf=/mnt/apps/sonarqube-10.0.0.68432/temp/conf/es -Des.distribution.type=tar -cp /mnt/apps/sonarqube-10.0.0.68432/elasticsearch/lib/*:/mnt/apps/sonarqube-10.0.0.68432/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 134
INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /mnt/apps/sonarqube-10.0.0.68432/temp
INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:41057]
INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/mnt/apps/sonarqube-10.0.0.68432/elasticsearch]: /usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/mnt/apps/sonarqube-10.0.0.68432/elasticsearch -Des.path.conf=/mnt/apps/sonarqube-10.0.0.68432/temp/conf/es -Des.distribution.type=tar -cp /mnt/apps/sonarqube-10.0.0.68432/elasticsearch/lib/*:/mnt/apps/sonarqube-10.0.0.68432/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 134
INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /mnt/apps/sonarqube-10.0.0.68432/temp
INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:43911]
INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/mnt/apps/sonarqube-10.0.0.68432/elasticsearch]: /usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/mnt/apps/sonarqube-10.0.0.68432/elasticsearch -Des.path.conf=/mnt/apps/sonarqube-10.0.0.68432/temp/conf/es -Des.distribution.type=tar -cp /mnt/apps/sonarqube-10.0.0.68432/elasticsearch/lib/*:/mnt/apps/sonarqube-10.0.0.68432/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 134
INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /mnt/apps/sonarqube-10.0.0.68432/temp
INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:40151]
INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/mnt/apps/sonarqube-10.0.0.68432/elasticsearch]: /usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/mnt/apps/sonarqube-10.0.0.68432/elasticsearch -Des.path.conf=/mnt/apps/sonarqube-10.0.0.68432/temp/conf/es -Des.distribution.type=tar -cp /mnt/apps/sonarqube-10.0.0.68432/elasticsearch/lib/*:/mnt/apps/sonarqube-10.0.0.68432/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running

What have I tried so far:

  • Tried to set the JVM heap size to 4g using the export command by referring Heap: Sizing and Swapping | Elasticsearch: The Definitive Guide [2.x] | Elastic

  • export ES_HEAP_SIZE=4g

  • Added export ES_JAVA_OPTS="-Xms4g -Xmx4g" in the {install_dir}/sonarqube/bin/linux-x86-64/sonar.sh

  • Tried fresh installation by terminating the server and creating it again
    None of the above steps helped. The 9.9.6 version is working fine on the server.

Let me know if any more information is needed to troubleshoot this

2

Hi,

Welcome to the community!

It’s not clear to me why you would upgrade from 9.9.6 to 10.0 when 10.0 is already EOL. You should be targeting 10.6.

And you should check your Elasticsearch log.

 
HTH,
Ann

3

Hi Ann,
Per your recommendation, I installed the 10.6 version still face the same issue.

tail -f sonar.log 
2024.07.31 16:30:22 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /mnt/apps/sonarqube-10.6.0.92116/temp
2024.07.31 16:30:22 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:34045]
2024.07.31 16:30:22 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/mnt/apps/sonarqube-10.6.0.92116/elasticsearch]: /usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/mnt/apps/sonarqube-10.6.0.92116/elasticsearch -Des.path.conf=/mnt/apps/sonarqube-10.6.0.92116/temp/conf/es -Des.distribution.type=tar -cp /mnt/apps/sonarqube-10.6.0.92116/elasticsearch/lib/*:/mnt/apps/sonarqube-10.6.0.92116/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
2024.07.31 16:30:22 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2024.07.31 16:30:25 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 134
2024.07.31 16:30:25 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2024.07.31 16:30:25 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped
2024.07.31 16:30:26 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /mnt/apps/sonarqube-10.6.0.92116/temp
2024.07.31 16:30:26 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on [HTTP: 127.0.0.1:9001, TCP: 127.0.0.1:43523]
2024.07.31 16:30:27 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[ELASTICSEARCH] from [/mnt/apps/sonarqube-10.6.0.92116/elasticsearch]: /usr/lib/jvm/java-17-amazon-corretto.x86_64/bin/java -Xms4m -Xmx64m -XX:+UseSerialGC -Dcli.name=server -Dcli.script=./bin/elasticsearch -Dcli.libs=lib/tools/server-cli -Des.path.home=/mnt/apps/sonarqube-10.6.0.92116/elasticsearch -Des.path.conf=/mnt/apps/sonarqube-10.6.0.92116/temp/conf/es -Des.distribution.type=tar -cp /mnt/apps/sonarqube-10.6.0.92116/elasticsearch/lib/*:/mnt/apps/sonarqube-10.6.0.92116/elasticsearch/lib/cli-launcher/* org.elasticsearch.launcher.CliToolLauncher
2024.07.31 16:30:27 INFO  app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2024.07.31 16:30:29 INFO  app[][o.s.a.SchedulerImpl] Stopping SonarQube
2024.07.31 16:30:29 INFO  app[][o.s.a.SchedulerImpl] Sonarqube has been requested to stop
2024.07.31 16:30:29 INFO  app[][o.s.a.SchedulerImpl] Stopping [Compute Engine] process...
2024.07.31 16:30:29 INFO  app[][o.s.a.SchedulerImpl] Stopping [Web Server] process...
2024.07.31 16:30:29 INFO  app[][o.s.a.SchedulerImpl] Stopping [ElasticSearch] process...
2024.07.31 16:30:29 INFO  app[][o.s.a.SchedulerImpl] Process[ElasticSearch] is stopped
2024.07.31 16:30:29 WARN  app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [ElasticSearch]: 143
2024.07.31 16:30:29 INFO  app[][o.s.a.SchedulerImpl] SonarQube is stopped

Also, there are no es.log and web.log files populated, not sure why

ls -la
total 32
drwxr-xr-x.  2 sonarqube sonarqube    23 Jul 31 16:24 .
drwxr-xr-x. 18 root      root      16384 Jul 31 16:00 ..
-rw-r--r--.  1 sonarqube sonarqube 10483 Jul 31 16:30 sonar.log

Let me know if I should provide any other info.
Thanks,
Swapnil.

4

Hi Swapnil,

So sonarqube owns the logs directory, but root owns the SonarQube directory? I would start with chown -R sonarqube [name of SQ directory].

At a guess, Elasticsearch can’t write its data files - unless you’ve configured them to go somewhere else?

 
Ann

5

Hi Ann,
Thanks for your quick responses.

I tried the steps you provided, still have the same issue, no es.log file populated after restarting the sonarqube service

ls -la
total 916
drwxr-xr-x.  2 sonarqube sonarqube     23 Jul 31 16:24 .
drwxr-xr-x. 18 sonarqube sonarqube  16384 Jul 31 17:00 ..
-rw-r-----.  1 sonarqube sonarqube 459292 Jul 31 17:25 sonar.log

Status of Sonarqube service (getting restarted)

systemctl status sonarqube
â—Ź sonarqube.service - SonarQube service
   Loaded: loaded (/etc/systemd/system/sonarqube.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2024-07-31 17:26:07 UTC; 186ms ago
  Process: 26337 ExecStop=/mnt/apps/sonarqube/bin/linux-x86-64/sonar.sh stop (code=exited, status=0/SUCCESS)
  Process: 26438 ExecStart=/mnt/apps/sonarqube/bin/linux-x86-64/sonar.sh start (code=exited, status=0/SUCCESS)
 Main PID: 26515 (java)
   Memory: 48.7M
   CGroup: /system.slice/sonarqube.service
           └─26515 java -Xms8m -Xmx32m --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/sun.ni...

Jul 31 17:26:06 xxxxx systemd[1]: Starting SonarQube service...
Jul 31 17:26:06 xxxxx sonar.sh[26438]: /usr/bin/java
Jul 31 17:26:07 xxxxx sonar.sh[26438]: Starting SonarQube...
Jul 31 17:26:07 xxxxx systemd[1]: Started SonarQube service.

But I am curious why it is working perfectly for the 9.9.6 version.

Below are the log files from the server with version 9.9.6 (skipping some zipped files as the list is long)

ls -lat
total 11308
-rw-r-----.  1 sonarqube sonarqube 158799 Jul 31 17:33 access.log
-rw-r-----.  1 sonarqube sonarqube  10024 Jul 31 17:29 web.log
-rw-r-----.  1 sonarqube sonarqube    756 Jul 31 17:20 es.log
-rw-r-----.  1 sonarqube sonarqube  64988 Jul 31 17:17 ce.log
drwxr-xr-x. 19 root      root      139264 Jul 31 17:00 ..
drwxr-xr-x.  2 sonarqube sonarqube  57344 Jul 31 17:00 .
.
.
.
-rw-r-----.  1 sonarqube sonarqube   4510 Jul 13 11:18 sonar.log

There is the same config used for both servers, just the change in the Sonarqube version.

Swapnil.

6

Hi,

I’m not sure what to tell you. If it were me, my text steps would be to touch the log files into existence & chown sonarqube:sonarqube them. To see if that helps.

After that, it’s time to look at your service setup. Are you trying to start 10.6 as a service, or just as a process?

 
Ann

7

Hi Ann,
I tried that also in past, just forgot to mention it.

After the touching log files manually, restarted the Sonarqube and faced the same issue.
I am starting Sonarqube as a service using systemctl

I tried to move the log files to the installation directory of Sonarqube, but I still had the same issue. Log files were empty.

ls -la
total 220
drwxr-xr-x.  2 sonarqube sonarqube   4096 Jul 31 18:21 .
drwxr-xr-x. 12 sonarqube sonarqube   4096 Jun 21 06:24 ..
-rw-r--r--.  1 sonarqube sonarqube      0 Jul 31 18:20 access.log
-rw-r--r--.  1 sonarqube sonarqube      0 Jul 31 18:20 ce.log
-rw-r--r--.  1 sonarqube sonarqube      0 Jul 31 18:20 es.log
-rw-r--r--.  1 sonarqube sonarqube    127 Jul 31 18:27 nohup.log
-rw-r--r--.  1 sonarqube sonarqube     88 Jun 21 06:19 README.txt
-rw-r--r--.  1 sonarqube sonarqube 204215 Jul 31 18:27 sonar.log
-rw-r--r--.  1 sonarqube sonarqube      0 Jul 31 18:20 web.log

My question still remains the same, why does it occur in the 10.x series only? In the past, I reverted back to 9.9.5 and 9.9.6 several times to verify and it works perfectly for those versions with the same setup. But as soon as I upgrade to any version from the 10.x series, the Sonarqube service gets restarted frequently.

Swapnil.

8

Hi,

Can you try starting it directly, not as a service?

I can’t answer this until we know the root cause.

 
Ann

9

Hi,

I tried to start the Sonarqube directly,

/mnt/apps/sonarqube/bin/linux-x86-64/sonar.sh start
/bin/java
Starting SonarQube...
Started SonarQube.

It showed that the Sonarqube had started, but it stopped immediately. As I went to the URL it showed 502 Bad Gateway
Screenshot 2024-08-06 at 11.51.01

I tried to stop the Sonarqube, but it showed, SonarQube was not running.

/mnt/apps/sonarqube/bin/linux-x86-64/sonar.sh stop
/bin/java
Gracefully stopping SonarQube...
SonarQube was not running.

I compared the sonar.sh script from version 10.6 and 9.9.5, and I found a difference,
sonar.sh from 10.6 had below lines

# Location of the pid file.
PIDDIR="${PIDDIR-.}"

and

PIDFILE="$PIDDIR/$APP_NAME.pid"

whereas sonar.sh from 9.9.5 had below lines with respect to the 10.6

PIDFILE="./$APP_NAME.pid"

I tried to match these lines and then started the Sonarqube

/mnt/apps/sonarqube/bin/linux-x86-64/sonar.sh start
/bin/java
Starting SonarQube...
Started SonarQube.

but it still failed

/mnt/apps/sonarqube/bin/linux-x86-64/sonar.sh stop
/bin/java
Gracefully stopping SonarQube...
SonarQube was not running.

and sonar.log had a similar error as before.
The es.log file was not populated with any logs.

I tried to terminate the server and perform all these steps again, but I found the same error again.

Swapnil

11

Hi Swapnil,

If SonarQube can’t create its files, it certainly won’t be able to run.

Going back to an earlier post, it looks like the SonarQube directory is owned by root? You should make sure SonarQube owns its own directory, recursively.

 
Ann

12

Hi Ann,

We were able to resolve this issue.

In the file {sonar_installation_dir}/sonarqube/conf/sonar.properties there was a line

sonar.search.javaAdditionalOpts=-Dlog4j2.formatMsgNoLookups=true -javaagent:/opt/jolokia/jolokia-jvm-agent.jar=port=7779,host=localhost

removed the part

-javaagent:/opt/jolokia/jolokia-jvm-agent.jar=port=7779,host=localhost

And the service started running normally.
We verified it by relaunching the server again with this change.

However, we did not change any other parts of the configuration like sonar.web.javaAdditionalOpts and sonar.ce.javaAdditionalOpts

Thanks for all the support.

Swapnil.

1 Like
13

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.