8.4.2-Developer Edition intermittent restarts

K8S:1.18
Sonar: 8.4.2-Developer
[managed via terraform/helm charts]

Ever since we upgraded from 8.2-Developer to 8.4.2-Developer, we have experienced problems:

  1. Memory we had allocated/working for 8.2-Developer was not even close enough to get 8.4.2 happy [There is a dedicated case written on this forum asking about why 8.4.2 requires that much more compute resources]
  2. Compute resources have been increased though now 8.4.2 restarts intermittently. Logs are clean - can not see any error/warning prior to the restart though can see all the expected processes coming back up after the restarts (and all of them do come back up)

We need some help to further isolate/address the underlying problem leading to these intermittent restarts. We have seen 3-6 a day.

verbosity on the logs needs to be changed? any suggestions/ideas would be appreciated.

thanks

Hello,
Well, since you’re running SonarQube in K8S there are plenty of things that are not under our control and it may be the root cause of your problem. A few questions:

  • Do you use the SonarSource official docker image ?

  • Did you try running that image outside of K8S ?

  • Can you tell what are your SonarQube processes memory settings (attaching the sonar.properties file would be helpful) ?

  • Can you reference the community thread about resources consumption on 8.4.2 ?

  • Indeed can you activate DEBUG logs (set the below in sonar.properties OR corresponding environment variables SONAR_LOG_LEVEL_* when starting your docker image (not sure the latter will work since not all env variables may be honored)

    sonar.log.level.app=DEBUG
    sonar.log.level.web=DEBUG
    sonar.log.level.ce=DEBUG
    sonar.log.level.es=DEBUG

Then send a zip of the $SONARQUBE_HOME/logs directory

Olivier

Ok, I found the other thread you’re referring to. Here: Sonarqube-8.4.2-Developer - higher memory footprint (vs 8.2-Developer)

I think your problem is simply that the SonarQube default memory settings are incompatible with the K8S resource allocation that you’re giving. [Requests: 1GiB, Limits: 2GiB] is ridiculously low.
The first thing to do is to simultaneously:

  • Lower the SonarQube default memory settings (Web: 512MB, CE: 512MB, Search: 768 MB)
  • Increase the K8S resource allocation (Requests: 2GiB, Limits: 3GiB)

Details on how to do that in the other thread.

Olivier

Hi Oliver,

Thanks.

question#1:
Yes, we do. See below
Container ID: docker://2685a993cd3c628d41b658261f9a029c6e73820230080eeeacf28d9a7390c5b6 Image: sonarqube:8.4.2-developer
Image ID: docker-pullable://sonarqube@sha256:655cf10861e40ba369e478b03a1f46d8be6cdad1db7e781a5cf8e699f5df03b5

question#2 nope. for months we had 8.2-developer edition (over K8S) working without single restart. ‘upgraded’ to 8.4.2 and started to experience these random restarts…

question#3 we have not set (neither for 8.2 nor 8.4.2) specific settings, therefore we have been using the default. Below is what 8.4.2 looks like:
ES: -Xmx512m -Xms512m
WEB: -Xmx512m -Xms128m
CE: -Xmx512m -Xms128m

question#4: looks like you found it. awesome.

suggestion#1: we will set logs to DEBUG and await until next restart takes place (we already had 2 in the last 12 hours). once we have debug logs, we will share them here.

thanks /Pedro

@OlivierK: enabled debug/all and restarted the deployment. it has not restarted (yet) though noticed 2 exceptions (pls find it attached file). Not sure if additional JVM OPTS needs to be set? Once it restarts, will send you full log…for now, check this one out…please.
8_4_2_Dev_illegal_access_startup.log (13.3 KB)

Hi @OlivierK,

I hope the DEBUG logs made its way to you (someone on our side (who has relationship towards Sonar) sent them and referred to this thread. Nevertheless, it seems we have found a way to get it stable (45 hours without disturbances/restarts). Think the biggest challenge is that K8S is not (yet - any updates on when will it be officially supported?) supported imhpo.
Memory footprint in 8.4.2 is considerable higher than what we used to have back 8.2 (both DevEdition). Day before the upgrade, 8.2 was able to handle all the work with a considerable smaller footprint. Here implies 2 changes: deployment request/limit resources (request/limit before 1G/2G now 4G/6G) as well as java/heap for all processes (twice the size of what it used to be).
Last but not least, the fact that we have not found a way to have dev-edition with replicas > 1 (not possible to mount AWS/EBS across nodes…let alone that ES (without tweaking) perhaps wont be happy either) and even if we find a way/tweak - only one will be used at time (after some ingress nginx annotations). HA? seems only on Enterprise but then again - K8S is not officially supported…(yet)
We will continue to monitor it though so far, looks very stable now. Thanks for all the help. /Pedro

Hello @peppe1977,

I got your logs. Next time I believe that you can send them through the forum:

  • It will be much easier for me to reconcile a community thread and logs received through another channel
  • There not much confidential in these logs, and you can redact the few things you want to hide
  • We may not allow to send data by any other channel. With your developer edition, the support is through the community. Your Account Mgr made you a favor to allow that as a one time exception.

So I looked in the logs and here’s what I can say:

  • So I confirm that your SonarQube is started with overall memory allocation of 1.5 GB, the standard for Developer Edition, and the standard since a long time
  • On top of the 1.5 GB for SonarQube you probably need a bit of memory for the OS to run… 512 MB or 1 GB I’d say. So technically there is no reason for SonarQube to not run with only 2-3 GB of RAM for your docker image… even if that is very little for a production system.
  • I believe that we changed nothing between 8.2 and 8.4.2 in terms of memory requirements so I doubt that this is the root cause
  • I noticed from the logs that you have 3rd party plugins installed on your platform. Even though they are quite mainstream, I would really recommend to remove them to see how things are doing without them. Plugins are Checkstyle, Code Smells, Findbugs and PMD.
  • I also noticed some ElasticSearch “debug level” errors related to K8S. We did not change anything since 8.2 so I would assume these are “normal errors”, but anyhow K8S is not formally supported as you wrote, so we cannot help much if that would be a possible explanation.
2020.09.14 17:25:03 DEBUG es[][o.e.m.o.OsProbe] error reading control group stats
java.nio.file.NoSuchFileException: /sys/fs/cgroup/cpuacct/kubepods/burstable/podcb118fdc-0eb6-414c-92f0-f1c81c1a6bd4/437f9ab67f9efd2dab7b5489658e99a119dbdaf5343d4f6484f8cb5b7d311966/cpuacct.usage

I really don’t see much that would explain a change of behaviour between 8.2 and 8.4.2 :-(.

I re-read the thread and there’s something I would like to clarify: What exactly happens when SonarQube “restarts” ? There’s absolutely no particular error in the logs before each time of restart, I highly suspect a high K8S monitoring to trigger that …
eg for your restart at 12:56 on 2020-09-14:
sonar.log:

2020.09.14 08:12:54 INFO  app[][o.s.a.SchedulerImpl] SonarQube is up
2020.09.14 12:56:09 INFO  app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2020.09.14 12:56:09 INFO  app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2020.09.14 12:56:09 INFO  app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch

web.log

2020.09.14 08:12:48 INFO  web[][o.s.s.p.Platform] WebServer is operational
2020.09.14 12:56:20 INFO  web[][o.s.p.ProcessEntryPoint] Starting web

es.log

2020.09.14 08:12:32 INFO  es[][o.e.c.r.a.AllocationService] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[metadatas][0]] ...]).
2020.09.14 12:56:11 INFO  es[][o.e.e.NodeEnvironment] using [1] data paths, mounts [[/opt/sonarqube/data (/dev/nvme1n1)]], net usable_space [48.5gb], net total_space [48.9gb], types [ext4]

ce.log:

2020.09.14 11:49:13 INFO  ce[AXSMcogsFz89Ji2G6r7U][o.s.c.t.CeWorkerImpl] Executed task | project=com.volvocars.ccdp:om-monitoring | type=REPORT | pullRequest=4 | id=AXSMcogsFz89Ji2G6r7U | submitter=gitlab_ci | status=SUCCESS | time=1360ms
2020.09.14 12:56:35 INFO  ce[][o.s.p.ProcessEntryPoint] Starting ce

The only thing that is suspect is the constant calls to SonarQube sessions/new URL, call made by kube-probe user. (See access.log file).

  • What is this for ?
  • Is this normal that this is performed so often (from several times per second, to once every 3 seconds) ?

Regards, Olivier

Hi @OlivierK,

Thank you! Appreciated. It is important to notice* that we neither changed upstream helm chart version utilized nor anything else besides 8.2-Dev to 8.4.2-Dev and whatever footprint we had allocated, was no longer good. As of now, we had to considerable increase footprint as well as heap allocation (twice what used to be be (==default)) and it has been stable now for 3-4 days.

kube-probe is part of the readiness/liveness probing and it is exactly the same as we had back with 8.2-Dev. I would not it call ‘normal’ to be taking place as often as you flagged and will look into it. As far as I recall, should be 30 or 60 seconds apart (will double check).

  • originally we upgraded 6 out of the 11 plugins as well as Sonar to 8.4.2-Dev though as it started to be very unstable, we decided to rollback to very same versions we had on 8.2-Dev for months. Idea was to reduce search domain to only sonar sw change.

Unfortunately k8s/promethus is not supported as it would have been fairly easy to create dashboards/queries and also monitor heap/gc and so on.

Cheers / Pedro