Sonarqube helm deployment is crashing on running the init-sysctl container

Must-share information (formatted with Markdown):

  • which versions are you using SonarQube 9.9-LTS (lts-community tag)
  • how is SonarQube deployed: Helm
  • what are you trying to achieve deploy working sonarqube
  • what have you tried so far to achieve this deploying the helm chart with helm install helm install sonarqube sonarqube/sonarqube

Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!

Hi I am new to sonarqube and we are trying to deploy the official helm chart. Unfortunatelly there is a problem with the init-sysctl container. It is crashing and stopping the whole deployment.

We tried first to deploy the basic chart by the documentation:

helm repo add sonarqube https://SonarSource.github.io/helm-chart-sonarqube
helm repo update
kubectl create namespace sonarqube
helm upgrade --install -n sonarqube sonarqube sonarqube/sonarqube

But executing it results in:

pod/sonarqube-sonarqube-0   0/1     Init:CrashLoopBackOff   6 (46s ago)   6m14s

We were digging around in the logs and found that the init-sysctl container is crashing. This results in sonarqube not deploying.
This is the output of “kubectl logs pods/sonarqube-sonarqube-0 -c init-sysctl” :

sysctl: permission denied on key "vm.max_map_count", ignoring
sysctl: permission denied on key "fs.file-max", ignoring
ulimit -n 131072
/tmp/scripts/init_sysctl.sh: line 10: ulimit: open files: cannot modify limit: Operation not permitted

I’ve tried also to run the newest version, but unfortunatelly it didn’t work.
I am not sure what can I do about this, I’ve tried to use custom values and allow it to run as root, but this seemed not to solve this problem.

What can I do about this?
I would appreciate any help.

1 Like

I have the same problem but in the latest version, 10.3.0 on a paid enterprise license. For many versions I have updated Sonar via the helm chart, but all of a sudden (without any upgrade) the azure aks instance restarted and now the vm.max_map_count does not seem to be applied. I cannot tell what is going on, but I get the same error as described above when looking at logs in that same pod/container. When I look at the “sonarqube” container the logs indicate the following:

Blockquote bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

My helm chart mimics this one, where you can see that the vmmaxmapcount is set appropriately: https://github.com/SonarSource/helm-chart-sonarqube/blob/e7176ff10bd4cb753020ef93d19cfbd3e0c93a99/charts/sonarqube/values.yaml#L238

Sonar is not starting up. We are several days into the problem, a few people’s hours drained and we are not sure what to look at next. Help appreciated.

1 Like

Hi,

Welcome to the community and thanks for these reports!

Can you try setting vm.max_map_count and file descriptors for each node before deploying the SonarQube Helm chart?

 
Thx,
Ann

1 Like

Sure, I’ve found a workaround for setting the vm.max_map_count with this:

extraInitContainers: 
- name: init
  image: busybox
  command:
  - sysctl
  - -w
  - vm.max_map_count=524288
  imagePullPolicy: IfNotPresent
  securityContext:
    privileged: true

Tried this to set the ulimit also:

- name: increase-fd-ulimit
  image: busybox
  command: 
  - sysctl
  - -w
  - fs.file-max=65535
  securityContext:
    privileged: true

But this doesn’t change anything

Hi,

Thanks for trying. I’ve flagged this for more expert eyes.

 
Ann

Our production 10.3.0 Enterprise instance went down this morning with the same issue, and we are currently down trying to resolve the issue.

The confusing part is that the upgrade from 10.2.0 went smoothly and didn’t see this behavior. My best guess is that the node used to run the 10.3.0 SonarQube instance was the same node, and the sysctl value was already applied from the previous running 10.2.0 instance.

I have mitigated this issue by rolling back the Helm chart version and pinning the image tag to 10.3.0. I realize this is an unsupported deployment method, nevertheless it is the only way I was able to get our systems operational in a reasonable time frame.

I added these to my custom terraform:

################################################################
# Workaround startup issue in 10.3.0 helm chart

  set {
    name  = "appVersion"
    value = "10.3.0"
  }

  set {
    name  = "image.tag"
    value = "10.3.0-enterprise"
  }

################################################################

This translates into these user-supplied values:

appVersion: 10.3.0
image:
  tag: 10.3.0-enterprise

For the SonarSource developer(s) working on this issue, think this is the bug:

It looks like the initContainers were suppose to have the ability to escalate their permissions so they could apply the sysctl values. I haven’t the time right now to work through this, but I hope this helps find the issue.

There also is another bug with the implementation of the init_sysctl.sh script. Since there is no set -e in the script, and none of the commands have || exit $? the entire script return code will always be the last command. Since the last command is a ulimit command, it’s pretty much guaranteed to always have a 0 exit status.

On a related note, I don’t believe that ulimit -u commands persist outside the container from which they are executed, and these may need to be moved back into the SonarQube container before ElasticSearch is invoked. Since ulimit is a not a privileged command, this should be safe to do.

I hope this helps!

P.S.
The Delete message button pops up right under the hamburger and doesn’t try to confirm, and I keep accidentally deleting my posts! :man_facepalming:

2 Likes

Hi @RyanH,

Thanks for sharing what worked for you.

We’re planning to update the docs on this topic soon, and your input is helpful.

 
Thx,
Ann

1 Like

Any news about this error?
Sonar: 10.3.0-community
Helm: 10.3.0+2009

@danilodorgam It looks like a fix was merged to grant the init containers root permissions, but I haven’t tried it myself yet. There may still be some issues with the ulimit persistence for Elasticsearch as I didn’t notice any merged changes around that code, so you may want to validate that works properly to be safe.

2 Likes

How can I test this merged fix? Will it be release sometime soon?

Hi, @stephanlv_sanitas.

If you clone the repository, you can use it as helm install sonarqube ./charts/sonarqube/ [...].

You can subscribe to our Releases channel to be notified of our releases.

1 Like

Hi @stephanlv_sanitas.

We released a new version of our Helm Chart 2024-02-05T23:00:00Z. You can check our Changelog here.

1 Like