SonarQube Upgrade breaking - failed to assign an IP address to container

SonarQube Server / Community Build
1

Must-share information (formatted with Markdown):

  • sonarqube-10.3.0+2009 App version 10.3.0
  • Helm deployment
  • Upgrading SonarQube from sonarqube-10.2.1+800 to 10.3.0+2009
  • Change the Chart version in the values.yaml

The container events show the following:

  • │ Warning FailedCreatePodSandBox 2m9s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “f5255f8e1d52301ca522ae38e50f131c51bc4db3ddb0173a8a7761 │
    │ 8a73ec7b2f”: plugin type=“aws-cni” name=“aws-cni” failed (add): add cmd: failed to assign an IP address to container

  • Describing the pod shows following error for the init Container:

  init-sysctl:
    Container ID:  containerd://2314df2a4f77e8954569b3780fcd44a5609d11e6587e484c7a2b970c87c4f509
    Image:         containers.gus.bsf.tools/busybox:1.32
    Image ID:      containers.gus.bsf.tools/busybox@sha256:ae39a6f5c07297d7ab64dbd4f82c77c874cc6a94cea29fdec309d0992574b4f7
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -e
      /tmp/scripts/init_sysctl.sh
    State:          Waiting
      Reason:       RunContainerError
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown
      Exit Code:    128
2

Looks like we found the culprit or at least what fixed it for us. Bug?

We reverted this change back to sh

image
image757×98 2.75 KB

1 Like
5

Hi @samir.causevic,

welcome to our community! Thanks for posting your issue, I’ll try to give you some hints about the solution.

In the 10.3.0 release of the Helm chart, we changed the default image used by our initContainers. For example, in the case of init-sysctl, we moved from using the busybox image to our own (i.e., sonarqube). This is because we realized that multiple customers (like you, probably) need to scan and copy the images from DockerHub to their internal registries. Relying on one image in our helm chart will speed up this approval and copy process!

Coming back to your problem, the solution would be to set the SonarQube image to be the one used in the init-sysctl init container. You can achieve that by setting something like this in your values.yaml file

initSysctl:
  enabled: true
  image: containers.gus.bsf.tools/sonarqube:10.3.0-<edition>

Could you please try and give us feedback?

p.s.
You might need to do the same for the other init containers, setting a similar value in initContainers.image.

1 Like
7

I can not understand the solution. We copy dozens of images to our private registry. It does not matter to copy one more. Specially busybox is almost there and is used by many helm charts as an init container.

To use the sonarqube image as an init container is more than overkill.

9

Dear @totobaa,

Thanks for your feedback. Let me try to clarify once again why we opted for using the SonarQube image also for the initContainers:

  • Quite a few companies need to undergo an internal process for getting approved even minor updates of the image. Example: if a SonarQube 10.6 version is released, they need to get a new approval for having it in the private registry. It doesn’t matter if they got a previous approval for SQ 10.5. Therefore, having to maintain the updates of one image only in our chart translates into a big time saving for them.
  • Reducing the need for multiple images can translate into a more efficient readiness of our application (e.g., reducing the likelihood a pod initialization gets stuck while pulling new images).

Please let us know if you have strong arguments against this.

1 Like
10

Hi @carmine.vassallo ,

to use the sonarqube image as an init container is against the least privilege principal and is a potential security risk. The init containers have an other pod security policy than the application container.

I have never saw a other helm chart which uses a 500MB large image for an init container.

Maybe it is possible to allow the setting of command attribute in the helm chart for the init containers. So we can change it to /bin/sh and use the busybox image.

Regards

Torben

11

Hi again @totobaa,

Thanks for your constructive feedback!

I understand your first point. However, please consider that a few default init containers need to run in a privileged mode, for example to set a kernel parameter required by our Elasticsearch dependency. If you need to adhere to stricter security policies, you can still deploy our helm chart and comply with the restricted Pod Security level. In that case, you would need to setup the nodes by yourself before deploying the Helm chart (more info here).

True, the image is bigger than a usual init container. However, once you pull it into the init face, you won’t need to pull it again when deploying the actual container.

Maybe it is possible to allow the setting of command attribute in the helm chart for the init containers. So we can change it to /bin/sh and use the busybox image.

We allow users to set their own init container image but our chart is less flexible on the actual shell program. I just defined an improvement task ([SONAR-22158] - Jira) where we will tackle your suggestion in one of the next hardening sprints. Hopefully this will enable you to use busybox again soon :slight_smile: