ReadinessProbe timeout using Helm chart version 8.0.0+463

I’m attempting to deploy the sonarqube 8.0.0+463 helm chart into an OpenShift cluster and it’s failing due to the Readiness probe. After some investigation, I’ve uncovered the following.

The ReadinessProbe executes a wget to the api system status and includes the “–proxy off” parameter. As far as I can tell, that “–proxy” parameter isn’t supported by wget and hasn’t been for a long time. That hasn’t been an issue with previous versions (< SQ 9.9) because the SonarQube container was using “Alpine Linux” as its OS and the version of wget included would ignore the “–proxy off” parameter. Now that the latest SQ 9.9 version has switched to Ubuntu, it’s using a different version of wget. When wget is now executed with the “–proxy off” parameter, wget attempts to resolve the hostname “off”, fails, and then resolves the API URL. The issue with our setup is that the DNS timeout for “off” takes over a second as it recursively checks DNS names, and by the time it retrieves the API system status result (which is UP), the ReadinessProbe timeout kicks in at 1 sec and thus exits 1.

It seems the best way to deal with this is to change the “–proxy off” parameter to the expected parameter of “–no-proxy”.

If for whatever reason that isn’t feasible, then expose the ReadinessProbe timeout value to the chart so that it can be increased to allow for a failed resolving of the hostname “off”.

Hopefully this makes sense and can be verified internally.

Thanks!

2 Likes

Dear @nflohr,

thanks for reporting this issue. We were actually able to replicate it and defined a bug-fix (SQHELM-118) task to tackle it :slight_smile:

You say that you are installing our chart on Openshift: that’s great! We do actually not support Openshift yet, but we would love to gain feedback from users that are deploying SQ on it successfully with the official chart. Can you tell us which were the (extra) steps that you had to perform in order the have our chart successfully deployed? Did you need to act on some security constraints (SCC) of your cluster?

Cheers,
Carmine

6 Likes

Hello Carmine,

We are also using the sonarsource helm chart in our Openshift environment.
And it’s running fine.
We had some trouble with the SecurityContextConstraint which 2 initContainers needed.
( init-fs and init-sysctrl ) Our team made a ‘temporary’ SCC in our gitops-management environment.
We are stil looking for an alternative.

And with the upgrade to LTS 9.9 ( from 9.7.1 ) we had some trouble with the install-plugins init container. Something(?) went wrong when deleting the present plugins.
At the restart of the new LTS 9.9 environment, it complaned that there were two versions of the AzureAD plugin, so the sonarqube pod didn’t start.
We had to remove the physical volume to get it up and running again.
I clould see that the rights on the extension directory are different.

The rights in 9.7.1 are:
drwxr-sr-x. 2 sonarqube sonarqube 4096 Mar 7 09:09 downloads
drwxrwsr-x. 2 root sonarqube 4096 Mar 7 09:08 plugins

and in 9.9 LTS it is:

drwx------ 2 sonarqub sonarqub 4096 Feb 9 15:28 downloads
drwx------ 2 sonarqub sonarqub 4096 Feb 9 15:28 plugins

Question?
Why don’t you follow the releases of sonarqube 9.11.0 · oteemo/oteemo-charts. The last version on GitHub - SonarSource/helm-chart-sonarqube is 8.0.0. There are missing components in it. Like the yaml(s) for the prometheus podmonitor.

1 Like

hello @meedel, thanks a lot for that feedback, it is really valuable for us!

Regarding the SecurityContextConstraint we are aware it is not the optimal solution, and we will work on improving that in the future (no current date).

On the plugin deletion topic, we have an ongoing Pull Request that should fix the wrong behavior you had.

Nonetheless, currently we only release a new version of the Helm Chart when a new SonarQube version is available.

For the podMonitor, that feature was added recently to our chart and will be available on the next Release.

On top of that the oteemo chart is now deprecated, please always refer to the official one you were linking :slight_smile:

Jérémy.

Hi Jeremy,

Thanks for your reply.
I wil keep you posted if we find some other improvements. :slight_smile:
And we wil wait for the next release to use the podMonitor.

Bert

Thanks for your feedback, @meedel!

@nflohr before we mark the discussion post as solved (we are gonna tackle your suggested fix before the next release, thanks again for reporting it), do you also have time to give us feedback on the following subject?

Hi @carmine.vassallo, appreciate your team providing a fix for the chart!

As far as our OpenShift deployment, we had to set the following for a successful deployment:

Postgresql security and hugepages:

postgresql:
      securityContext:
        enabled: false
      containerSecurityContext:
        enabled: false
      serviceAccount:
        enabled: true
        name: postgresql
      volumePermissions:
        enabled: true
      primary:
        extraVolumeMounts:
          - mountPath: /dev/hugepages
            name: hugepage
        extraVolumes:
          - name: hugepage
            emptyDir:
              medium: HugePages

Service account:

serviceAccount:
      name: sonarqube

As a side note, for using a ServiceMonitor (instead of a PodMonitor), I had to use postRenderers to add the Prometheus Exporter ports to the Service. It would be a nice enhancement to allow configuring of more ports for the Service.
postRenderers:
    - kustomize:
        patches:
          - patch: |-
              - op: add
                path: "/spec/ports/-"
                value:
                  name: "jmx-metrics-web"
                  port: 8000
                  protocol: "TCP"
                  targetPort: "monitoring-web"
            target:
              kind: Service
              labelSelector: "app=sonarqube"
          - patch: |-
              - op: add
                path: "/spec/ports/-"
                value:
                  name: "jmx-metrics-ce"
                  port: 8001
                  protocol: "TCP"
                  targetPort: "monitoring-ce"
            target:
              kind: Service
              labelSelector: "app=sonarqube"

Hope this help!
-Nick

Thanks a lot for your feedback too, @nflohr! Your insights are very valuable.

@jeremy.cotineau fixed the above issue in this PR, thanks again to everyone involved in the discussion.

Hello @carmine.vassallo ,

only a quick question: Will this fix also be included in the chart for the 9.9 LTS version?

thank you

Hi @christiang,

thanks for your question. At the moment, we don’t plan to backport the fix. Please consider using the latest helm charts on ArtifactHUB.

Hello Jeremy,

Regarding the SecurityContextConstraint for the init-fs and init-sysctrl ( deploying with sonarsource helm chart in Openshift ) !
Has a schedule already been issued when this improvement will be implemented?

Bert

Hello @meedel, as we still dont support Openshift an improvement is not on the pipeline.

Nonetheless for most use cases, and even more for production, those two feature/container (init-sysctl and init-fs) should be deactivated at the chart level.

You can find some explanation here

To summarize:

  • the init-fs should not be needed if you have an up to date CSI-Driver.
  • the init-sysctl should be setup at the node level, because it change kernel parameter, this is too big to be allowed by a container. it should be setup by your openshift admins, and maybe provided on few nodes you can stick to with toleration and so on.
1 Like

Thank for you quick reply Jeremy.
I wil share this info with our team, and implement your reccomendations

If you find some hiccups, or have any other questions, do not hesitate to follow this thread or post a new one.

Even if we do not support officially Openshift, we can make overall improvement that benefits to Openshift users !