Helm chart version: 8.0.0
SonarQube is deployed in GKE (Google Kubernetes) behind a proxy (IAP) and communicates with our self-hosted GitLab deployment using an http kubernetes service URL to avoid needing to authenticate with the IAP proxy.
We have configured the GitLab devops integration using the kubernetes service URL (internal to the cluster) and every project in our deployment is configured from a corresponding project in GitLab.
Trying:
Decorate GitLab MRs with analysis results after running SonarScanner in a GitLab CI pipeline.
Issue:
Most of the time, there is no issue and everything works fine, however, sometimes sonar scanner tries decorating an MR using the public URL for the gitlab server. Since it cannot authenticate with the IAP proxy over the public URL, this fails.
Relevant logs:
2023.04.05 17:51:07 INFO ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.B] GitLab's instance URL from the scanner ('https://gitlab.freenome.net/api/v4') is overridden by the settings ('http://gitlab-webservice-default.gitlab.svc:8080/api/v4')
2023.04.05 17:51:07 INFO ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.B] GitLab's project ID from the scanner ('214') is overridden by the settings ('214')
2023.04.05 17:51:07 ERROR ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.A] An exception was thrown during Merge Request decoration : unexpected end of stream on http://gitlab.freenome.net:443/...
2023.04.05 17:51:07 ERROR ce[AYdSiyhEZl0SwYh0tgbM][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Execution of task class com.sonarsource.F.D.d failed
java.lang.IllegalStateException: unexpected end of stream on http://gitlab.freenome.net:443/...
...
Caused by: java.io.EOFException: \n not found: limit=0 content=…
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:332)
at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178)
... 46 common frames omitted
As you can see, from the logs it says that the settings are overridden and it should be using the kube service URL, however, the error says it’s using the public URL.
What I’ve tried
Removing the GitLab CI environment variables that point to the public GitLab URL
Modifying the GitLab CI environment variables to point to the non-public GitLab URL
First of all, I’m pretty surprised that these log lines exist:
This implies sonar.pullrequest.gitlab.instanceUrl and sonar.pullrequest.gitlab.projectId being configured manually somewhere… and these have really never been publicly exposed analysis parameters.
I know this doesn’t seem to exactly relate to your problem, given what the expected URL should be, but I’d really like to know where this is coming from. Are you setting this directly, somehow?
All projects in our GitLab instance are managed/created via terraform IaC. We likewise are also creating our corresponding SonarQube projects as well as configuring the GitLab integration for each project with terraform.
On an affected PR, can you look at the background task (Project Settings > Background Tasks) and open up the Scanner Context, and see if you find the unexpected values for sonar.pullrequest.gitlab.instanceUrl (https://gitlab.freenome.net/api/v4)
It should give some context about where they are coming from too: scanner, server, project…
Hey @Colin I just checked on this and I do not see any sonar.pullrequest.* values anywhere in the context. The only reference to gitlab at all was the sonar.core.serverBaseURL(https://sonarqube-gitlab.freenome.net)
@Colin yeah I’m positive. I’ve just double checked on a different task for a different project. The last analysis came from a GitLab MR pipeline. The analysis succeeded, but no MR decoration occurred and there are no sonar.pullrequest properties in the Scanner Context. The analysis in SonarQube has the following warning as well:
I just assumed a bunch of the MR data was grabbed from the environment since GitLab sets a bunch of CI_MERGE_REQUEST_* environment variables for those pipelines.
Back to the problem at hand, digging around there are a few more environment variables we should check on (which are no longer used at all in SonarQube v10.0, so if upgrading is a possibility )
I think the relevant ones are:
CI_API_V4_URL (probably relevant)
CI_PROJECT_ID (probably not relevant)
CI_MERGE_REQUEST_PROJECT_URL (probably not relevant)
Can you check how CI_API_V4_URL is being set in the context of your Gitlab pipelines when you have the merge requests with failed decoration? Is it hitting the IAP?
Yeah I already know that the CI_API_V4_URL points to the IAP URL. In the “What I’ve Tried” section I mentioned how I overwrote all the GitLab CI variables in the pipeline to point to the kubernetes service URL rather than the IAP URL.
Since then we have upgraded to SonarQube v10.0 (on May 9th). My apologies for not mentioning that earlier, that probably would have been useful for you to know . I haven’t retried my test of rewriting the environment variables in the pipeline, but I could if you think that might be worthwhile?
If you’re on SonarQube v10.0, then we can scratch out anything related to CI_API_V4_URL because this should no longer be used anywhere.
I’m starting to grasp at straws here, but here are a few things to consider:
This feels vaguely suspicious – like if there is another terraform provider (perhaps that is supposed to be targeting a test instance or something like that) that is somehow adjusting this setting before being snapped back into place by the “real” one. If you are able to check the integration configuration right after a merge request decoration fails, it might point to this/rule it out.
After that, I start to look away from the SonarQube configuration and consider if something is happening when the request is made that points to the wrong server (DNS caching, for example). You could try to set the DNS cache to… not cache, and see if it makes a difference.
)
. I haven’t retried my test of rewriting the environment variables in the pipeline, but I could if you think that might be worthwhile?