GitLab MR decoration fails

Versions:

  • SonarQube: 9.9.0.65466 (Developer edition)
  • Scanner: Docker image: sonarsource/sonar-scanner-cli:4.8.0

Deployment:

Helm chart version: 8.0.0
SonarQube is deployed in GKE (Google Kubernetes) behind a proxy (IAP) and communicates with our self-hosted GitLab deployment using an http kubernetes service URL to avoid needing to authenticate with the IAP proxy.

We have configured the GitLab devops integration using the kubernetes service URL (internal to the cluster) and every project in our deployment is configured from a corresponding project in GitLab.

Trying:

Decorate GitLab MRs with analysis results after running SonarScanner in a GitLab CI pipeline.

Issue:

Most of the time, there is no issue and everything works fine, however, sometimes sonar scanner tries decorating an MR using the public URL for the gitlab server. Since it cannot authenticate with the IAP proxy over the public URL, this fails.

Relevant logs:

2023.04.05 17:51:07 INFO  ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.B] GitLab's instance URL from the scanner ('https://gitlab.freenome.net/api/v4') is overridden by the settings ('http://gitlab-webservice-default.gitlab.svc:8080/api/v4')
2023.04.05 17:51:07 INFO  ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.B] GitLab's project ID from the scanner ('214') is overridden by the settings ('214')
2023.04.05 17:51:07 ERROR ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.A] An exception was thrown during Merge Request decoration : unexpected end of stream on http://gitlab.freenome.net:443/...
2023.04.05 17:51:07 ERROR ce[AYdSiyhEZl0SwYh0tgbM][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Execution of task class com.sonarsource.F.D.d failed
java.lang.IllegalStateException: unexpected end of stream on http://gitlab.freenome.net:443/...
...
Caused by: java.io.EOFException: \n not found: limit=0 content=…
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:332)
    at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
    at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178)
    ... 46 common frames omitted

As you can see, from the logs it says that the settings are overridden and it should be using the kube service URL, however, the error says it’s using the public URL.

What I’ve tried

  • Removing the GitLab CI environment variables that point to the public GitLab URL
  • Modifying the GitLab CI environment variables to point to the non-public GitLab URL

Hey @Jacob_Williams

Sorry for the delay on this thread.

I just wanted to check in on this thread since I saw a lot of work over on this thread:

Are you still facing the problem described in this thread?

Hey @Colin, yes we are still seeing this same issue even after resolving the link issue in GitLab pipelines.

Thanks.

First of all, I’m pretty surprised that these log lines exist:

This implies sonar.pullrequest.gitlab.instanceUrl and sonar.pullrequest.gitlab.projectId being configured manually somewhere… and these have really never been publicly exposed analysis parameters.

I know this doesn’t seem to exactly relate to your problem, given what the expected URL should be, but I’d really like to know where this is coming from. Are you setting this directly, somehow?

No we are not directly setting those anywhere.

All projects in our GitLab instance are managed/created via terraform IaC. We likewise are also creating our corresponding SonarQube projects as well as configuring the GitLab integration for each project with terraform.

The SonarQube terraform provider is just making API calls in the backend. Mainly: /alm_settings/create_gitlab to configure the instance-wide GitLab setup and then /alm_settings/set_gitlab_binding on each of the projects (after using /projects/create to create each project).

EDIT: Used the public API documentation links instead of our private instance’s web docs

Thanks.

On an affected PR, can you look at the background task (Project Settings > Background Tasks) and open up the Scanner Context, and see if you find the unexpected values for sonar.pullrequest.gitlab.instanceUrl (https://gitlab.freenome.net/api/v4)

It should give some context about where they are coming from too: scanner, server, project…

Hey @Colin I just checked on this and I do not see any sonar.pullrequest.* values anywhere in the context. The only reference to gitlab at all was the sonar.core.serverBaseURL(https://sonarqube-gitlab.freenome.net)

Hm. That’s really odd. Are you sure you’re checking the Scanner Context for a pull request analysis?

@Colin yeah I’m positive. I’ve just double checked on a different task for a different project. The last analysis came from a GitLab MR pipeline. The analysis succeeded, but no MR decoration occurred and there are no sonar.pullrequest properties in the Scanner Context. The analysis in SonarQube has the following warning as well:

I just assumed a bunch of the MR data was grabbed from the environment since GitLab sets a bunch of CI_MERGE_REQUEST_* environment variables for those pipelines.

Okay. I just tested this and (to be honest, I’m a little dumbstruck) these properties don’t appear in the scanner context.

For the record:

  • GITLAB_CI is used to determine that it’s running in a GitLab CI environment
  • CI_MERGE_REQUEST_IID gjves sonar.pullrequest.key
  • CI_MERGE_REQUEST_SOURCE_BRANCH_NAME gives sonar.pullrequest.target
  • CI_MERGE_REQUEST_TARGET_BRANCH_NAME gives sonar.pullrequest.base

Back to the problem at hand, digging around there are a few more environment variables we should check on (which are no longer used at all in SonarQube v10.0, so if upgrading is a possibility :innocent:)

I think the relevant ones are:

  • CI_API_V4_URL (probably relevant)
  • CI_PROJECT_ID (probably not relevant)
  • CI_MERGE_REQUEST_PROJECT_URL (probably not relevant)

Can you check how CI_API_V4_URL is being set in the context of your Gitlab pipelines when you have the merge requests with failed decoration? Is it hitting the IAP?

Yeah I already know that the CI_API_V4_URL points to the IAP URL. In the “What I’ve Tried” section I mentioned how I overwrote all the GitLab CI variables in the pipeline to point to the kubernetes service URL rather than the IAP URL.

Since then we have upgraded to SonarQube v10.0 (on May 9th). My apologies for not mentioning that earlier, that probably would have been useful for you to know :sweat_smile:. I haven’t retried my test of rewriting the environment variables in the pipeline, but I could if you think that might be worthwhile?

Thanks for the update.

If you’re on SonarQube v10.0, then we can scratch out anything related to CI_API_V4_URL because this should no longer be used anywhere.

I’m starting to grasp at straws here, but here are a few things to consider:

This feels vaguely suspicious – like if there is another terraform provider (perhaps that is supposed to be targeting a test instance or something like that) that is somehow adjusting this setting before being snapped back into place by the “real” one. If you are able to check the integration configuration right after a merge request decoration fails, it might point to this/rule it out.

After that, I start to look away from the SonarQube configuration and consider if something is happening when the request is made that points to the wrong server (DNS caching, for example). You could try to set the DNS cache to… not cache, and see if it makes a difference.

echo "networkaddress.cache.ttl=30" >> "${JAVA_HOME}/conf/security/java.security"