GitLab MR decoration fails

[Jacob_Williams]

Versions:

• SonarQube: 9.9.0.65466 (Developer edition)
• Scanner: Docker image: sonarsource/sonar-scanner-cli:4.8.0

Deployment:

Helm chart version: 8.0.0
SonarQube is deployed in GKE (Google Kubernetes) behind a proxy (IAP) and communicates with our self-hosted GitLab deployment using an http kubernetes service URL to avoid needing to authenticate with the IAP proxy.

We have configured the GitLab devops integration using the kubernetes service URL (internal to the cluster) and every project in our deployment is configured from a corresponding project in GitLab.

Trying:

Decorate GitLab MRs with analysis results after running SonarScanner in a GitLab CI pipeline.

Issue:

Most of the time, there is no issue and everything works fine, however, sometimes sonar scanner tries decorating an MR using the public URL for the gitlab server. Since it cannot authenticate with the IAP proxy over the public URL, this fails.

Relevant logs:

2023.04.05 17:51:07 INFO  ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.B] GitLab's instance URL from the scanner ('https://gitlab.freenome.net/api/v4') is overridden by the settings ('http://gitlab-webservice-default.gitlab.svc:8080/api/v4')
2023.04.05 17:51:07 INFO  ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.B] GitLab's project ID from the scanner ('214') is overridden by the settings ('214')
2023.04.05 17:51:07 ERROR ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.A] An exception was thrown during Merge Request decoration : unexpected end of stream on http://gitlab.freenome.net:443/...
2023.04.05 17:51:07 ERROR ce[AYdSiyhEZl0SwYh0tgbM][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Execution of task class com.sonarsource.F.D.d failed
java.lang.IllegalStateException: unexpected end of stream on http://gitlab.freenome.net:443/...
...
Caused by: java.io.EOFException: \n not found: limit=0 content=…
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:332)
    at okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
    at okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:178)
    ... 46 common frames omitted

As you can see, from the logs it says that the settings are overridden and it should be using the kube service URL, however, the error says it’s using the public URL.

What I’ve tried

• Removing the GitLab CI environment variables that point to the public GitLab URL
• Modifying the GitLab CI environment variables to point to the non-public GitLab URL

[Colin]

Hey @Jacob_Williams

Sorry for the delay on this thread.

I just wanted to check in on this thread since I saw a lot of work over on this thread:

Sonar Scanner uses the wrong link in pipeline output

Thanks for these extra details @Jacob_Williams. Now I manage to reproduce your behavior. If I set the value of sonar.core.serverBaseURL in my sonar.properties file and not in the UI, I do see in the logs, the same value as SONAR_HOST_URL, while in the MR decoration I see the value of sonar.core.serverBaseURL being used for completing the rules. I can say that for fixing your issue you can add in the Administration panel of your instance the same value as the one available in your values.yml f…

Are you still facing the problem described in this thread?

[Jacob_Williams]

Hey @Colin, yes we are still seeing this same issue even after resolving the link issue in GitLab pipelines.

[Colin]

Thanks.

First of all, I’m pretty surprised that these log lines exist:

@Jacob_Williams

2023.04.05 17:51:07 INFO  ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.B] GitLab's instance URL from the scanner ('https://gitlab.freenome.net/api/v4') is overridden by the settings ('http://gitlab-webservice-default.gitlab.svc:8080/api/v4')
2023.04.05 17:51:07 INFO  ce[AYdSiyhEZl0SwYh0tgbM][c.s.F.D.F.B] GitLab's project ID from the scanner ('214') is overridden by the settings ('214')

This implies sonar.pullrequest.gitlab.instanceUrl and sonar.pullrequest.gitlab.projectId being configured manually somewhere… and these have really never been publicly exposed analysis parameters.

I know this doesn’t seem to exactly relate to your problem, given what the expected URL should be, but I’d really like to know where this is coming from. Are you setting this directly, somehow?

[Jacob_Williams]

No we are not directly setting those anywhere.

All projects in our GitLab instance are managed/created via terraform IaC. We likewise are also creating our corresponding SonarQube projects as well as configuring the GitLab integration for each project with terraform.

The SonarQube terraform provider is just making API calls in the backend. Mainly: /alm_settings/create_gitlab to configure the instance-wide GitLab setup and then /alm_settings/set_gitlab_binding on each of the projects (after using /projects/create to create each project).

EDIT: Used the public API documentation links instead of our private instance’s web docs

[Colin]

Thanks.

On an affected PR, can you look at the background task (Project Settings > Background Tasks) and open up the Scanner Context, and see if you find the unexpected values for sonar.pullrequest.gitlab.instanceUrl (https://gitlab.freenome.net/api/v4)

It should give some context about where they are coming from too: scanner, server, project…

[Jacob_Williams]

Hey @Colin I just checked on this and I do not see any sonar.pullrequest.* values anywhere in the context. The only reference to gitlab at all was the sonar.core.serverBaseURL(https://sonarqube-gitlab.freenome.net)

[Colin]

Hm. That’s really odd. Are you sure you’re checking the Scanner Context for a pull request analysis?

[Jacob_Williams]

@Colin yeah I’m positive. I’ve just double checked on a different task for a different project. The last analysis came from a GitLab MR pipeline. The analysis succeeded, but no MR decoration occurred and there are no sonar.pullrequest properties in the Scanner Context. The analysis in SonarQube has the following warning as well:

[Jacob_Williams]

I just assumed a bunch of the MR data was grabbed from the environment since GitLab sets a bunch of CI_MERGE_REQUEST_* environment variables for those pipelines.

[Colin]

Okay. I just tested this and (to be honest, I’m a little dumbstruck) these properties don’t appear in the scanner context.

For the record:

• GITLAB_CI is used to determine that it’s running in a GitLab CI environment
• CI_MERGE_REQUEST_IID gjves sonar.pullrequest.key
• CI_MERGE_REQUEST_SOURCE_BRANCH_NAME gives sonar.pullrequest.target
• CI_MERGE_REQUEST_TARGET_BRANCH_NAME gives sonar.pullrequest.base

Back to the problem at hand, digging around there are a few more environment variables we should check on (which are no longer used at all in SonarQube v10.0, so if upgrading is a possibility )

I think the relevant ones are:

• CI_API_V4_URL (probably relevant)
• CI_PROJECT_ID (probably not relevant)
• CI_MERGE_REQUEST_PROJECT_URL (probably not relevant)

Can you check how CI_API_V4_URL is being set in the context of your Gitlab pipelines when you have the merge requests with failed decoration? Is it hitting the IAP?

[Jacob_Williams]

Yeah I already know that the CI_API_V4_URL points to the IAP URL. In the “What I’ve Tried” section I mentioned how I overwrote all the GitLab CI variables in the pipeline to point to the kubernetes service URL rather than the IAP URL.

Since then we have upgraded to SonarQube v10.0 (on May 9th). My apologies for not mentioning that earlier, that probably would have been useful for you to know . I haven’t retried my test of rewriting the environment variables in the pipeline, but I could if you think that might be worthwhile?

[Colin]

Thanks for the update.

If you’re on SonarQube v10.0, then we can scratch out anything related to CI_API_V4_URL because this should no longer be used anywhere.

I’m starting to grasp at straws here, but here are a few things to consider:

@Jacob_Williams

The SonarQube terraform provider is just making API calls in the backend. Mainly: /alm_settings/create_gitlab to configure the instance-wide GitLab setup

This feels vaguely suspicious – like if there is another terraform provider (perhaps that is supposed to be targeting a test instance or something like that) that is somehow adjusting this setting before being snapped back into place by the “real” one. If you are able to check the integration configuration right after a merge request decoration fails, it might point to this/rule it out.

After that, I start to look away from the SonarQube configuration and consider if something is happening when the request is made that points to the wrong server (DNS caching, for example). You could try to set the DNS cache to… not cache, and see if it makes a difference.

echo "networkaddress.cache.ttl=30" >> "${JAVA_HOME}/conf/security/java.security"

[Jacob_Williams]

@Colin

This feels vaguely suspicious – like if there is another terraform provider (perhaps that is supposed to be targeting a test instance or something like that) that is somehow adjusting this setting before being snapped back into place by the “real” one. If you are able to check the integration configuration right after a merge request decoration fails, it might point to this/rule it out.

This is highly unlikely, given how our terraform system is deployed where test/prod are completely isolated from each other (separate k8s clusters, contexts, gitlab/sonarqube user accounts, etc). It would be nearly impossible for a config to be pointed at the wrong instance.

To be safe, I did walk through this exercise to ensure we hadn’t overlooked anything.

@Colin

After that, I start to look away from the SonarQube configuration and consider if something is happening when the request is made that points to the wrong server (DNS caching, for example). You could try to set the DNS cache to… not cache, and see if it makes a difference.

This is interesting…I’ll have to dig more to see if DNS could be the problem