Gitlab MR decoration broken if Gitlab IP changes on same hostname

SonarQube
1

Versions and context

  • Gitlab self-hosted 12.7.6
  • Sonarqube self-hosted 8.2.0.32929
  • Gitlab and sonar installed through docker in AWS ECS cluster behind load-balancers

Error observed

MR decoration is set up in Gitlab and Sonarqube and working well.

After certain (long) time, MR decoration stops working.

image
image774×341 14.7 KB

Compute engine log shows errors like :

Caused by: javax.net.ssl.SSLPeerUnverifiedException: Hostname my.gitlab.url not verified:
    certificate: sha256/JYP42CCMuLi87AC6qUOOfaICgr819itewHstb1W+2cA=
    DN: CN=*.a_totally_different_url_i_dont_know_anything_about
    subjectAltNames: [*.a_totally_different_url_i_dont_know_anything_about]
  • It seems that the ip of my.gitlab.url has changed.
  • It seems then that this old ip has been reassigned to a_totally_different_url_i_dont_know_anything_about.
  • It seems that sonar is still using the old ip and then tries to interrogate a_totally_different_url_i_dont_know_anything_about.

Steps to reproduce

  • Install a sonar and gitlab
  • Configure MR decoration
  • Change public ip of gitlab
  • Update IP in DNS
  • Try an analyze
  • Observe that decoration is broken
  • Wait expiration of DNS configured TTL
  • Observe that decoration is still broken

Workaround

Restart SonarQube from administration module / system information page.

It enforces a DNS interrogation

TL;DR

If DNS entry changes and a new ip is assigned to gitlab, even after DNS TTL expiration, sonar will not re-request the DNS and still use obsolete IP.

2 Likes
3

Hello @florck,

This is more a Java consideration than a purely SQ one. As explained on the JavaDoc (** Address Cache** at the end), results of DNS lookups are cached.
I don’t know the default value, it likely depends on the JVM implementation you use.

The Amazon doc gives good explanations on the topic, and a way to solve the issue.

I hope it will help!

4

Thank you Antoine for your answer. However I’m using the official docker image (https://hub.docker.com/_/sonarqube) and would prefer not to customize it manually.

One other point is that TTL is not fixed here, it varies depending on several elements, so best would be for each DNS request to cache it on the TTL providen by the DNS itself in its answer, the source of trust.

5

Good point indeed. But again, I’m quite sure it’s Java internal stuffs that we are dealing with here, nothing specifically implemented in SQ, so we don’t have control I’d say.

As you use the SQ Docker image, I know which JVM you use, so I’ll try to reproduce and see what could be done. BTW, SQ 8.3.1 docker image had been released, and as we changed the base image (from openjdk:11-jre-slim – to – alpine:3.11 with AdoptOpenJDK jdk-11.0.6+10) so it is possible that this behaves differently. Could you give it a try and trigger an IP change of GitLab on your infra?

6

Sure I will do, thank you !

7

Sorry for the loooong delay. I have not been able to generate an ip change manually, but one just occured automatically, and then I got same issue.

Have you any other idea?

Thank you for your help.