Sonarqube-8.4 [DevEdition] - MR/Decoration - fails to validate hostname (dns caching)

SonarQube
1

Our gitlab runs on EC2 instance. It is well known that AWS changes (from time to time) public IPs allocated to LBs that are in front of EC2 instances.
Problem here is that sonarqube seems to rely on previously cached IP (which no longer would point to LB in front of Gitlab) and therefore is not able to validate the hostmane (mismatch between hostname vs certificate). Can either dns caching be disabled (work-around) or better yet (feature request/fix) upon not being able to verify the host/certificate then sonar could flush its dns cache and do a lookup of the passed hostname towards public dns?

An exception was thrown during Merge Request decoration : Hostname gitlab.infrastructure.test.com not verified:
certificate: sha256/XXXXXXXXXXXX
DN: CN=demo.notgitlab.com

1 Like
2

hostname, certificate and CN => redacted! Though (and as expected), there is no match due to the public IP re-allocation that AWS does from time to time.

4

The DNS record for our gitlab has TTL of 60 seconds only. Therefore, even after AWS changed public IP of LB (in front of our gitlab), sonar should have eventually done a lookup though apparently did not happen. The below is experienced quite a few times within a 2-3 hours window…

2020.09.04 00:17:04 ERROR ce[AXRWeUnd5GAGCf4MOaRp][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 00:35:08 ERROR ce[AXRWic3r5GAGCf4MOaRt][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 00:50:08 ERROR ce[AXRWl4qJ5GAGCf4MOaRw][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 01:06:33 ERROR ce[AXRWppmP5GAGCf4MOaR1][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 01:38:24 ERROR ce[AXRWw7wf5GAGCf4MOaR4][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 02:06:40 ERROR ce[AXRW3aB05GAGCf4MOaSp][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:

It seems that Sonar process/libraries/container/VM host isn’t treating TTL properly. Case here, DNS TTL for gitlab record is only 60 seconds

5

https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-jvm-ttl.html

6

one idea here, please comment, would be to add file java.security to define -> networkaddress.cache.ttl=60

and place this file here…thoughts?
opt/java/openjdk/lib/security # ls -ltr
total 352
-rw-r–r-- 1 root root 233902 Jan 15 2020 public_suffix_list.dat
-rw-r–r-- 1 root root 8911 Jan 15 2020 default.policy
-rw-r–r-- 1 root root 103147 Jan 15 2020 cacerts
-rw-r–r-- 1 root root 1253 Jan 15 2020 blacklisted.certs

7

Hello @peppe1977,

Yes indeed, setting this property could help in your case. Although keep in mind that setting it in java directory means every application on that server will have this setting set.

8

Hi @jacek.poreda, thank you! In our case it should be fine (container/pod running on k8s). question here is - would not be better to have that java property set directly on the provided image for community, dev edition and so on? As of now (property does not exist), therefore DNS TTL == forever. We are looking into upstream repo that we use (manage sonar/dev edition) as to find a way to set the property for now though still believe that perhaps image should have that already set. thoughts?

9

Also, it seems to be a bug - sorry that did not notice the below before - the properties have already been set (I did not have to do it), therefore this is not working as it should be. If you scroll up, you will see that within 2 hours time-window, sonar was still using its cached values :frowning:

Here is the printout showing sonar already sets the suggested property:

18 sonarqub 0:25 java -jar lib/sonar-application-8.4.1.35646.jar -Dsonar.log.console=true -Dsonar.jdbc.username=XXXXXXXXX -Dsonar.jdbc.password=XXXXXXXXXXXXX -Dsonar.jdbc.url=jdbc:postgresql://infrastructure-postgresql.cluster-c
43 sonarqub 1:11 /opt/java/openjdk/bin/java -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouc
119 sonarqub 1:28 /opt/java/openjdk/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang
169 sonarqub 1:29 /opt/java/openjdk/bin/java -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djava.io.tmpdir=/opt/sonarqube/temp -XX:-OmitStackTraceInFastThrow --add-opens=java.base/java.util=ALL-UNNAMED -Xmx512m -Xms128m -XX:+HeapDump

Here that it does not work as it should:
2020.09.04 00:17:04 ERROR ce[AXRWeUnd5GAGCf4MOaRp][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 00:35:08 ERROR ce[AXRWic3r5GAGCf4MOaRt][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 00:50:08 ERROR ce[AXRWl4qJ5GAGCf4MOaRw][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 01:06:33 ERROR ce[AXRWppmP5GAGCf4MOaR1][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 01:38:24 ERROR ce[AXRWw7wf5GAGCf4MOaR4][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:
2020.09.04 02:06:40 ERROR ce[AXRW3aB05GAGCf4MOaSp][c.s.C.D.D.B] An exception was thrown during Merge Request decoration : Hostname gitlab.cdls.infrastructure.test.com not verified:

Please advise on how to address this dns ttl that though set, does not seem to be working. thanks

10

The setting you are referring to is es.networkaddress.cache.ttl and it is set for Elasticsearch process only.

Web process does not define such property, did you give it a try?

11

Hi @jacek.poreda , thanks.

We use terraform/helm(provider) to manage it. We were able to add JVM_OPTS to the .tpl and pass the property. Will monitor to ensure that next time public ip changes, sonar can properly handle that. (still think this property should be set in the image though). Cheers /Pedro

19 sonarqub 0:06 java -jar lib/sonar-application-8.4.2.36762.jar -Dsonar.log.console=true -Dsonar.jdbc.username=sonarqube-de -Dsonar.jdbc.password=XXXXXXXXXX -Dsonar.jdbc.url=jdbc:postgresql:/XXXXXXXXXXXX/sonarqube-de -Dsonar.web.javaAdditionalOpts=-Dnetworkaddress.cache.ttls=40

1 Like