@ganncamp
Here is information that I believe you were asking for about resources when it happens
bash-5.0$ netstat -np
netstat: showing only processes with your user ID
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:9001 127.0.0.1:60424 ESTABLISHED 25/java
tcp 0 0 10.150.111.84:57474 10.150.1.201:5432 ESTABLISHED 149/java
tcp 0 0 127.0.0.1:37050 127.0.0.1:9001 ESTABLISHED 100/java
tcp 0 0 127.0.0.1:9001 127.0.0.1:37050 ESTABLISHED 25/java
tcp 0 0 10.150.111.84:34926 10.150.1.201:5432 ESTABLISHED 100/java
tcp 0 0 10.150.111.84:46038 10.150.1.201:5432 ESTABLISHED 149/java
tcp 0 0 127.0.0.1:60424 127.0.0.1:9001 ESTABLISHED 100/java
tcp 0 0 10.150.111.84:46428 10.150.1.201:5432 ESTABLISHED 100/java
tcp 0 0 10.150.111.84:34920 10.150.1.201:5432 ESTABLISHED 100/java
tcp 0 0 10.150.111.84:34928 10.150.1.201:5432 ESTABLISHED 100/java
tcp 0 0 127.0.0.1:36150 127.0.0.1:9001 ESTABLISHED 149/java
tcp 0 0 10.150.111.84:46426 10.150.1.201:5432 ESTABLISHED 100/java
tcp 0 0 127.0.0.1:9001 127.0.0.1:36150 ESTABLISHED 25/java
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ] STREAM CONNECTED 159855467 100/java
unix 2 [ ] STREAM CONNECTED 159863398 149/java
unix 2 [ ] STREAM CONNECTED 159857058 25/java
unix 2 [ ] STREAM CONNECTED 159853412 25/java
unix 2 [ ] STREAM CONNECTED 159853344 1/java
unix 2 [ ] STREAM CONNECTED 159856571 100/java
unix 2 [ ] STREAM CONNECTED 159864898 149/java
unix 2 [ ] STREAM CONNECTED 159853831 1/java
It doesn’t look like the list is of concerning length. That being said, given what is being seen it looks like SonarQube is caching DNS lookups instead of doing a lookup each time and our Jenkins server is behind a load balancer where the IPs can change dynamically. I noticed today when the issue happened that the IPs of the load balancer have changed since it last worked.
We found this info here in the SonarQube docs.
Sonarqube fails to decorate merge requests when DNS entry to ALM changes
If you run SonarQube in an environment with a lot of DNS friction, you should define a DNS cache time to live policy as, by default, SonarQube will hold the DNS cache until it is restarted. You can set this policy to five seconds by doing the following:
`echo "networkaddress.cache.ttl=5" >> "${JAVA_HOME}/conf/security/java.security" `
Please be aware that this increases the risk of DNS spoofing attacks.
So it appears to be something that is being done on purpose. I am not sure how well that fits with todays dynamic cloud environments.
Thoughts?