Web Interface unresponsive

[ThE (Thomas Erlenmayer)]

I’m running SonarQube Developer Edition Version 9.9 (build 65466), deployed from zip, running in a highly available Virtual Machine.

What’s happening:
Currently I’m facing recurring unresponsive Web UI and backend which only can get back to work by restarting the SonarQube Service manually.
This happens sporadically few times a day (~3-6 times).

• The Web UI is not loading any content (pages keep empty with a load indicator) but it’s still possible to switch the tab pages.
  

  

• CI jobs interacting with SonarQube are failing to connect to SonarQube for scan publications.
  Failed to request and parse 'https://vistrpmcsonar2.vi.vector.int/api/server/version': A task was canceled.

• SonarQube keeps unresponsive (Because their are workflows depending on the analysis I have to restart the service asap. Therfor I can’t tell if it ever will return to life by itself, max. time i can confirm was 3 hours.

What I’ve tried so far:

• Extended RAM of the Virtual Machine to 16GB (Memory usage ~30%)
• sonar.properties
  • Extended default to: sonar.web.javaOpts=-Xmx1G -Xms128m -XX:+HeapDumpOnOutOfMemoryError
  • Extended default to: sonar.ce.javaOpts=-Xmx2G -Xms128m -XX:+HeapDumpOnOutOfMemoryError
  • Elastic search is still on default.
• Searched in the logs (attached) for any abnormality
• Increased the log level after the latest occurrence - waiting for the next unresponsive phase.
• As long as this happens I roll back new projects one by one - but this can’t be the solution.

Suspect things:

• Lately we increased our licensed LoC and migrated multiple projects, each >250k LoC.
• Many occurrences of SonarLint in access.log
  "GET /api/developers/search_events?projects=VCS&from=2023-04-24T04%3A51%3A05%2B0200&projects=VCS&from=2023-04-24T04%3A51%3A05%2B0200 HTTP/1.1" 200 - "-" "SonarLint IntelliJ 7.2.0.56810" "AYezWHdLU5J0f15dAA4P"

Any suggestions to narrow down the problem?

logs.zip (284.8 KB)

[anon67236913]

Hi,

at first glance, i don’t see anything suspicious in your logs.
The many Sonarlint entries are from the Sonarlint IntelliJ plugin connected to this Sonarqube server.

Since you are using https, there is a reverse proxy involved.

@ThE

CI jobs interacting with SonarQube are failing to connect to SonarQube for scan publications.
Failed to request and parse 'https://vistrpmcsonar2.vi.vector.int/api/server/version': A task was canceled.

Have you also checked the logs of your reverse proxy?
Are there other proxies / firewalls / ips on the line that may be preventing a connection ?

Gilbert

[ThE (Thomas Erlenmayer)]

Hi,

thanks for your reply.

What surprises me is the fact that after restarting the SonarQube Service everything worked as usual.
That actually speaks against an incorrectly configured proxy.

Anyways, I need to check this with the infrastructure department as internal network infrastructure is out of my scope.

I let you know if I get further information.

Thomas

Edit: The infrastructure department checked the proxy and they did not see anything suspicious. Due to the fact that the VM is still reachable for restarting the servie via RDP they exlcude a network issue.

[anon67236913]

@ThE

What surprises me is the fact that after restarting the SonarQube Service everything worked as usual.
That actually speaks against an incorrectly configured proxy.

Don’t know if you are using Apache, nginx or IIS. We are using Apache and had to increase
the mpm_winnt_module/threadsperchild because of intermittent problems.

Had another look at your logs and noticed some exceptions related to the ldap connection
web.log

2023.04.25 12:24:53 ERROR web[AYe311F7T56mR1EmAAU/][o.s.s.a.LdapCredentialsAuthentication] Error during authentication
org.sonar.auth.ldap.LdapException: Unable to retrieve details for user — in default
[…]
Caused by: javax.naming.CommunicationException: simple bind failed: ldaps.vi.vector.int:3269

btw web log has also

2023.04.25 11:55:53 WARN web[o.s.s.p.d.CheckAnyonePermissionsAtStartup] Authentication is not enforced, and permissions assigned to the ‘Anyone’ group globally expose the instance to security risks. Unauthenticated visitors may unintentionally have permissions on projects.

Your “hardware” and memory settings look fine, for comparsion these are my settings for Sonarqube Enterprise with ~15.000.000 LOC, > 3000 projects, > 500 users

VM running on Windows Server 2019
8 CPU, 12G RAM
Sonarqube partition has 75G, 40G free
4 CE workers
sonar.web.javaOpts=-server -Xmx1G -Xms128m -XX:+HeapDumpOnOutOfMemoryError
sonar.ce.javaOpts=-server -Xmx4G -Xms512m -Xss1G -XX:+HeapDumpOnOutOfMemoryError
otherwise default

Is your VM dedicated to Sonarqube or are there more applications running?
How much free space ?

@ThE

Due to the fact that the VM is still reachable for restarting the servie via RDP they exlcude a network issue.

Even if it is reachable via RDP, there might be blocked requests / connections.

For further investigation use =

Browser developer tools and inspect network activity and browser console

Javamelody plugin is also helpful for monitoring your Sonarqube instance

GitHub - javamelody/sonar-javamelody: JavaMelody plugin for Sonar

In Jenkins, increase the loglevel for Sonarqube scanner with -X cli parameter and create a custom Jenkins logger (/manage/log/new) with configuration logger hudson.plugins.sonar and org.sonarsource.

After all you might also try reindexing the ES indices

1. Stop SonarQube
2. Delete the data/es7 directory
3. Restart SonarQube

[ThE (Thomas Erlenmayer)]

Proxy:

• we’re using IIS but I really don’t think it’s an proxy issue.

Authentication Error:

• I disabled the ‘Anyone’ authentication, which was not intended anyways.

VM:

• the VM is used for SonarQube exclusively - there are no other services running on the VM
• free space: C:\ 24GB, D:\ 145GB
• memory usage ~35%

I didn’t check the suggested helper tools yet because it was not required since my last changes:

• One week ago I disabled one of the recently added projects with 400k LoC in the build automation - no new analysis since last week for this project
• I ordered the team using SonarLint in Eclipse to disable it once again
• Set the LogLevel to detailed

Since those adjustments the SonarQube is running fine.
I’ll slowly bring the changes back to production now.
In case of further errors I will come back to this thread.

[ThE (Thomas Erlenmayer)]

Hello,

as mentioned in my previous statement I brought back the disabled projects one by one.
SonarQube worked without any issues for the past 8 days.
Today the team using IntelliJ IDEA switched on their SonarLint Plugin again.
It ran fine for ~4 hours, then SonarQube freezed.

I’ve stored all the logs since my initial post.
All access.log have one line in common before the unresponsive UI:

// 27/Apr/2023
10.49.34.108 - - [27/Apr/2023:11:17:02 +0200] "GET /api/push/sonarlint_events?projectKeys=PROJECT&languages=web,java,js,kotlin,php,py,ruby,scala,secrets,swift,ts,xml,yaml HTTP/1.1" 200 - "-" "SonarLint IntelliJ 7.2.1.58118" "AYe9bpAVCf4JKHrrAB5w"
0:0:0:0:0:0:0:1 - - [27/Apr/2023:11:22:26 +0200] "GET /api/push/sonarlint_events?projectKeys=PROJECT&languages=web,java,js,kotlin,php,py,ruby,scala,secrets,swift,ts,xml&projectKeys=PROJECT&languages=web,java,js,kotlin,php,py,ruby,scala,secrets,swift,ts,xml HTTP/1.1" 404 - "-" "SonarLint IntelliJ 7.1.1.54565" "-"
// 29/Apr/2023
10.49.34.108 - - [29/Apr/2023:14:58:09 +0200] "GET /api/push/sonarlint_events?projectKeys=PROJECT&languages=web,java,js,kotlin,php,py,ruby,scala,secrets,swift,ts,xml,yaml HTTP/1.1" 200 - "-" "SonarLint IntelliJ 7.2.1.58118" "AYfCBZ10cxJJrDS6ABBf"
// 12/May/2023
10.49.34.108 - - [12/May/2023:10:21:15 +0200] "GET /api/push/sonarlint_events?projectKeys=PROJECT&languages=css,web,java,js,kotlin,php,py,ruby,scala,secrets,swift,ts,xml,yaml,cloudformation,docker,kubernetes,terraform&projectKeys=vCDMServer&languages=css,web,java,js,kotlin,php,py,ruby,scala,secrets,swift,ts,xml,yaml,cloudformation,docker,kubernetes,terraform HTTP/1.1" 200 - "-" "SonarLint IntelliJ 8.2.0.68615" "AYf69l5TSeeXqXoIAEaI"

It seems to me that the GET /api/push/sonarlint_events is causing the trouble, even though there a lots of those requests which do not freeze SonarQube.

• What are those requests doing at all?

I therefore strongly suspect that it is related to the SonarLint IntelliJ Plugin.
All developers are using the latest available version 8.2.0.68615 now, but as the latest log shows, it is not solved by using the latest version.

As for now SonarLint will be disabled again.
Any further ideas?

Thanks in advance.
Thomas

[anon67236913]

Hi Thomas,

that’s strange. We have several hundred Sonarlint instances (IntelliJ, Eclipse, VS Code) with ‘Receive Notificiation’ enabled and from what i see in the Sonarqube access.log it’s not that Sonarlint is just waiting for new events pushed from Sonarqube server, but a Sonarlint instance is polling every 5 minutes for new events - there’s also a mix of different versions, as not every developer is updating Sonarlint frequently.

I found no setting to change that intervall of 5 minutes, but you could try to deactivate the ‘Receive Notitification’ feature, means the sync between Sonarlint ↔ Sonarqube has to be done manually via Update Binding, and see if that fixes your problem.

IMO an interval of 5 minutes is too ambitious, that should be configurable.

Gilbert

[ganncamp (G Ann Campbell)]

Hi guys,

I’ve flagged this for team attention.

 
Ann

[ThE (Thomas Erlenmayer)]

Hi Ann & Gilbert,

SonarQube is running fine since my last post (without SonarLint for IntelliJ IDEA).

I’ve instructed the team to disable the “Receive notifications from SonarQube” checkbox in IntelliJ IDEA.
We’ll give another try.

Thomas

[ThE (Thomas Erlenmayer)]

Hello again,

SonarQube was stuck in the afternoon, 3-4h after enabling the SonarLint plugin for IntelliJ IDEA.
The access log shows the same behavior as already mentioned.
access.2023-05-22.log (1.9 MB)

The last thing I can see is the GET /api/push/sonarlint_events.

We have to disable SonarLint for IntelliJ IDEA until there is a solution for this.

Thomas

[leo.geoffroy]

Hello @ThE

The GET /api/push/sonarlint_events endpoint is using the Server-Sent Event (text/event-stream), which implies keeping an HTTP connection active for each consumer of the endpoint.

I believe the IIS server may have a limitation on the number of active connections, and after multiple SonarLint connections, the IIS is not able to serve the requests anymore.

Can you check that IIS is configured to support Server-Sent Events correctly? You can also try to increase the size of the connection pool and check if there are any differences.

Would you be able to access locally to SonarQube from the virtual machine (and bypass the IIS instance) to check if it is working correctly (When it is stuck)? It would show that the issue comes from the IIS instance.

[ThE (Thomas Erlenmayer)]

Hello Leo,

I’m able to access the SonarQube from the virtual machine but its stuck as shown in the image I attached on my first post.
It doesn’t make a difference if accessing via the internal URL or the local host - both are not responding anymore.

I’ll check the IIS Settings with the Infrastructure Department as soon as possible.