.ProjectFileIndexer exceptions and timeouts call SonarQube to get the project.protobuf file

[rfennell (Richard Fennell)]

Issue

Over the past few weeks we have been seeing intermittent occurrences of the ProjectFileIndexer exception sure SonarQube analysis

##[error]java.lang.IllegalStateException: Unable to load component class org.sonar.scanner.scan.filesystem.ProjectFileIndexer

If I look closer at that the exception stack I can see at the bottom there is timeout

Caused by: java.lang.IllegalStateException: Fail to request url: https://sonarqube.mydomain.co.uk/batch/project.protobuf?key=mykey

Configuration

We are running our SonarQube 10.7 Dev instance on Azure as a Docker container

Note we also saw this issue when running 10.6

The instance contains around 1M LOC on a 2ML LOC license

• Azure Web App (Premium V3 P1V3 2vCPU & 8Gb)
• Azure SQL DB (Standard S2 50 DTU)
• Logs/Config/ES indexes on Azure Standard General Purpose V2) File storage

Analysis

Resources

I initially scaled the SQL instance from 20 DTU to 50 DTU, this made no difference to the problem

Deleting Elasticsearch Indexes

I stopped the container, deleted the Elasticsearch Indexes, restarted the container. The indexes rebuilt as expected in around 20 minutes, this made no difference to the problem

Trying to access to failing URL

I tried opening the URL https://sonarqube.mydomain.co.uk/batch/project.protobuf?key=mykey in an SQ authenticated browser session.

• Usually it returned the expected file instantly.
• However from time to time, I could not spot a pattern, the URL timed out, took around 30 seconds

Question

The root of the issue I think is the intermittent failure of call to the SQ instance to get information required for the analysis.

The issue is I am not sure how to fix this, do I just throw more resources at it?

Can anyone suggest an approach, or better still a fix if you have seen the problem before?

[rfennell (Richard Fennell)]

A bit more information, the issue looks to be some very slow running SQL queries after the container restart, maxing out my S2 50 DTU Azure SQL instance

The slowest queries all seem to related to the live_measures, is this normal after an upgrade, and deletion if large (400K LOC) project?

During these running of the queries the SQ became intermittently unresponsive, I think causing the originally reported problems

After a few hours these queries ended, and thus far SQ seems OK.

So I guess the questions become

1. Can I do anything about the impact of these queries e.g.
   • A larger SQL instance?
   • Purge/tidy older data in the DB
2. Or stop them query running on a restart?

Or have It got this wrong, and this is normal behavior and the timeout issues are unrelated?

[rfennell (Richard Fennell)]

My problems seem to be been addressed by scaling my Azure SQL instance from S2 (50 DTU) to an S3 (100 DTU). Once this completed SonarQube analysis within my Azure DevOps pipelines have completed without any of the intermittent errors access SonarQube URLs.

Since the resizing, my SQL instance has not been ‘maxing out’, which I suspect was the root of the problem