From time to time Sonar END process fail with java.lang.IllegalStateException

  • SonarQube server - Enterprise Edition Version 9.5 (build 56709)
    sonar-scanner-msbuild - 5.0.4

Build fails from time to time, but more often it fails than finish successfully.

ERROR: Error during SonarScanner execution
java.lang.IllegalStateException: Unable to load component class org.sonar.scanner.scan.filesystem.ProjectFileIndexer
  at org.sonar.core.platform.SpringComponentContainer.getComponentByType(SpringComponentContainer.java:139)
  at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:352)
  at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
  at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
  at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:135)
  at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
  at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
  at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72)
  at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66)
  at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:567)
  at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
  at com.sun.proxy.$Proxy0.execute(Unknown Source)
  at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
  at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
  at org.sonarsource.scanner.cli.Main.execute(Main.java:112)
  at org.sonarsource.scanner.cli.Main.execute(Main.java:75)
  at org.sonarsource.scanner.cli.Main.main(Main.java:61)

Check detailed stack trace in attached file
filed stacktrace.txt (36.3 KB)

  • I thougth there where a problem with memory lack on agent\sonar server machine\sonar-scanner process

Hi,

The underlying problem (you have to scroll right about 3 miles to find it) is:

HttpException: Error 504 on https://simcorpsonarqubetest.azurewebsites.net/batch/project.protobuf?key=IMS-CSharp&branch=MG-IMS/master : 504.0 GatewayTimeout

There is likely something on your network, e.g. a proxy, that’s blocking the analysis report submission.

Because this is intermittent, I’m going to guess that it’s related to the size of the analysis report crossing some arbitrary threshold set in the corporate proxy/firewall/etc.

You should check with your network folks.

 
HTH,
Ann

1 Like

Each time when the build fail rules loading takes pretty long time
Is it possible that there is something wrong with current rules set?

Hi,

I doubt it.

Instead, I’d suspect that your network conditions are headed south and the slow rule loading is an early indicator.

 
FWIW,
Ann

@ganncamp
Can you explain me about such a requst(project.protobuf)?
What is the reason of the request and what SQ server generate here?
what depends on the server’s response time?
Is it possible that problem will disappear if we increase timeout?

Hi,

That URL doesn’t resolve for me. I guess it’s something you picked out of an analysis log? In general, analysis starts with a number of requests to the server to get the resources needed to run analysis. Then it uses those resources locally to compile an analysis report. It finishes by submitting that report to the server.

I know that’s pretty generic. Does it help at all?

 
Ann

Hi,
the issue is still there. we have 2 servers on the same containers. Version 9.3 (build 51899) and * Version 9.5 (build 56709) - problem server. Each time fails with the same request.
Can some developer check why it could happan exactly there?

Hi,

Is there anything in your server logs that corresponds to these errors?

 
Ann

Hi Ann,
I’m Harry’s colleague and I was looking into this issue from the web app end.
I checked web.log and access.log. and I did not find what can cause
this error. Also, I talked with network guys and they did not find any issue with the firewall or blocked traffic.
I will add the logs, maybe I missed something. Thank you in advance.
SQ_access_log.txt (871.5 KB)
SQ_web_log.txt (91.5 KB)

Hi @Vitalii,

Welcome to the community!

I’m not seeing anything unusual in your server logs. I’m going to have to refer you back to your '504.0 GatewayTimeout` error & your network folks.

 
Ann

Hi Ann,
Thanks.

Our network folks did not find anything from the network end.
I read about “sonar.ws.timeout” parameter and want to try to change it. Can it be helpful in my case? If yes, where it should be done?

Hi,

A 504 error comes from the server (or some proxy or other “helpful” service in the middle). So setting a client-side timeout wouldn’t help.

Going back to the full error initially provided by your colleague, the error appears to happen at the end, during analysis report submission:

ERROR:
The SonarScanner did not complete successfully
09:24:46.823  Post-processing failed. Exit code: 1
An error occurred when executing task 'SonarScannerEnd'.
Executing custom task teardown action (SonarScannerEnd)...
Error: System.AggregateException: One or more errors occurred. ---> System.Exception: SonarScanner failed.
   at Submission#0.RunSonarScanner(String cmdArgs, Dictionary`2 env) in C:\buildAgent\work\fe57f4080812cf27\src\CsCode\tools\SimCorp.Tools.Cake.Sonar.4.0.1\sonar.cake:line 103
   at Submission#0.SonarScannerEnd() in C:\buildAgent\work\fe57f4080812cf27\src\CsCode\tools\SimCorp.Tools.Cake.Sonar.4.0.1\sonar.cake:line 93

When this happens intermittently, as your colleague reported, it’s typically about the size of the analysis report tripping some threshold in those “helpful” intermediaries on your network.

 
HTH,
Ann

I found interesting thing.
The issue disappears when I reduced analysis scope.
Full solution is pretty large - 3.4M LoC
I did several test with some projects from this solution and all test have been done as well (600К LoC).

btw, previously we also scanned full solution, but exception appears only after update sonar server.

Hi,

Analyzing a smaller scope is going to result in a smaller analysis report, meaning it’s less likely to trip the file size limit in your “helpful” network intermediaries.

 
HTH,
Ann