java.lang.IllegalStateException while Probing armclang

frn_cj · August 14, 2019, 9:09am

Hi,
we are using the following Sonarqube setup:

SonarQube 6.7.0.33306
SonarQube Scanner 3.0.3.778 (I tested 3.4.0.1729 as well.)
SonarCfamily 6.3.0.11371

I tried to run a scan on a project with the armclang compiler 6.12 in a docker container
in a GitLab CI stage. The system below is a CentOs 7 with kernel 3.10.0-514.6.2.el7.x86_64
The docker container runs an Ubuntu 18.04.

This is how I’m running the scan:

/opt/build-wrapper/build-wrapper-linux-x86-64 --out-dir sonar_dir make all
/opt/sonar-scanner/bin/sonar-scanner -X

I’m getting the follwing error message:

22:21:48.463 DEBUG: Probing compiler: [/usr/local/ARMCompiler6.12/bin/armclang, -x, c, --std, c99, -target, arm-arm-none-eabi, -mcpu=cortex-m3, -v, -dM, -E, -]
22:21:53.504 INFO: ------------------------------------------------------------------------
22:21:53.504 INFO: EXECUTION FAILURE
22:21:53.504 INFO: ------------------------------------------------------------------------
22:21:53.504 INFO: Total time: 1:34.373s
22:21:53.649 INFO: Final Memory: 54M/223M
22:21:53.649 INFO: ------------------------------------------------------------------------
22:21:53.649 ERROR: Error during SonarQube Scanner execution
java.lang.IllegalStateException
	at com.sonar.cpp.driver.ProcessExecutor.execute(ProcessExecutor.java:32)
	at com.sonar.cpp.analyzer.ClangDriver.lambda$probeCompiler$7(ClangDriver.java:491)
	at java.util.HashMap.computeIfAbsent(HashMap.java:1126)
	at com.sonar.cpp.analyzer.ClangDriver.probeCompiler(ClangDriver.java:489)
	at com.sonar.cpp.analyzer.ClangDriver.onCapture(ClangDriver.java:318)
	at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:273)
	at com.sonar.cpp.plugin.CFamilySensor.execute(CFamilySensor.java:205)
	at org.sonar.scanner.sensor.SensorWrapper.analyse(SensorWrapper.java:53)
	at org.sonar.scanner.phases.SensorsExecutor.executeSensor(SensorsExecutor.java:88)
	at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:82)
	at org.sonar.scanner.phases.SensorsExecutor.execute(SensorsExecutor.java:73)
	at org.sonar.scanner.phases.AbstractPhaseExecutor.execute(AbstractPhaseExecutor.java:88)
	at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:180)
	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)
	at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121)
	at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:288)
	at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:283)
	at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:261)
	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)
	at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121)
	at org.sonar.scanner.task.ScanTask.execute(ScanTask.java:48)
	at org.sonar.scanner.task.TaskContainer.doAfterStart(TaskContainer.java:84)
	at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:135)
22:21:53.653 DEBUG: Execution getVersion
22:21:53.654 DEBUG: Execution stop
	at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:121)
	at org.sonar.scanner.bootstrap.GlobalContainer.executeTask(GlobalContainer.java:121)
	at org.sonar.batch.bootstrapper.Batch.doExecuteTask(Batch.java:116)
	at org.sonar.batch.bootstrapper.Batch.executeTask(Batch.java:111)
	at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
	at com.sun.proxy.$Proxy0.execute(Unknown Source)
	at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:233)
	at org.sonarsource.scanner.api.EmbeddedScanner.runAnalysis(EmbeddedScanner.java:151)
	at org.sonarsource.scanner.cli.Main.runAnalysis(Main.java:123)
	at org.sonarsource.scanner.cli.Main.execute(Main.java:77)
	at org.sonarsource.scanner.cli.Main.main(Main.java:61)

So it looks like that probing armclang crashes.

What I tried so far?

run the scanner job locally on my Ubuntu 18.04 - works
run the scanner job in docker on my Ubuntu 18.04 - works
use my local Ubuntu as GitLab runner + docker - works
use a GitLab runner that has CentOS7 below, same Ubuntu 18.04 docker - fails
use just another Ubuntu 18.04 as GitLab runner + same Ubuntu 18.04 docker - fails
run the scanner job on the forementioned Ubuntu locally works

So I tried different underneath OS, kernel versions and scanner versions but could not really find a difference.

I can provide a minimal project (Makefile, c-file, etc.) and a Dockerfile on request.

Any idea what’s wrong here?

Thanks,
Franziska

mpaladin · August 14, 2019, 9:42am

Hi @frn_cj,

could you please rerun with sonar.verbose=true? That will help because it is going to log the output of the probe which would help me to understand the issue.

frn_cj · August 14, 2019, 10:20am

Hi @mpaladin

Please see attached file for the log.

output.txt (32.7 KB)

Thanks,
Franziska

mpaladin · August 14, 2019, 12:07pm

Hi @frn_cj,

the probe fails because it takes more than 5 seconds to execute the following command:

/usr/local/ARMCompiler6.12/bin/armclang -x c --std c99 -target arm-arm-none-eabi -mcpu=cortex-m3 -v -dM -E -

frn_cj · August 14, 2019, 12:10pm

Hi @mpaladin,
this could be a network issue then. (License server or whatever)
Can I set a timeout somewhere or is there another workaround?

Thanks,
Franziska

mpaladin · August 14, 2019, 12:15pm

Hi @frn_cj ,

no, it is not possible to change the timeout.

Could you check if it is ok to take such a long time?

frn_cj · August 14, 2019, 12:19pm

Hi @mpaladin,

I’ll try to run just the probing command in my environment and check how long it takes.
I’ll update this thread later.

Thank you so much, your answer helps a lot to further debug what’s wrong here.

frn_cj · August 14, 2019, 2:59pm

Hi @mpaladin,

we are still investigating why the armclang probe job takes so long on some machines.

May I request a new feature for sonar scanner to have this timeout as setting?
What do you think?

Thanks,
Franziska

mpaladin · August 16, 2019, 9:29am

Hi @frn_cj,

did you manage to investigate why armclang probe takes so long on some machines?

We discussed it at some point but didn’t take any action as no one ever reported such issue so far.

frn_cj · August 16, 2019, 10:11am

Hi @mpaladin,

I think it has something to do with the OS caching.
If I run armclang probing the first time - it’s slow, if I run it immediately afterwards - it’s a way faster.
How long “slow” and “fast” is, dependes on the HW/Virtualization.

Running on a non-virtualized system (Ubuntu 18.04), probing armclang takes 2 seconds for the first run and less than 1 second on the second run. (This kind of caching effect is away after 5 seconds.)

Running on a virtualized system (VMWare + Ubuntu 18.04), probing armclang takes 8 seconds for the first run and less than 1 second on the second run.

Trying to find out why the first run is slower showed: It’s not the CPU and not RAM.
(Tested with prime number calculation, RAM reservation from VMWare)

I tried one workaround: Calling the armclang probe job directly before starting the sonar job, but it didn’t work, probably because the sonarjob itself takes so long.

One further question I have: Is it possible to disable the compiler probing or is this a stupid idea?

I’d still appreciate the timeout setting

Thanks,
Franziska

frn_cj · August 16, 2019, 11:54am

Hi @mpaladin,

I checked the armclang probing with strace now.
It looks like the first call of probing armclang pulls the license from our license server.
The second call of probing armclang does not pull the license (maybe cached), therefore it’s faster.

Might be a network issue if I run probing on a virtual machine … I’ll investigate further.

Thanks,
Franziska

mpaladin · August 19, 2019, 7:47am

Hi @frn_cj,

probing of compiler cannot be disabled, it is needed to fetch the correct configuration for the analysis.

The only reasonable thing would be to increase the timeout.

mpaladin · August 20, 2019, 7:47am

A post was split to a new topic: java.lang.IllegalStateException while probing compiler

frn_cj · August 23, 2019, 6:55am

Hi @mpaladin,

finally we found the root cause. The WAN accelerator obviously decelerated the network traffic in one of our networks. This lead to compiler probing taking more than 5 seconds, since the compiler pulled the license from a license server in the US (We are located in Germany).

So the issue did not depnend on virtualization, operating system, docker, GitLab or SonarQube.

Your hint, that sonar scanner stops probing after 5 seconds was crucial to find the root cause.

Thanks again
Franziska

mpaladin · August 23, 2019, 7:29am

Hi @frn_cj,

great to hear that you found the issue, let us know whether you face any other difficulties.

Sameer_Soni · August 26, 2019, 6:40am

@frn_cj thanks for telling us root cause. Please tell us how you solved this issue. We are also facing this issue.

frn_cj · August 26, 2019, 6:58am

Hello @Sameer_Soni
our IT change some firewall rules to bypass the packets that should go to our US license server (for armclang). Packets that have the license server ip address as destiantion will not go through the WAN accerlerator anymore.