Found empty cache on server | sonar-scanner CLI for scanning C/C++ on Ubuntu

Hello,
Hope you are doing good!
Need a help with c,c++ scan issue on Ubuntu 18.04 or 20.04

  • SonarQube server version: Data Center EditionVersion 9.9 (build 65466)
    SonarQube scanner cli version: SonarScanner 4.8.0.2856
    Build wrapper version: build-wrapper-linux-x86-64 version 6.41 (linux-x86)

We are trying to analyze c,c++ code using sonarQube and facing below error during the scan:

13:06:55.807 DEBUG: Not enough content in 'devices/fansystem/translator/fan_system_translator_errors.cpp' to have CPD blocks, it will not be part of the duplication detection
13:06:55.900 DEBUG: Not enough content in 'devices/bmc_standard/logical/bmc_errors_standard.cpp' to have CPD blocks, it will not be part of the duplication detection
13:06:56.140 DEBUG: Not enough content in 'devices/bmc_fru_prom/builder/bmc_fru_prom_ld_builder.h' to have CPD blocks, it will not be part of the duplication detection
13:06:56.227 INFO: Found empty cache on server
13:06:56.282 DEBUG: Probing compiler: [/usr/bin/ccache, -x, c++, --std, c++17, -v, -dM, -E, -]
13:07:06.372 INFO: ------------------------------------------------------------------------
13:07:06.372 INFO: EXECUTION FAILURE
13:07:06.372 INFO: ------------------------------------------------------------------------
13:07:06.373 INFO: Total time: 1:05.091s
13:07:06.615 INFO: Final Memory: 97M/388M
13:07:06.615 INFO: ------------------------------------------------------------------------
13:07:06.615 ERROR: Error during SonarScanner execution
java.lang.IllegalStateException:
The compiler execution timed out, make sure that:
* the analysis is running on appropriate hardware
* eventual compiler license check over network is not taking too much time
The executed process:
  [/home/jenkins/anu/full_build/chameleon-plugin/.scannerwork/.sonartmp/613699590374514809/subprocess, -exec, /usr/bin/ccache, g++, -x, c++, --std, c++17, -v, -dM, -E, -]

        at com.sonar.cpp.driver.ProcessExecutor.execute(ProcessExecutor.java:59)
        at com.sonar.cpp.driver.ProcessExecutor.execute(ProcessExecutor.java:44)
        at com.sonar.cpp.analyzer.ClangDriver.lambda$probeCompiler$9(ClangDriver.java:759)
        at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
        at com.sonar.cpp.analyzer.ClangDriver.probeCompiler(ClangDriver.java:749)
        at com.sonar.cpp.analyzer.ClangDriver.onCapture(ClangDriver.java:488)
        at com.sonar.cpp.analyzer.CompilerDrivers.onCapture(CompilerDrivers.java:34)
        at com.sonar.cpp.plugin.CFamilySensor.lambda$process$17(CFamilySensor.java:908)
        at com.sonar.cpp.plugin.BuildWrapperJsonReader.readCaptures(BuildWrapperJsonReader.java:86)
        at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:905)
        at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:410)
        at com.sonar.cpp.plugin.CFamilySensor.execute(CFamilySensor.java:215)
        at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:64)
        at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:88)
        at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:64)
        at org.sonar.scanner.scan.SpringModuleScanContainer.doAfterStart(SpringModuleScanContainer.java:82)
        at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
        at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
        at org.sonar.scanner.scan.SpringProjectScanContainer.scan(SpringProjectScanContainer.java:403)
        at org.sonar.scanner.scan.SpringProjectScanContainer.scanRecursively(SpringProjectScanContainer.java:399)
        at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:368)
        at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
        at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
        at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:137)
        at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
        at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
        at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72)
        at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66)
        at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
        at com.sun.proxy.$Proxy0.execute(Unknown Source)
        at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
        at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
        at org.sonarsource.scanner.cli.Main.execute(Main.java:126)
        at org.sonarsource.scanner.cli.Main.execute(Main.java:81)
        at org.sonarsource.scanner.cli.Main.main(Main.java:62)

I ran the command below and seeing an error that -x option is not supported:

node:~/anu/full_build/chameleon-plugin$ time /usr/bin/ccache -x c++ --std c++17 -v -dM -E - < /dev/null
/usr/bin/ccache: invalid option -- 'x'
Usage:
    ccache [options]
    ccache compiler [compiler options]
    compiler [compiler options]          (via symbolic link)

Common options:
    -c, --cleanup             delete old files and recalculate size counters
                              (normally not needed as this is done
                              automatically)
    -C, --clear               clear the cache completely (except configuration)
    -F, --max-files=N         set maximum number of files in cache to N (use 0
                              for no limit)
    -M, --max-size=SIZE       set maximum size of cache to SIZE (use 0 for no
                              limit); available suffixes: k, M, G, T (decimal)
                              and Ki, Mi, Gi, Ti (binary); default suffix: G
    -p, --show-config         show current configuration options in
                              human-readable format
    -s, --show-stats          show summary of configuration and statistics
                              counters in human-readable format
    -z, --zero-stats          zero statistics counters

    -h, --help                print this help text
    -V, --version             print version and copyright information

Options for scripting or debugging:
        --dump-manifest=PATH  dump manifest file at PATH in text format
    -k, --get-config=K        print the value of configuration key K
        --hash-file=PATH      print the hash (<MD4>-<size>) of the file at PATH
        --print-stats         print statistics counter IDs and corresponding
                              values in machine-parsable format
    -o, --set-config=K=V      set configuration item K to value V

See also <https://ccache.dev>.

real    0m0.002s
user    0m0.000s
sys     0m0.002s

RAM details below:

vmstat -s
32868960 K total memory
2292024 K used memory
10485280 K active memory
12972924 K inactive memory
5002864 K free memory
2581196 K buffer memory
22992876 K swap cache
4194300 K total swap
3084 K used swap
4191216 K free swap
43018547 non-nice user cpu ticks
209264 nice user cpu ticks
14196843 system cpu ticks
19353603054 idle cpu ticks
2760190 IO-wait cpu ticks
0 IRQ cpu ticks
310830 softirq cpu ticks
0 stolen cpu ticks
7863207 pages paged in
620783392 pages paged out
99 pages swapped in
430 pages swapped out
2772647625 interrupts
143084698 CPU context switches
1685742816 boot time
17019819 forks

ccache doesn’t support -x option on Ubuntu OS. Is it something related to cache stored on the server?

Any update on this ticket?

@ganncamp Please help

The code base is around 8GB in size.

CPU info

jenkins@vm_ubt1804_143_60:~/anu/full_build/chameleon-plugin$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-47
Thread(s) per core: 1
Core(s) per socket: 24
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
Stepping: 2
CPU MHz: 2494.223
BogoMIPS: 4988.44
Virtualization: VT-x
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 30720K
NUMA node0 CPU(s): 0-47

jenkins@vm_ubt1804_143_60:~/anu/full_build/chameleon-plugin$ cat sonar-project.properties

SonarQube Project

sonar.projectKey=xxxx::xxxx
sonar.projectName=xxxx::xxxx
sonar.verbose=true
sonar.projectVersion=1.0

Sonar source directories to scan and excludes

sonar.sources=xxx,
xxx,
xxx
sonar.sourceEncoding=UTF-8
sonar.ws.timeout=60
sonar.cfamily.reportingCppStandardOverride=c++14
sonar.cfamily.threads=24
sonar.cfamily.analysisCache.mode=server

Hi @anushasandeep ,

could you please share the build-wrapper output directory? If you prefer to share it privately I can send you a private message.

@mpaladin Please send me a private message.

Hi @anushasandeep ,

you should have received a private message.

Hi @anushasandeep ,

I would like to clarify a couple of things prior to the investigation.

This is not a problem, this means that no cache is available on the server, this is just an info message, nothing to worry about.

We are not executing such a command, but we are executing the equivalent of /usr/bin/ccache g++ -x c++ --std c++17 -v -dM -E - < /dev/null, could you try to time such a command instead?

@mpaladin This works and gives some output.

I timed that command and ran scan again. Getting same :

jenkins@vm_ubt1804_143_60:~/anu/full_build/chameleon-plugin$ sonar-scanner -X -Dsonar.login=sqa_b49e57c8d67ff4e550b996bedc287b7324aae726 -Dsonar.cfamily.build-wrapper-output=bw-output>>sonarscan.log
09:47:30.023 ERROR: Error during SonarScanner execution
java.lang.IllegalStateException:
The compiler execution timed out, make sure that:

  • the analysis is running on appropriate hardware

  • eventual compiler license check over network is not taking too much time
    The executed process:
    [/home/jenkins/anu/full_build/chameleon-plugin/.scannerwork/.sonartmp/3868856652018897315/subprocess, -exec, /usr/bin/ccache, g++, -x, c++, --std, c++14, -v, -dM, -E, -]

      at com.sonar.cpp.driver.ProcessExecutor.execute(ProcessExecutor.java:59)
      at com.sonar.cpp.driver.ProcessExecutor.execute(ProcessExecutor.java:44)
      at com.sonar.cpp.analyzer.ClangDriver.lambda$probeCompiler$9(ClangDriver.java:759)
      at java.base/java.util.HashMap.computeIfAbsent(Unknown Source)
      at com.sonar.cpp.analyzer.ClangDriver.probeCompiler(ClangDriver.java:749)
      at com.sonar.cpp.analyzer.ClangDriver.onCapture(ClangDriver.java:488)
      at com.sonar.cpp.analyzer.CompilerDrivers.onCapture(CompilerDrivers.java:34)
      at com.sonar.cpp.plugin.CFamilySensor.lambda$process$17(CFamilySensor.java:908)
      at com.sonar.cpp.plugin.BuildWrapperJsonReader.readCaptures(BuildWrapperJsonReader.java:86)
      at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:905)
      at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:410)
      at com.sonar.cpp.plugin.CFamilySensor.execute(CFamilySensor.java:215)
      at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:64)
      at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:88)
      at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:64)
      at org.sonar.scanner.scan.SpringModuleScanContainer.doAfterStart(SpringModuleScanContainer.java:82)
      at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
      at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
      at org.sonar.scanner.scan.SpringProjectScanContainer.scan(SpringProjectScanContainer.java:403)
      at org.sonar.scanner.scan.SpringProjectScanContainer.scanRecursively(SpringProjectScanContainer.java:399)
      at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:368)
      at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
      at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
      at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:137)
      at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
      at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
      at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72)
      at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66)
      at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      at java.base/java.lang.reflect.Method.invoke(Unknown Source)
      at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
      at com.sun.proxy.$Proxy0.execute(Unknown Source)
      at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
      at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
      at org.sonarsource.scanner.cli.Main.execute(Main.java:126)
      at org.sonarsource.scanner.cli.Main.execute(Main.java:81)
      at org.sonarsource.scanner.cli.Main.main(Main.java:62)
    

@mpaladin

When I run the sonar scan, the execution hangs at compiler probing step. In another session, I can see that the below process is running for a longer time in the background causing the timeout. This is happening on Ubuntu 18.04 and 20.04

jenkins 1266347 1 0 07:57 pts/0 00:00:00 g++ -x c++ --std c++17 -v -dM -E -

I can share the trace logs in a provate message.

The process has to be manually killed to free the resources.

scan tool is directly invoking the command “g++ -x c++ --std c++17 -v -dM -E -” due to which there is a timeout.

The number of times I run a scan, those number of compiler probe processes are created in the background which does not complete smoothly.

When I run the command manually, it doesn’t finish until I kill using ctrl-C

jenkins@ubt2004143049:~$ g++ -x c++ --std c++17 -v -dM -E -
Using built-in specs.
COLLECT_GCC=g++
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: …/src/configure -v --with-pkgversion=‘Ubuntu 9.4.0-1ubuntu1~20.04.2’ --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-9QDOt0/gcc-9-9.4.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.2)
COLLECT_GCC_OPTIONS=‘-std=c++17’ ‘-v’ ‘-dM’ ‘-E’ ‘-shared-libgcc’ ‘-mtune=generic’ ‘-march=x86-64’
/usr/lib/gcc/x86_64-linux-gnu/9/cc1plus -E -quiet -v -imultiarch x86_64-linux-gnu -D_GNU_SOURCE - -mtune=generic -march=x86-64 -std=c++17 -fasynchronous-unwind-tables -fstack-protector-strong -Wformat -Wformat-security -fstack-clash-protection -fcf-protection -dM
ignoring duplicate directory “/usr/include/x86_64-linux-gnu/c++/9”
ignoring nonexistent directory “/usr/local/include/x86_64-linux-gnu”
ignoring nonexistent directory “/usr/lib/gcc/x86_64-linux-gnu/9/include-fixed”
ignoring nonexistent directory “/usr/lib/gcc/x86_64-linux-gnu/9/…/…/…/…/x86_64-linux-gnu/include”
#include “…” search starts here:
#include <…> search starts here:
/usr/include/c++/9
/usr/include/x86_64-linux-gnu/c++/9
/usr/include/c++/9/backward
/usr/lib/gcc/x86_64-linux-gnu/9/include
/usr/local/include
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.

Could you please test this case on Ubuntu for a combination of c and c++ code?

Hi @anushasandeep ,

we found the reason for the timeout, it is because of fakeroot. You are using fakeroot during your build and then during the analysis our analyzer try to invoke the compiler as it was invoked during build. Fakeroot hangs during the analysis because it is not running under fakeroot.

We have a ticket for it: [CPP-4724] - Jira

While waiting for the fix, you could either:

  • remove the use fakeroot from the build or restrict it to the part of the build where you need it. I don’t think that fakeroot is needed for the compiler invocation
  • run everything under fakeroot, even the sonar-scanner

Thank you very much @mpaladin

When i ran build wrapper and sonar scan using the root user, the issue disappeared and I was able to get the results.

Let me wait for the bug fix. Please notify me on this ticket once the ticket is fixed.

@anushasandeep great to hear that! Out of curiosity: what is the reason to use fakeroot?

@mpaladin

Our build needs access to some root owned kernel binary folders during the build and sudo access is a must.