Scan C++ code with compilation database stored in compile_commands.json
what have you tried so far to achieve this
It works, but on some builds sonar-scanner fails with a null pointer exception.
We get the following output:
ERROR: Error during SonarScanner execution
java.lang.NullPointerException
at com.sonar.cpp.plugin.CFamilySensor.lambda$save$20(CFamilySensor.java:925)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(Unknown Source)
at com.sonar.cpp.plugin.CFamilySensor.save(CFamilySensor.java:925)
at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:876)
at com.sonar.cpp.plugin.CFamilySensor.lambda$process$17(CFamilySensor.java:765)
at com.sonar.cpp.plugin.CompileCommandsReader.readCaptures(CompileCommandsReader.java:61)
at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:744)
at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:368)
at com.sonar.cpp.plugin.CFamilySensor.execute(CFamilySensor.java:203)
at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:64)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:85)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:62)
at org.sonar.scanner.scan.SpringModuleScanContainer.doAfterStart(SpringModuleScanContainer.java:81)
at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
We’re trying to scan a C++ project using a compile_commands.json. This error is definitely an error in the sonar-scanner itself, since whatever array it’s trying to index into is unsound.
We need this to be resolved, since currently about 20% of builds are failing this way.
For the purpose of investigation could you please provide the full content of the sonar-scanner-log with debug enabled (-X passed to sonar-scanner invocation) and your compilation database file (compile-commands.json)? If you think this file contains private information, let us know, and we’ll send you a private message that will allow you to send it privately.
This is a bit of an educated guess, but could you please check if setting the following property (in sonar-project.properties or in the command line using -Dsonar.analysisCache.enabled=false) will fix the crash for you:
Since this takes some time to run on our project I can’t confirm this yet. But we’ve already started clearing the analysisCache (since we still want to create a fresh one then to keep the compile times from becoming prohibitive), and it seems to usually work after that.
However this still means that the sonar-scanner is not checking properly what it’s indexing into here, since even if something else is cached it shouldn’t take down the whole build.
We’ve also tried using the new server-side caching due to the deprecation message, but no matter how many runs we do we always get the empty server cache warning.
We understand and acknowledge that there is an issue in the sonar-scanner that causes some projects to fail your analysis, and thus greatly appreciate your report. However, this issue does not reproduce on any of our internal tests (we analyze a lot of projects daily with cache enabled), and the line in the report points to a place that is never expected to be null. Thus to find the root cause and fix the issue we need more information.
This is why I asked you to privately share the analysis log (PM was already sent), and to perform some tests that would either confirm or eliminate some of the possible root causes.
After adding the --debug or -X flag I got a different failure. I removed the identifiable file names
2022-08-10T14:03:49.6985684Z 14:03:49.697 DEBUG: [pool-3-thread-30] /usr/include/x86_64-linux-gnu/qt5/QtCore/qglobal.h:1187 "You must build your code with position independent code if Qt was built with -reduce-relocations. " "Compile your code with -fPIC (-fPIE is not enough)."
2022-08-10T14:03:49.7002761Z 14:03:49.699 DEBUG: [pool-3-thread-30] /src/absolute/path/to/REDACTED.cpp '../include/relative/path/to/OTHER_REDACTED.cpp' file not found
2022-08-10T14:03:49.7348076Z 14:03:49.733 INFO: ------------------------------------------------------------------------
2022-08-10T14:03:49.7348630Z 14:03:49.733 INFO: EXECUTION FAILURE
2022-08-10T14:03:49.7349203Z 14:03:49.733 INFO: ------------------------------------------------------------------------
2022-08-10T14:03:49.7349629Z 14:03:49.734 INFO: Total time: 1:44.463s
2022-08-10T14:03:49.8107507Z 14:03:49.809 INFO: Final Memory: 146M/616M
2022-08-10T14:03:49.8109488Z 14:03:49.809 INFO: ------------------------------------------------------------------------
2022-08-10T14:03:49.8110145Z 14:03:49.809 ERROR: Error during SonarScanner execution
2022-08-10T14:03:49.8110929Z java.lang.IllegalStateException: java.io.IOException
2022-08-10T14:03:49.8111705Z at com.sonar.cpp.plugin.cache.CacheInImpl.readExternalIssuesCacheName(CacheInImpl.java:192)
2022-08-10T14:03:49.8112382Z at com.sonar.cpp.plugin.cache.CacheInImpl.query(CacheInImpl.java:86)
2022-08-10T14:03:49.8113014Z at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:871)
2022-08-10T14:03:49.8113633Z at com.sonar.cpp.plugin.CFamilySensor.lambda$process$17(CFamilySensor.java:765)
2022-08-10T14:03:49.8114367Z at com.sonar.cpp.plugin.CompileCommandsReader.readCaptures(CompileCommandsReader.java:61)
2022-08-10T14:03:49.8115411Z at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:744)
2022-08-10T14:03:49.8116009Z at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:368)
2022-08-10T14:03:49.8116604Z at com.sonar.cpp.plugin.CFamilySensor.execute(CFamilySensor.java:203)
2022-08-10T14:03:49.8117246Z at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:64)
2022-08-10T14:03:49.8117924Z at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:85)
2022-08-10T14:03:49.8118617Z at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:62)
2022-08-10T14:03:49.8119341Z at org.sonar.scanner.scan.SpringModuleScanContainer.doAfterStart(SpringModuleScanContainer.java:81)
2022-08-10T14:03:49.8120099Z at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
2022-08-10T14:03:49.8120828Z at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
2022-08-10T14:03:49.8121548Z at org.sonar.scanner.scan.SpringProjectScanContainer.scan(SpringProjectScanContainer.java:392)
2022-08-10T14:03:49.8122296Z at org.sonar.scanner.scan.SpringProjectScanContainer.scanRecursively(SpringProjectScanContainer.java:388)
2022-08-10T14:03:49.8123064Z at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:357)
2022-08-10T14:03:49.8123801Z at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
2022-08-10T14:03:49.8124537Z at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
2022-08-10T14:03:49.8125271Z at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:135)
2022-08-10T14:03:49.8126016Z at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
2022-08-10T14:03:49.8126735Z at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
2022-08-10T14:03:49.8127381Z at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72)
2022-08-10T14:03:49.8127935Z at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66)
2022-08-10T14:03:49.8128599Z at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
2022-08-10T14:03:49.8129261Z at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-08-10T14:03:49.8129839Z at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2022-08-10T14:03:49.8130420Z at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2022-08-10T14:03:49.8131086Z at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2022-08-10T14:03:49.8131702Z at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
2022-08-10T14:03:49.8132299Z at com.sun.proxy.$Proxy0.execute(Unknown Source)
2022-08-10T14:03:49.8132852Z at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
2022-08-10T14:03:49.8133490Z at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
2022-08-10T14:03:49.8134082Z at org.sonarsource.scanner.cli.Main.execute(Main.java:112)
2022-08-10T14:03:49.8134617Z at org.sonarsource.scanner.cli.Main.execute(Main.java:75)
2022-08-10T14:03:49.8135138Z at org.sonarsource.scanner.cli.Main.main(Main.java:61)
2022-08-10T14:03:49.8135559Z Caused by: java.io.IOException
2022-08-10T14:03:49.8136093Z at com.sonar.cpp.plugin.cache.CacheInImpl.readExternalIssuesCacheName(CacheInImpl.java:200)
2022-08-10T14:03:49.8136879Z at com.sonar.cpp.plugin.cache.CacheInImpl.readExternalIssuesCacheName(CacheInImpl.java:190)
2022-08-10T14:03:49.8137279Z ... 35 more
Did both of these errors appear on the same configuration or on different ones? Could you provide more details on your setting, especially if and what kind of cache was used? You have mentioned that you tried to use both “SonarQube server-side cache” and “filesystem cache”.
Finally, where you able to confirm that any of above issues appears with cache disabled?
Yes, same configs, same system version, same SonarQube version, same sonar-scanner version. Different compile_commands and build and cache though.
With empty cache (which is similar to cache disabled I guess) I can’t confirm any failures yet.
We’re currently using the deprecated caching strategy, because neither the new filesystem cache, nor the server side cache appear to work. The sonar-scanner never finds a cache.
By having more frequent builds these errors happen less often now, which means that this is a cache invalidation issue, however they are not solved. When clearing the cache (not good, takes too long to analyze stuff), it usually doesn’t fail.
I’m guessing somebody should maybe analyze the code in sonar-scanner and look for ways that an older cache could lead to null pointers and buffer overflows. It seems not all corner cases are considered.
I assume the cache is per translation unit. One measure to mitigate this issue almost completely would build every translation unit with cache, and catch these failures, and then clear the cache only for this translation unit. That way we would only need to redo that small part of the whole project analysis. Barring solving all programming errors in the sonar-scanner this should give us 90% of the benefit for 10% of the effort.