SonarQube support for .h files machine generated by ROS during compilation

I’m using SonarQube to analyze code written primarily in ROS’ C++ interface. A part of ROS’ build system involves converting .msg files into .h files and placing them in a dedicated installation directory. So after building, a file that was in the following location during compilation:

/src/package1/custom_msgs/msg/CustomMessage.msg

Will create an auto-generated .h file in the following location, able to be referenced through Catkin makefile magic:

/install/custom_msgs/include/custom_msgs/CustomMessage.h

My current issue is that SonarQube doesn’t know about the .h file during its scan and ends up with hundreds of warnings in the log to the tune of:

'custom_msgs/CustomMessage.h' file not found

And as a result of those missing files, the SonarQube analysis is severely crippled.

My question is, is there a way to give SonarQube information about the header file’s contents or new location? Preferably automatically since these messages are added and moved pretty frequently.

Hi,

I have to ask the dumb question: Are you using the build-wrapper? I ask because it seems like those header files would be found during compilation & thus the build-wrapper, and by extension analysis, would know where they are…?

 
Ann

Yes, but I’m an amateur with Sonar so I’ll lay out exactly what I’m doing.

First, I build the entire codebase with colcon and make sure to include the buildwrapper:

echo "START BUILD COMMAND"
configuration/SonarQube/build-wrapper-linux-x86/build-wrapper-linux-x86-64 --out-dir bw-output colcon --log-base log build --base-paths . --build-base build --install-base install --event-handlers console_cohesion+ --cmake-args -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -G Ninja -DCMAKE_BUILD_TYPE=Debug
echo "END BUILD COMMAND"

Next, I run the sonarscan command:

configuration/SonarQube/sonar-scanner-4.7.0.2747-linux/bin/sonar-scanner -X -Dproject.settings=configuration/SonarQube/sonar-project.properties

The supplied property files is as follows, with sensitive information removed:

# must be unique in a given SonarQube instance
sonar.projectKey=XXXXX
sonar.sources=src
sonar.host.url=XXXXX
sonar.login=XXXXX
sonar.cfamily.compile-commands=build/compile_commands.json
sonar.exclusions=**/*.doc,**/*.xml
sonar.cfamily.build-wrapper-output=bw-output
sonar.cfamily.threads=6

#Disable caching
sonar.cfamily.cache.enabled=false
 
# Encoding of the source code. Default is default system encoding
sonar.sourceEncoding=UTF-8

And then the resultant SonarQube analysis is on the web GUI where I’d expect it to be, where it’s right enough to prove it has most of the insight I’d expect it to have, but wrong often enough that I think it’s missing critical insight. These missing .h files seem like a likely culprit. See my linked issue for more details on the actual issue I’m trying to solve.

Hi,

I’m not seeing colcon in the list of supported compilers. Does it fall into one of the broad support categories like “based on GCC”?

 
Ann

If I’m following your question, Colcon isn’t a compiler. It’s a command line tool to improve the building workflow. We use it to cleanly build ROS code.

https://colcon.readthedocs.io/en/released/

colcon is a command line tool to improve the workflow of building, testing and using multiple software packages. It automates the process, handles the ordering and sets up the environment to use the packages.

Okay!

The build-wrapper needs to wrap… the build. The actual compile command. I think this is the crux of the problem.

 
Ann

I’m not sure I understand what exactly I need to wrap. The colcon command isn’t a compiler, but it runs the thousands of auto-generated compilation commands like the following (pulled from the archived compile_commands.json)

"/usr/bin/c++  -DGTEST_CREATE_SHARED_LIBRARY=1 -Dgmock_EXPORTS -I/usr/src/googletest/googlemock/include -I/usr/src/googletest/googlemock -I/usr/src/googletest/googletest/include -I/usr/src/googletest/googletest  -g -fPIC    -Wall -Wshadow -DGTEST_HAS_PTHREAD=1 -fexceptions -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -o gtest/googlemock/CMakeFiles/gmock.dir/__/googletest/src/gtest-all.cc.o -c /usr/src/googletest/googletest/src/gtest-all.cc"

So all of the compilation is happening in the colcon command that I have wrapped by the build wrapper, and the individual commands should be supported build commands. It’s similar to cmake/make/meson: not technically a compiler, but very close to the operation.

Hi,

I’m a bit out of my depth here. Calling in more expert eyes…

 
Ann

Hello @jrl2n4,

In your config file, I can see you specify both the compilation database (sonar.cfamily.compile-commands) and the output of the build wrapper (sonar.cfamily.build-wrapper-output), but only one is needed.

Can you try with only the compilation database? If you still get the same error, sharing compile_commands.json and the debug logs of the scanner would be useful. If you don’t want them to be public, I can send you a private message.

I attempted running with the same command and properties file with the build-wrapper-output line commented out and that resulted in this error message explicitly asking me to use the build wrapper:

15:44:43.925 INFO: ------------------------------------------------------------------------
build	08-Feb-2023 09:44:43	15:44:43.925 INFO: EXECUTION FAILURE
build	08-Feb-2023 09:44:43	15:44:43.925 INFO: ------------------------------------------------------------------------
build	08-Feb-2023 09:44:43	15:44:43.926 INFO: Total time: 21.127s
error	08-Feb-2023 09:44:44	15:44:44.120 ERROR: Error during SonarScanner execution
build	08-Feb-2023 09:44:44	15:44:44.120 INFO: Final Memory: 144M/487M
build	08-Feb-2023 09:44:44	15:44:44.120 INFO: ------------------------------------------------------------------------
error	08-Feb-2023 09:44:44	java.lang.UnsupportedOperationException: 
error	08-Feb-2023 09:44:44	
error	08-Feb-2023 09:44:44	The only way to get an accurate analysis of C/C++/Objective-C files is by using the SonarSource build-wrapper
error	08-Feb-2023 09:44:44	and setting the property "sonar.cfamily.build-wrapper-output", but it was not specified.
error	08-Feb-2023 09:44:44	
error	08-Feb-2023 09:44:44	If you don't want to analyze C/C++/Objective-C files, then prevent them from being analyzed by setting the following properties:
error	08-Feb-2023 09:44:44	
error	08-Feb-2023 09:44:44	    sonar.c.file.suffixes=-
error	08-Feb-2023 09:44:44	    sonar.cpp.file.suffixes=-
error	08-Feb-2023 09:44:44	    sonar.objc.file.suffixes=-
error	08-Feb-2023 09:44:44	
error	08-Feb-2023 09:44:44	
error	08-Feb-2023 09:44:44		at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:210)
error	08-Feb-2023 09:44:44		at com.sonar.cpp.plugin.CFamilySensor.execute(CFamilySensor.java:177)
error	08-Feb-2023 09:44:44		at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:48)
error	08-Feb-2023 09:44:44		at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:85)
error	08-Feb-2023 09:44:44		at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:62)
error	08-Feb-2023 09:44:44		at org.sonar.scanner.scan.ModuleScanContainer.doAfterStart(ModuleScanContainer.java:82)
error	08-Feb-2023 09:44:44		at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
error	08-Feb-2023 09:44:44		at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
error	08-Feb-2023 09:44:44		at org.sonar.scanner.scan.ProjectScanContainer.scan(ProjectScanContainer.java:392)
error	08-Feb-2023 09:44:44		at org.sonar.scanner.scan.ProjectScanContainer.scanRecursively(ProjectScanContainer.java:388)
error	08-Feb-2023 09:44:44		at org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:357)
error	08-Feb-2023 09:44:44		at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
error	08-Feb-2023 09:44:44		at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
error	08-Feb-2023 09:44:44		at org.sonar.scanner.bootstrap.GlobalContainer.doAfterStart(GlobalContainer.java:150)
error	08-Feb-2023 09:44:44		at org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
error	08-Feb-2023 09:44:44		at org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
error	08-Feb-2023 09:44:44		at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72)
error	08-Feb-2023 09:44:44		at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66)
error	08-Feb-2023 09:44:44		at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
error	08-Feb-2023 09:44:44		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
error	08-Feb-2023 09:44:44		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
error	08-Feb-2023 09:44:44		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
error	08-Feb-2023 09:44:44		at java.base/java.lang.reflect.Method.invoke(Unknown Source)
error	08-Feb-2023 09:44:44		at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
error	08-Feb-2023 09:44:44		at com.sun.proxy.$Proxy0.execute(Unknown Source)
error	08-Feb-2023 09:44:44		at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
error	08-Feb-2023 09:44:44		at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
error	08-Feb-2023 09:44:44		at org.sonarsource.scanner.cli.Main.execute(Main.java:112)
error	08-Feb-2023 09:44:44		at org.sonarsource.scanner.cli.Main.execute(Main.java:75)
error	08-Feb-2023 09:44:44		at org.sonarsource.scanner.cli.Main.main(Main.java:61)

Before I check with my people to see if I can even send the commands file to you, can you confirm that I should be able to even do this? The error message seems pretty explicit when it says that I cannot.

Sorry, you were using SonarQube LTS 8.9.10, right? The bundled C++ analyzer (6.20) does not support the compilation database.

To better understand why the analyzer is missing those headers we need to have a look then at the files build-wrapper-dump.json and build-wrapper.log stored inside bw-output.

Sorry for the delay, I wasn’t able to secure permission to share our wrapper logs due to the nature of the project being worked on. However, I did experiment and determined that moving the code to an install directory in the build command appears to cause the issue. Details are here: SonarQube support for --install flag in catkin/colcon