Trying to achieve: measure C++ code coverage of files which have only declarations and constexpr statements
So far, I am running a code coverage tool for C++ (Bullseye) and importing the XML export of that tool to SonarQube via sonar.cfamily.bullseye.reportPath. This works in principle; most coverage information is correct. However, there is one thing which does not work as expected: for source files which contain only definitions and constexpr statements (say, certain header files), SonarQube requires code coverage data for precisely the constexpr statements, and hence the files. This is a problem, since these statements are evaluated at compile time. There is no “coverage” possible at runtime, since the code is basically never generated. They are constants. The coverage tool does not generate any coverage data, like for other header files which do not contain constexpr statements as well. For these files, SonarQube correctly does not require coverage data.
This is a problem now for us: SonarQube requires coverage data for files which cannot be covered by tests at runtime. Due to regulatory requirements, we have to set our code coverage goal to 100% (which may include well-explained exclusions for certain lines of code, so it’s realistic to achieve).
Is it not a wrong requirement which SonarQube makes here? Requiring code coverage for constexpr statements?
I believe this may be an issue with the tool that generates the coverage data. For example, before Clang 18, the compiler used to produce incorrect coverage data. FWIW, their release notes say this:
Clang now emits correct source location for code-coverage regions in if constexpr and if consteval branches. Untaken branches are now skipped. Fixes (#54419)
It may be worth checking if there is an update for Bullseye that fixes the same problem, or possibly report the problem on their side.
Well, this is now the classic - Bullseye say it’s probably the fault of SonarQube and vice versa (I reported this also to them). I can see both points. However, I am wondering what is the rationale for SQ to require coverage for a file for which it usually would not. For example, where is the difference between a header file that does not involve constexpr and header files that do - from an SQ perspective?
I am asking since for files that do not have constexpr, SQ does not complain about “missing coverage” - despite the fact that the files are not in the coverage data as well (I checked). So there must be some kind of logic in SQ to determine which files require coverage in the first place and which do not.
SonarQube supports the reporting of test coverage information as part of the analysis of your C/C++/Objective-C project.
However, SonarQube does not generate the coverage report itself. Instead, you must set up a third-party tool to produce the report as part of your build process. You then need to configure your analysis to tell the SonarScanner where the report is located so that it can pick it up and send it to SonarQube, where it will be displayed on your project dashboard along with the other analysis metrics.
To paraphrase, SonarQube displays the data from the provided report without any “smart” adaptation. We leave this to the tools that generate the report in accordance with the users’ settings.
This also means files not included in the report will not be reported in SonarQube as covered/not covered.
Well, if it worked exactly like that I would understand it. However, it does not - as I wrote, SQ clearly checks the content. Concrete example: for a header file that contains no constexpr (let’s call this regular header file), SQ does not require coverage data to be present. The file is reported as “OK” from a coverage perspective, even if the coverage report from the tool (again, Bullseye in our case) does not contain any data for that file.
Don’t get me wrong - it is “kind of right”, of course. There are only declarations, no “executable code” in such files. However, it shows that there is “some kind of interpretation” going on in SQ about “what to cover.”
FWIW, digging a bit deeper, we found this piece of documentation describing when SQ considers on its own a line of code “executable”. This is probably historic, and can be understood why it exists, and is difficult to get right for “any language” - in the context of constexpr it kind of “backfires” now
