I wanted to see if the internal verification methods/results for SonarQube are available for customers to see? Or if there are ways for customers to verify rules are working as expected?
Let’s say I want to scan C/C++ code and make sure it is compliant with the latest MISRA coding standards (excluding MISRA 2023 as I believe that is not fully implemented in SonarQube yet). While I can easily search through any issues with my code for issues tagged with a “MISRA” tag, how has SonarQube been validated/verified internally to make sure that its checks are compliant with that standard?
The reason I ask is because I am working on a product that has some very specific verification/validation requirements. Any tools I use will need to be verified.
You might not be very surprised if I tell you we take these things very seriously.
How C++ rules are made and continuously improved ?
First, the developers implementing those checks are not only very skilled C++ developers, they are also very expert in the language. When it comes to our own rules, they design these rules and all the special and corner cases. They obviously don’t alter any MISRA rule and they know what to look at to make the implementation as thorough as possible.
Second, when our developers implement these checks, they write some extensive unit tests for all sorts of cases. Then, all rules are tested as we develop them multiple times on more than 200 notorious OSS projects. It allows spotting some cases that might have been missed in the initial implementation. We loop this process multiple times if needed. As a result, the final rule implementation is already pretty solid.
Still, things might go under the radar and thanks to our great community and customers, we get feedback on our rules’ implementation and put it in our backlog, False-Positives and False-Negatives.
These collected defects are fixed in very regular hardening sprints we carry out on our analyzer.
What verification can be provided or demonstrated
There are 3 points about this I want to mention.
At the moment, we don’t openly provide any material in this regard. As you may know our C and C++ analyzer is closed-source, so you cannot see our unit tests. If this is a major problem for you, I recommend you contacting the support.
We are considering developing some form of MISRA tool qualification kit to help our users check that our implementation does what it is supposed to do according to MISRA compliance.
We are also considering getting SonarQube Server certified with ISO 26262 and IEC 61508. It would require demonstrating some of this verification to a certification authority, making it less of a burden for our MISRA users.