C/C++ bugs detection performance

[francois14 (Francois A)]

Dear community

We are currently assessing the performances of the C/C++ code analyzer.
On the very simple test code below, it reports only bug 1 while other analyzers like cppcheck found all excepted bug 5

Are we simply misusing the tool or is the detection performance really that bad?

Thanks

#include <stdio.h>
#include <stdlib.h>
#include <strings.h>

int main(void)
{
            char *ptr;
            char buffer[1];
            
            *ptr=0; // Bug 1 (ptr is not initialized/allocated)
            
            ptr=malloc(1);
            
            strcpy(ptr,"1"); // Bug 2 (out of bounds due to \0)
            
            free(ptr);
            
            *ptr=0; // Bug 3 (ptr was freed)
            
            buffer[1]='0'; // Bug 4 (out of bounds)
            
            ptr=&buffer[1];
            
            *ptr=0; // Bug 5 (out of bounds)
            
            return 1;
}

[Abbas_Sabra]

Hello @francois14,

Welcome to the community!

I think you are misusing the tool in the sense that you are not using a real-life scenario.

All the issue that you are expecting relates to rules with symbolic-execution tag. Quoting the documentation:

symbolic-execution: this tag is for rules that reason about the state of the program. They usually work together to find path sensitive bugs and vulnerabilities. Once a fatal state of the program is reached, one issue will be raised and the symbolic execution analysis of the current path will stop. For that reason, it is not recommended to evaluate these rules independently of each other as it might give a false sense of undetected issues.

Now in your example, after the first bug, you reach an undefined behavior. the rest of the code will not be executed, so it doesn’t make sense to reason about the state of your program since it will not be reached.

If you want to make sure that these issues are detectable by the analyzer, I would suggest you split the example, put every undefined behavior/bug in a separate function. This way every bug can be reachable and they will all be reported.

Note: If you have two consecutive undefined behavior in your actual code the first one will be reported. The second one will be reported if you fix the first one without fixing the second.

Let me know if you have further questions

Thanks,

[francois14 (Francois A)]

Hi Abbas

Thanks for the clarification,

I understand the reasoning, however from a productivity perspective I think it would be preferable to not stop the analysis on the first such bug, as it forces in this case to go through 4 cycles of fixing / reanalysing before finding all the issues.
As I said, other analysers like cppcheck found 4 out of the 5 bugs in one single pass.

Also in real life, the software will not necessarily stop execution after the first bug. It depends really on what default value “ptr” gets (it will be 0 with most compiler but it is not a guarantee).

[Abbas_Sabra]

@francois14,

@francois14

preferable to not stop the analysis

Just to be clear, the analysis doesn’t stop. The analysis of one path stops.

@francois14

It depends really on what default value “ptr” gets (it will be 0 with most compiler but it is not a guarantee).

The stop depends on the type of the bug if it is fatal or not according to the language standard.

@francois14

it forces in this case to go through 4 cycles of fixing / reanalysing before finding all the issues.

I totally understand this point, it makes sense from the user perspective. At the same time doing that impacts the quality of the analysis as you are trying to analyze something just after an undefined behavior. On the positive side, you get all the issues from the first time. On the negative side, the analysis will not be as accurate since it is trying to deduce information about unreachable code/ the state of your program just after an undefined behavior. We chose a better quality of analysis over the inconvenience of multiple iterations in the rare case where fatal error happens immediately after each other.

Thanks,

[francois14 (Francois A)]

Thanks, it makes sense.

Indeed after splitting each bug into a separate function the analyzer found the 5 bugs, so finally better than cppcheck which found 4.

Great support, thanks