Quality Gate changes after re-analyzing without issues being fixed

[JuMi (Julian)]

Hi all,

a dev colleague reported a strange behavior regarding PR analysis.
After an analysis which failed the quality gate, a second analysis passed the quality gate even though they didn’t change anything (see screenshots). The code is getting analyzed during a Azure DevOps build and they’re using the build breaking feature (qualitygate.wait)
We’re using SonarQube 9.9 LTS, the project’s language is C/C++, new code period is set to ‘previous version’.
Maybe there’s some reasonable explanation for this behavior, or could it be a bug?

Thanks and best regards,
Julian

[Colin]

Hey there.

How is your Quality Gate configured? What conditions were failing on the first run that were no longer failing on the second?

[KUGA2]

I am taking over here.

The headline is pobably missleading. The quality gate itself does not change. But It does not fail on the second analysis.

The quality gate is:

I can see in the logs, that the second time, the file in question (with issues) has a cache hit.

[Colin]

And on what conditions do you expect it to fail? Can you share a screenshot of your project dashboard where you expect to see a failing Quality Gate?

[KUGA2]

Hi Colin,

i finally found the time for a good answer.

We are running SQ as PR verification (with sonar.qualitygate.wait=true).
When I create a PR with new changes (bla1.cpp was changed with new sq issues), the first run always fails.
It says:

INFO: Using 16 threads for analysis, according to value of "sonar.cfamily.threads" property.
INFO: Found empty cache on server
INFO: [pool-2-thread-1] C:\b01\_w\13\bla1.cpp
INFO: [pool-2-thread-3] C:\b01\_w\13\bla2.cpp
INFO: [pool-2-thread-2] C:\b01\_w\13\bla3.cpp
INFO: [pool-2-thread-4] C:\b01\_w\13\bla4.cpp
...
ERROR: QUALITY GATE STATUS: FAILED - View details on -url-

(I am not sure why is says empty cache here. bla2-4 are not changed and should have a cache hit, but this is not of concern here!)

The build step becomes red.

But runs after that (not all, but some) do not fail. I did not change the code, I only started the build again. The log looks different:

INFO: Loading cache from: server
INFO: Cache hit for: C:\b01\_w\3\bla1.cpp
INFO: Cache hit for: C:\b01\_w\3\bla2.cpp
INFO: Cache hit for: C:\b01\_w\3\bla3.cpp
INFO: Cache hit for: C:\b01\_w\3\bla4.cpp
...
INFO: QUALITY GATE STATUS: PASSED - View details on -url-

So my suspicion is, that the reason for this are the cache hits.

I will send you the full log in private so you can have a better look.

[cba (Chris)]

This may be a red herring, but: Is your quality gate configured to look only at new code, and have you configured your new code definition to be based on Previous Version? Your second run shows a version number, the first one does not. It’s possible that SonarQube sees all the existing issues as legacy code, and finds no issues in your “new” code, hence the quality gate passes.

What version number are you passing on to SonarQube with your PR scan configuration?

[KUGA2]

Hi Chris,

yes it is set up to “New Code”.

The version is always:

list(APPEND SCANNER_OPTIONS_LOCAL "-Dsonar.projectVersion=${PROJECT_VERSION_MAJOR}.${PROJECT_VERSION_MINOR}.${PROJECT_VERSION_PATCH}")

This version does not increase (no build counter). The version is 4.0.0 and I cannot find this in my logs, so what are you talking about?

Also this is set up:

list(APPEND SCANNER_OPTIONS_LOCAL "-Dsonar.newCode.referenceBranch=${REFERENCE_BRANCH}")

And it points to some branch running nightly.

[Colin]

Hey @KUGA2

I have asked a few times now on what conditions does the Quality Gate initially fail?, that it stops failing on the following runs?

[KUGA2]

This is the reason why the quality gate fails on first run.

Obviously this is an issue that I added for testing.

[Colin]

That is an issue being raised, but I am interested in the exact condition.

[KUGA2]

Ok, i have to reproduce this. I cannot see the condition on the old failed one as it is overwritten with new passed ones.

[KUGA2]

Here you go:

Do you need further info?

[Colin]

Thanks. A few things:

• It should be clear that this is not PR analysis (as the original post suggested), but branch analysis. Is this intentional?
• For giggles, can you see what happens when you stop passing sonar.projectVersion entirely? I think this will help us determine if it’s the cause of resetting the New Code Period.
• Can you also try setting the reference branch in the UI rather than via the command line? This again helps us eliminate/point to potential issues.

[KUGA2]

• This is correct. We do a regular branch analysis during PR builds. (Because we have multiple architectures/compilers and you can only build one as a PR analysis. I think this feature is requested somewhere.)
• Ok, will take some time though, I am on different Project now
• How? What UI?

[Colin]

@KUGA2

• How? What UI?

The project-level Project Settings > New Code