Quality gate not failing in pull request

  • Using:

    • SonarQube version 8.9.1.44547
    • Azure DevOps Server Version 17.143.28621.4 (2019 on premise)
    • SonarScanner for Azure DevOps Version 4.21.0
    • Mule SonarQube Plugin v1.0.4
  • We are attempting to test a quality gate failure to prevent pull request completion and merge, however our quality gate never seems to fail.

  • Rule we are attempting to violate:

  • Code added which should trigger vulnerability:

  • Quality gate shows 0 new vulnerabilities, meaning the gate responds passed:

  • Therefore the quality gate is shown as passed in Azure DevOps (we have ALM integration set up).

However, due to adding code which should trigger the rule, we would expect the quality gate to fail with 1 new vulnerability. Can you see why this is not happening?

Hi @timothyconnolly5,

This looks like a potential false negative with a rule from the Mulesoft plugin. This plugin isn’t developed or maintained by SonarSource, so you should direct this question to the plugin developers.

Regards,

Cameron.

Hi Cameron,

I don’t believe the issue is with the 3rd party plugin as the rules are flagged when doing overall code analysis on develop/master branches.

It only seems to fail to find the issues when analysing new code in pull request branches.
In the example originally shown, I introduced a line of code which should have shown as a vulnerability.

Regards,
Tim

Hi Tim,

So that exact line of code raises an issue in the develop branch but not in a PR?

Regards,

Cameron.

Hi @Cameron ,

that’s correct.

Regards,
Tim

Hi Tim,

That’s definitely not normal behaviour: if a line of code raises an issue in develop then it should also be raised in the PR. Do you still have the scanner-side logs for the PR analysis on PR 981? I’d also be interested to see the 3 warnings from that analysis (from the warnings link in the top right corner of your PR analysis page).

Regards,

Cameron.

Hi @Cameron ,

sure, warnings seen below image:

image

I do, and you can see the validation result as false for the specific file (starting line 3162)

Regards,
Tim

Hi Tim,

Thanks for the extra info. No relevant material in the warnings and (although I’m not familiar with the Mulesoft plugin) it does look from the logs as though the issue should be raised.

Looking at the plugin repo, I’m a little concerned that it’s listed as being testable against SonarQube 7.7. There were some changes to branch and PR analysis across the SonarQube 8.x series and it’s not clear whether the plugin was tested with the 8.9 LTS. Does the plugin raise any issues on your PRs? Or is it only this vulnerability that’s missing?

Regards,

Cameron.

Hi @Cameron ,

ah good spot, didn’t notice that. I will raise an issue on the repo.
The plugin currently doesn’t raise any form of issue on PR.

Regards,
Tim

Hi, I am experiencing the described problem on Sonarqube Enterprise Edition Version 9.7.1 (build 62043):

I have a Quality gate defined (only overall-code metrics; no metrics on new code)
The integration ADO<->Sonar works, the PR receives a comment with a serious error (which in the SONAR UI is even shown as BUG (reliability rating C which breaks my QG))
BUT the resulting QG for that Branch remains green and only when merged to master turns red.

I would expedt QG check for a PR to
a) BREAK: If newCode is defined and the content of the PR (being new code) breaks those restrictions
b) BREAK: If overall is defined and the content of the PR is the reason (e.g.one additional fail) for an overall-restriction on the merge result to fail ← this is the current “Problem”
c) Succeed: if newCode restriction passes and there is no overall defined
d) Succeed: if newCode restriction passes and overall passes for the merged code
e) UNDECIDED: if newCode passes and overall restrictions have failed on master (so the PR does not make it worse) … I tend towards FAIL: but this I would be willing to accept