I wanted to confirm quality gate behaviour for short-lived branches. It seems the short-lived branches do not adhere to the Quality Gate assigned to a project. Do the quality gate profiles only apply to long-lived branches?
For example, a Quality Gate for one of my projects contains a rules to throw warnings instead of a errors. However the sonar quality gate in my gitlab pipeline will fail (on a short-lived branch).
SonarQube analysis indicates that quality gate is failed.
Open Issues is failed: Actual value 37 > 0
Reopened Issues is passed: Actual value 0
SonarQube analysis reported 37 issues
2 critical
11 major
19 minor
5 info
The current branching implementation uses a hard-coded âminiâ Quality Gate (No open issues) for short-lived branches. So this is currently fixed and cannot be changed.
So you are right on below:
âDo the quality gate profiles only apply to long-lived branches?â
Yes.
Thank you for the clarification. The expectation for PR analysis is to consider the assigned quality profile for the project. Can you please give a hint when this will be supported.
Careful with the wording here: Quality Profile used is the same, what youâre enquiring about is Quality Gate. Thereâs no immediate plan to enable full-blown custom Quality Gates on Pull Requests, we rather believe in making sure that no bug/vulnerability/code_smells slips through the cracks, allowing the dev to assess any issue (with the ability to only acknowledge if really needed) before merging.
Hi Nicolas,
sorry for the confusion. Of course I wanted to refer to the quality gate which is configured for the project.
The info about the support of the code coverage for PR is great .
Will it also be possible to tweak the âhard codedâ quality gate for PRs and filter issue severity e.g. ignore âInfoâ ?
This means, if you want to get a green status on the PR, you can either fix the issues for real or âConfirmâ, âResolved as False Positiveâ or âWonât Fixâ all the issues available on the PR.
To be approached with care. This shouldnât be used to cheat obviously, itâs only SonarQubeâs way to let the ultimate decision in the devâs hand, as there can be human/context-specific considerations that apply.
I understand and this workflow is great for green field projects. We use SonarQube many years and used the issue severity to highlight important issues. This is now not possible anymore with the hard-coded quality gate for PR analysis and we disabled the external approval of PR analysis.
Updating the quality profiles for all languages and remove certain severities is not an option. Use a derived quality profile is also not possible because an activated rule cannot be disabled or removed.
Today I can only create copies of the quality profiles and remove the rules which shall not be applied for PR analysis. The developer shall not acknowledge issue which we will fix in another refactoring task (typically info or minor issues). I am not sure whether this the right way and it creates some effort as well.
It is wonderful that you believe that. I happen to believe that too. But the business has deadlines to meet that have marketing money on the line and coordinated product launches that sometimes cannot afford to be held up because someone commits code that has an informational issue raised in it. Or maybe even there isnât a deadline but it is a prototype that is quickly evolving and quality isnât the concern so much as proof of concept. Either way the organization should be able to make their own decisions about what quality is acceptable to them for all branches, short or long term. Enforcing a draconian zero leak policy that cannot be modified is ridiculous. The alternative is requiring an organization to increase their efforts maintaining their CI process with one off exceptions to quality gates or modifying rule sets rather than managing the actual quality gate at a global level to suit their ability to absorb technical debt while still meeting their organizational goals.
Assuming that a zero leak policy is even desirable, let alone feasible for everyone is short sighted and reduces the value of your own product in helping everyone improve their own products.
Personally we have had to turn off all quality gate checks because the zero leak policy became too detrimental to the business. It is horrible to not have an automated mechanism to hold developers accountable, but it is worse to hold them accountable to a level that loses money.
I fully agree with the statement from Joe Swanson.
Our SW architects complained about the PR annotation and I disabled this feature for the current milestone activities. I really would like to see an issue filter for rules or severity.
I find it unfortunate that this old topic was resurrected with such an aggressive tone. Maybe itâs not easy to find if you donât know the right key words, but this question has been asked - and answered - several times in this community in the last few months:
MMF-1369 - Real Quality Gates for PRs and short-lived branches
The goal of which is to enable you to âmake [your] own decisions about what quality is acceptable to [you] for all branches, short or long termâ. Feel free to watch and vote for this initiative, which is currently tentatively scheduled for 2019Q1.
In the meantime, yes there is a hard-coded gate on PRs and Short-Lived Branches. At initial implementation our choice was doing that or doing nothing. We picked the former as the lesser of two evils, especially since we donât impose a zero leak policy, we just make sure youâre aware of what youâre doing when you add issues in the leak. As described above,
To be super explicit about this, if your PR has no Status=Open issues, then it goes green, even with non-fixed issues. And it doesnât just go green in SonarQube; PR annotation is updated as well when the last Open issue is moved to some other status. So, to not be blocked by âa draconian zero leak policy that cannot be modifiedâ, simply âConfirmâ all your Open issues.
If youâd like a civil discussion of other pains you have around this topic, please feel free to start a new thread.
I read through this thread because it came through on the digest. Nowhere do I see an aggressive tone: Iâm not sure where you are getting that idea.
I too fought with this issue on one of the previous threads you are mentioning. And I eventually did find the documentation that has been pointed out here. I also remember that it was not easy to find this explanation. Personally, I think the reason is that the workflow SonarQube has in their mind is somewhat⌠unique. (Which causes one to not look in the right places, because we arenât using the same logic.) It really doesnât jive with any workflow Iâve been a part of. Now that I understand it, we work with/around it, but the most aggressive thing Iâve seen is SonarQubeâs stubbornness to realize they may have missed the use case here, and instead, are standing firm.
Iâm still not sure why we would have Quality Gates for after the fact analysis when the default âbefore the factâ analysis is for all 0âs. Unlike what was mentioned earlier, that seems that would only work for projects that are not greenfield. But even then, how could a Quality Gate get worse if you have to fix or mark everything with some type of resolution?
Could you clarify which documentation youâre talking about here? And where/how you think it would be more obvious to put it?
If by this youâre referring to the red/green status on SLBs and PRs, weâve said pretty much from the beginning that we want to get to the place where users can control what tips it one way or the other.
Thatâs an excellent question, and one weâve asked internally as well. When we do introduce user-controlled Quality Gates for SLBs and PRs itâs possible weâll take action on it, but right now that would probably be premature.
What is your understanding of what happens to issues that are marked Confirm in a PR?
Thank you for the summary and the MMF-1369 reference.
Unfortunately the scope of our quality profiles were created without the 0 issue Quality Gate policy in mind and I really want PR analysis with annotations enabled for all of our projects.
I will refine our quality profiles for PRs and use it as default quality profile. I will remove all rules which have severity âINFOâ, âMINORâ because we think comments for this kind of issues shall be avoided.
Nevertheless we want to see the issues for âINFOâ, âMINORâ severity rules for our code base every day but not for pull request analysis.
How can I achieve this? I cannot define a second quality profile which is used for our nightly builds. The project configuration has only one quality profile per language.
Maybe there is a configuration which I am not aware of.
And where/how you think it would be more obvious to put it?
This is a difficult question, as the manual sections are influenced by the workflow, and the workflow really doesnât jive with me. Maybe in the Overview section put a diagram of the designed-to workflow would then help people enter further into the documentation.
What is your understanding of what happens to issues that are marked Confirm in a PR?
My understanding has expanded based upon your previous answer in this thread, and I think it confirms my assertion of the awkwardness of this workflow. What you describe as âConfirmâ sounds like it would be worded something like âAbsorb as technical debt to be resolved in the future.â Confirm, on face value, without explanation, doesnât imply that at all.
When my team was presented with the choices:
âConfirmâ
âResolved as False Positiveâ
âResolved as Wonât Fixâ
âAssignâ
âCommentâ
it was very confusing. They donât seem mutually exclusive, confirm doesnât seem related to the next two (now we know why), and we are left wondering how to properly use them. As far as I can tell, this is attempted to be explained in Pull Requests and Issues. But the two pages use different terminology and does not clarify proper actions to the reader.
My take away is that SonarQube doesnât seem to trust itself and wants a human to either tell it, yes, you are right (confirm) or no, you are wrong (false positive) or yes, you are right, and I donât care (wonât fix) or yes, you are right, Iâm going to go fix that right now (assign). I would think SonarQube trusts that it is doing its job unless I tell it otherwise⌠the value of âConfirmâ is unclear.
I suppose our expectation/understanding of the âConfirmâ option is an outgrowth of our history using SonarQube. In that context, âConfirmâ has always meant just what you describe: âAbsorb as technical debt to be resolved in the future.â
We have had some internal discussions about renaming it, but couldnât come up with anything both terse and accurately descriptive.
Iâd say your characterizations of Wonât Fix and False Positive are correct. For assign - which to be clear is independent of for example choosing Confirm in the Actions menu - whether the fix is immediate or not is about working agreements within a team. By default issues should be assigned to their creators* so explicit assignment shouldnât generally be needed.
For Confirm the meaning is really âyes this is an issue, but Iâm not going to fix it nowâ. Itâs not a question of SonarQube trusting itself, but of whether you as a developer/team want to fix the issue before merge or move forward despite its presence. To flip the PR status to green despite having non-fixed issues in the code, you Confirm the remaining issues to acknowledge them and indicate that theyâre acceptable for now.
Ann, I too came across this thread because of the digest notification. Without getting into the issues surrounding quality gates for PRs, I would echo the comments regarding the documentation about the impact of confirming an issue. Until I read this thread, I had no idea that confirming an issue would count toward eliminating it as an open issue for the PR. I think the problem is that since the issue is still âopenâ (i.e. not resolved), itâs not clear that this resolution removes it from the open issue count for the PR. Iâm guessing thatâs documented somewhere, but it might be worth explicitly calling it out in the PR decoration docs.
In the context of PR and SLB status, âOpenâ refers specifically to the issue status rather than whether itâs fixed or not in code.
Re documentation, FYI (emphasis added):
When âConfirmâ, âResolved as False Positiveâ or âWonât Fixâ actions are performed on issues in SonarQube UI, the status of the PR is updated accordingly. This means, if you want to get a green status on the PR, you can either fix the issues for real or âConfirmâ, âResolved as False Positiveâ or âWonât Fixâ any remaining issues available on the PR.
It is possible to change the status of a short-lived branch from ERROR to OK (red to green), i.e. mergable, by manually confirming the issues. The same is true for the False-Positive and Wonât Fix statuses. It means the status of a short-lived branch will be red only when there are Open issues in the branch.
For easy reference, you can find these docs embedded in your SonarQube instance under the â?â in the main menu. Some docs are available before 7.4, but with 7.4 we completed the port of documentation into SonarQube itself.
.