CI system used (Bitbucket Cloud, Azure DevOps, Travis CI, Circle CI
Azure DevOps
Scanner command used when applicable (private details masked)
Regular Devops tasks, nothing fancy
Languages of the repository
C++
Only if the SonarCloud project is public, the URL
And if you need help with pull request decoration, then the URL to the PR too
Error observed (wrap logs/code around with triple quotes ``` for proper formatting)
Steps to reproduce
Potential workaround
I am looking for advise on how to configure SonarCloud properly for the following scenario:
I have a code base here (C++23, hosted on Github) which I am using SonarCloud for. Coverage and static analysis. The develop branch is reasonably clean, although it does contain a number of existing minor issues as well as coverage below our threshold of 80%.
Now we have configured a SonarCloud quality gate so we can incrementally increase develops quality. Each PR branch must adher to the quality gate settings or it can’t be merged. This works well so far but I have the problem, that the quality gate is also applied to develop itself. This means that even if the PR branch is clean (coverage above threshold, no new issues) the resulting merge commit still fails the quality gate because overalll develop isn’t quite there yet.
How would I approach this? I see two angles here.
Should I configure this in the Azure Devops Pipeline task? Right now it is simply
- task: SonarCloudAnalyze@2
… without any parameters. Should I add something so it only fails on short lived branches?
Configuration of the SonarCloud organization or project. Should I add some condition or something? I do want the analysis and everything, I just don’t want the pipeline to fail if develop blows the quality gate (which it will do for at least the forseeable mid-term future)
The pipeline will only fail if you explicitly tell it to. Here that would mean adding sonar.qualitygate.wait=true as an analysis parameter under SonarCloudPrepare. Without this, you can have a failing Quality Gate, but the build will simply proceed whether or not the Quality Gate is failing our not (if you use SonarCloudPublish, it will also add a little report on the build, but this doesn’t fail the pipeline)
Without changing anything from the defaults, you should be able to have a
Build that doesn’t fail when the Quality Gate fails
Block the merge of PRs when they don’t pass the Quality Gate
One final note:
How is your Quality Gate configured? In a perfect world:
Coverage on New Code conditions are applied to the Pull Request and Main Branch, and if all new code passes the Merge Request Quality Gate, you won’t break the Main Branch Quality Gate (on these conditions)
Conditions on Overall Code only apply to the Main Branch. You should consider not having these if you want to adhere to the Clean as You Code methodology.
So stated another way: if your Quality Gates are passing on the PR, your Quality Gates should be passing on the main branch. If it’s not working that way, I’d like to understand more.
Hello Colin,
thanks for your response. A few words in reply:
Here that would mean adding sonar.qualitygate.wait=true as an analysis parameter under SonarCloudPrepare .
This is already the case. We do wait for the analysis. In general, the pipeline fails for PR branches when the conditions are not met. So all this works, I just need it to not fail for develop even if it does have issues left, which it does.
How is your Quality Gate configured?
Well, I’m no expert in SonarCloud configuration. Here’s the config as it is visible to me:
There’s one quality gate defined for all our projects. It says:
Conditions on New Code
Conditions on New Code apply to all branches and to Pull Requests.
It lists a number of conditions such as coverage or reliability that all seem okay to me. Underneath there’s a list of projects it applies to, which include the one I’m talking about here.
I guess the problem is that there is no additional list of conditions for “Overall Code” as you say, right? It treats “develop” just like the PR branches and therefore fails, correct? When you say Main Branch, are you referring to master? Because strangely, it does run through on master, even when it is at the same state basically as “develop”. Only develop always fails. Does this make sense?
What I’m suggesting is that you can achieve this by not configuring sonar.qualitygate.wait=true (boom, no more failures of your `develop branch) and still have the desired effect of blocking the merge of PRs until the QG passes (using PR decoration and the branch policy).
I’m referring to whatever you have set as your main branch in Azure DevOps. Hopefully, that’s also your main branch in SonarCloud!
What you’re telling me, however, makes me think you might have the following setup in SonarCloud:
master - your main branch in SonarCloud, this may or may not be your main branch in Azure DevOps
develop a short-lived branch, only evaluating issues on new code
your PRs being analyzed
Is that right?
I think what you want is: develop defined as a long-lived branch, with its own New Code Period (not targeting any other branch as a reference).
The best way to get your project into this state would probably be to start over, immediately rename your main branch as develop, and make sure your branch pattern is configured correctly to detect long-lived branches.
I made a few assumptions here, so let me know if I"m completely off-base.
thanks again for the reply. Here’s some thoughts from my end.
What I’m suggesting is that you can achieve this by not configuring sonar.qualitygate.wait=true (boom, no more failures of your `develop branch) and still have the desired effect of blocking the merge of PRs until the QG passes (using PR decoration and the branch policy).
This sounds a bit off to me to be honest. It sounds very hacky to realize something that I would consider basic functionality that you also promote with the Clean-as-you-code page. To be honest, that’s what puzzles me about the whole situation. I’m doing nothing fancy here, all standard by the book and as you promote it and still it seems a long way to go.
When I understand this correctly, not waiting for the results of the analysis would always be a bug because the decision if something should pass the gate should be up SonarCloud and not up to the pipeline not properly synchronizing the results, no?
Also why would a PR branch block when I’m not waiting for the results anymore? Wouldn’t it always pass as well?
About the long and short lived branch settings:
This is all standard git-flow.
develop is considered the default branch on github (where the source is hosted). In Azure devops, the pipeline is triggered by this:
In SonarCloud it shows “develop” as the main, long lived branch. My team is currently experimenting with the sonar.projectVersion settings to hard wire the version we are supposed to be at but I abhor such additional complexity and would love to avoid having to change something with every version just to make the pipeline pass.
I think that your problems lies in the definition of the new code.
On a PR that is simple, you have nothing to define, the new code is all the code added in the PR.
On a branch, like your develop branch, it’s a bit more subtle. You have to define what is new code, and how you define it is important because your Quality Gate, defined on criteria on new code, may pass or fail based on this new code definition.
I understand your point of having the most hassle free configuration, but sorry, you must really understand this part and configure it the way that is suitable for you (on a per project basis) .
On SonarCloud you have 2 ways to define new code:
New version: All the code that is added since a change of project version is considered new code. This is the best configuration BUT requires discipline in your project versioning;
The project version must be passed in sonar.projectVersion at every scan.
The version must be changed only when you release, i.e. at a moment where the quality gate on that branch passes.
OK so what if you never release that branch, which may be the case for your develop branch ? 2 options:
You still pass a “virtual” version and you bump the version even if you don’t release, but only when the quality gate passes, so that you start a new new code period only when the previous one is clean
You fallback to the 2nd way of defining new code: A number of days. This will not apply to your PR that will always apply criteria on the PR new code. This can be an acceptable solution, but you have to understand the consequences: If something that causes the QG on new code to fail is left unresolved for enough time (beyond the number of days defined for the new code period), it will go out of the new code period and therefore will no longer fail the quality gate. Because of that, in your case, I would recommend to set a sufficiently long new code period, like 60 days.
Last point: Why not keeping an original beginning of new code forever (like when you started applying clean as you code) ? For 2 reasons:
Because SonarCloud constantly improves and add new rules that may find problems that were not found before. Your new code may have no issues at a given point in time, but due to new rules, suddenly have several issues triggered by these new rules, even if the code has not changed.
This is OK if the new code is not too much code, so you can fix on the fly the newfound issues, This does not work if the new code has accumulated for too long, represents a lot of code, and therefore can suddenly have plenty of new issues due too new rules (too much to fix to pass the QG).
Because this will prevent you to raise the bar on your quality gate if you want to. Let’s say that at some point you want to enforce 85% of code coverage instead of 80%. On a moderate size new code, it’s only a little extra effort for the devs, but if the new code is a ton of code, improving from 80% to 85% becomes a significant and disruptive burden for developers.
Hello Olivier,
thanks for your response. We did indeed not use sonar.projectVersion up until now, which we figure might be the issue.
To realize your suggestions we are in the process of switching to the following procedure:
We use git describe --tags --always --abbrev=0 to generate a projectVersion that is always provided to SonarCloud. On master that would reflect the release tag that is being set (the same as without --abbrev as there is no commit diff hash). As develop receives merges from PR branches that would mean that until we do the next release (towards master) this version is constant and always the last release. As far as I understand this would satisfy the conditions, right?
In any case, I just wanted to let you know how we progress. Feel free to comment in case this is not the way to go.
.