Error observed:
When a feature branch is build in CI ( Jenkins) the Quality Gates is green. But after merging this branch to the main line, the QG on the main line fails. This happens only for certain types of changes: e.g. when JUnit test code is upgraded from v4 to v5 other Rules are triggered, e.g. a Test class is no longer allowed to be “public”. If this code is not touched and the class remains public, the QG will not fail on the feature branch. The New Code definition on the feature branch is set to ‘develop’ (this is our long lived branch that feature branches are merged to). The New Code definition of the ‘develop’ branch is set to ‘Previous version’.
But the JUnit 5 rule (java:S5786) does get triggered when the feature is merged to ‘develop’ and fails the Quality Gate.
I would expect that this violation would already occur on the feature branch.
Note that the HEAD of ‘develop’ is merged to the feature branch before running the SonarQube analysis on the feature branch. As it should.
I think the problem is caused by a difference in New Code definition between the feature branch (with Reference branch set to ‘develop’) and the ‘develop’ branch. It seems like on the ‘develop’ scan all issues are found, but on the feature branch scan only issues on New Code are found and not on existing code that has new issues do to JUnit 5 upgrade (which is only a change in a Maven POM file).
Hello @Dick_Dijk
thanks for your post, and sorry for this delayed answer.
Let me try to rephrase the problem to see if I understood it well.
On a feature branch, a new issue on an untouched line of code is not raised in the New Code, which allows the Quality Gate on this branch to pass, but is raised as new in the target branch (develop) and therefore fails the Quality Gate there. Is that correct?
The example you provided with S5786 is specific in that it creates a situation which make this ‘blind spot’ possible:
the reference branch New Code period on the feature mandates the issue to be backdated as described with Understanding Issue Backdating. because the issue is detected In fact the S5786 code smell is raised, but in the overall code. Can you confirm?
then on the develop branch (after merge), issue backdating is not happening because it is not the first analysis of the branch, and the issue as new, and of course there is no issue tracking from feature branch to the reference branch, as there are with PRs.
So maybe issue backdating on the first branch analysis should not be done on branches with reference branch used as New Code Period. What do you think
There may be some other constraints that I can’t see right now so we’ll need to discuss about it. And I’ll keep you posted.
Meanwhile I can highlight that those issues are still caught before the project is released, which is the most important thing.
Thanks for your response. I can confirm your assumptions. It looks like backdating should not happen on the feature branch, but can I turn this off somehow?
And you’re right, we catch the issue eventually, but it blocks other PRs from merging until the SQ issues are resolved So it’s still a pain.
I was a little wrong yesterday when I confirmed your assumptions. Let me correct this:
You asked “On a feature branch, a new issue on an untouched line of code is not raised in the New Code, which allows the Quality Gate on this branch to pass, but is raised as new in the target branch (develop) and therefore fails the Quality Gate there. Is that correct?”
Yes this correct
But then, you asked:
“the reference branch New Code period on the feature prevents mandates the issue to be backdated as described with Understanding Issue Backdating. because the issue is detected In fact the S5786 code smell is raised, but in the overall code. Can you confirm?”
This is where I have to be more precise: The “Overall Code” tab (under Overview → Measures; see attached screenshots) is missing on our PRs, so I can not be sure that the issue is found in the PR scan. It is definitely found after the PR is merged. But I find it strange that the “Overall Code” tab is missing (only New Code tabs is show).
By the way, the “Overall Code” tab does get shown on a scan of the feature branch (see other attached screenshot).
I have attached 2 screenshots of a feature branch scan (Overall Code tab is shown). One green, one red.
And 1 screenshot of a PR, that does not have the Overall Code tab.
Hopefully this gives you an idea of what’s going on here.
By the way, this is a part of our Jenkinsfile that starts the SonarQube analysis. You can see that we treat a feature branch scan differently other than a PR scan.
Hello @Dick_Dijk
Indeed issues are handled differently between PR and branch analysis. You may read more on this topic with MMF-1994.
In a PR analysis, only issues attached to changed lines in the code will get raised, and indeed SonarQube has a specific display for PRs.
For project teams using PRs, PR analysis is the way to go with SonarQube. Some corner cases as the one mentioned in this thread, will eventually escape the PR analysis results, but these cases will remain rare and you will eventually catch all issues after merge.
Having only branch analysis should remain a fall back for teams not relying on PRs (e.g. using trunk-based branching model).
A small note on your QG from your screenshots: you have set a condition on 0 Code Smell. This is tough, and probably quite costly for the teams. I would recommend to use the A Maintainability rating instead (as with the built-in Quality Gate), allowing some low level of code smells in the New Code, while still allowing projects to improve their overall maintainability over time.
Hello @Dick_Dijk
a branch build would not change anything here on SonarQube results; a double branch analysis would have been needed in fact: a first one to establish the baseline and the second one to identify the issue. This work around seems too complex, and far too costly. As I see it, the best solution for your case would be to get back to a more standard Quality Gate.
