Feature branches are not comparing new code correctly, which affects quality gates
When we are building our feature branches, we compare against our main branch for new code, and our main branch compares against the previous version. Since upgrading to 8.5, the new code metrics have been inconsistent. Our feature branches are building and passing the quality gate, but once merged to the main branch, the quality gates fail due to missing coverage, drops in reliability, or security issues. I’ve attached screenshots showing the feature branch passing, the main branch after merge failing, and our new code settings.
Your branch snapshot (from 4:54 p.m.) shows 100% coverage on 1 line. Your develop/main snapshot (from 8:48 p.m. - presumably post-merge) shows 43% coverage on 14 lines. So I think this is not about the merging of that one branch but possibly of several - or at least of one more.
Several years ago we introduced a control to not apply QG conditions on Coverage (overall Coverage?) when there are fewer than 20 new lines. AFAIK, this is still in force. It would explain why a PR could pass with low coverage on just a few lines, but then cumulatively multiple merged PRs could push the Project over the QG threshold.
Your next question is why you would be seeing a failed QG on 14 “New Lines to cover”. Here we come to the differences among (Docs reference):
Lines - carriage returns
Lines of code - “Number of physical lines that contain at least one character which is neither a whitespace nor a tabulation nor part of a comment.”
Lines to cover - [We don’t actually have a definition of this, but I believe it’s a subset of Lines of Code which omits, e.g. includes, comments &etc.]
If you check your Measures page, you’ll probably see that while you only have 14 New Lines to cover, you have more than 20 New Lines.
And yes, some of this could certainly be improved. Feel free to watch and vote for the related tickets:
SONAR-10231 - refine metrics used by Quality Gate fudge factor
SONAR-10485 - Provide a way to configure the Quality Gate fudge factor
Your commit diff shows 2 changes: 1 comment line and 1 export statement in… a test file. Even if those lines were coverable, by definition, test files shouldn’t be included in coverage calculations. That’s reflected in the “0 New Lines to Cover” in your branch screenshot.
Your develop homepage screenshot shows 2 new lines to cover. If you click through on that “2”, what files do you see at the top of the list with non-zero values? Is it user-management-create-user-failure.test.ts or some other file merged into develop from another branch?
And conversely, if you click through on the 0 in your branch - assuming it still exists in SonarQube - what do you see for the file in question? Does it show 0 New lines? And if you click through on the file, does it have the appropriate lines highlighted in yellow or not?
It shows the user-management-action.ts file. On develop, it lists the 2 line changes shown in the screenshot. On the feature branch it shows 0, and those lines are not marked as new.
Yes, it has the lines with date Nov 25 2020. I checked the main branch and it shows the same date and commit. In the feature it does not show as new, and in main it shows as new.
There’s already a ticket open on this. Here
Sonar has a fudge factor of 20 lines whereas it’ll ignore some quality checks if the new lines of code is less than 20. So, it will probably pass your analysis (by ignoring some quality checks).
However, it’ll possibly fail, when your analysis runs against the after-merged code (develop or master)
That doesn’t explain this issue though. The feature branch shows 0 lines of new code to cover, and X when merged into the main branch, when only the changes from the feature branch are merged. Unless you’re saying that it only counts new lines to cover if they are over 20? If so, that doesn’t match the current functionality either, as again, the feature branch shows 0 lines and the main branch shows the correct number.
This also just occurred for us (new code seen as existing code) for a bug introduced in the new code, which dropped the reliability and caused the build to fail once merged to main branch.
If I’m not in wrong this is related to threshold of 20 new lines of code.
So if feature branch has les than 20 new lines that will not trigger quality check.
Is this than a solution for this situation? https://jira.sonarsource.com/browse/SONAR-10485
