New Code results differ between main and feature branches

Report a bug
,
#1
  • SonarQube Enterprise 8.5.1.38104
  • Feature branches are not comparing new code correctly, which affects quality gates

When we are building our feature branches, we compare against our main branch for new code, and our main branch compares against the previous version. Since upgrading to 8.5, the new code metrics have been inconsistent. Our feature branches are building and passing the quality gate, but once merged to the main branch, the quality gates fail due to missing coverage, drops in reliability, or security issues. I’ve attached screenshots showing the feature branch passing, the main branch after merge failing, and our new code settings.

new-code-settings
new-code-settings1295Ă—960 38.1 KB

main-branch
main-branch1370Ă—855 40.4 KB

feature-branch
feature-branch1465Ă—847 39.5 KB

Thanks

SonarCloud - New code coverage results differ between branches
#2

Hi,

Your branch snapshot (from 4:54 p.m.) shows 100% coverage on 1 line. Your develop/main snapshot (from 8:48 p.m. - presumably post-merge) shows 43% coverage on 14 lines. So I think this is not about the merging of that one branch but possibly of several - or at least of one more.

Several years ago we introduced a control to not apply QG conditions on Coverage (overall Coverage?) when there are fewer than 20 new lines. AFAIK, this is still in force. It would explain why a PR could pass with low coverage on just a few lines, but then cumulatively multiple merged PRs could push the Project over the QG threshold.

Your next question is why you would be seeing a failed QG on 14 “New Lines to cover”. Here we come to the differences among (Docs reference):

  • Lines - carriage returns
  • Lines of code - “Number of physical lines that contain at least one character which is neither a whitespace nor a tabulation nor part of a comment.”
  • Lines to cover - [We don’t actually have a definition of this, but I believe it’s a subset of Lines of Code which omits, e.g. includes, comments &etc.]

If you check your Measures page, you’ll probably see that while you only have 14 New Lines to cover, you have more than 20 New Lines.

And yes, some of this could certainly be improved. Feel free to watch and vote for the related tickets:

  • SONAR-10231 - refine metrics used by Quality Gate fudge factor
  • SONAR-10485 - Provide a way to configure the Quality Gate fudge factor

 
HTH,
Ann

#3

Nope, this is only merge of a single branch.

#4

Hi,

Your branch screenshot shows 1 New Line to cover, and your Develop screenshot shows 14. I don’t know how else to explain that.

 
Ann

#5

The new code metric is wrong on the branch. This was working in 8.4.

#6

Hi,

I don’t understand this. Wrong how? Are you saying it mis-counts the number of lines to cover? Can you provide analysis logs for the branch?

 
Ann

#7

Hi,

It happened again yesterday. Attached are screenshots and logs from the scans.

merge-commit
merge-commit1881Ă—844 66.9 KB
commit-list
commit-list1885Ă—491 53 KB
branch
branch1326Ă—801 32.9 KB
main
main1362Ă—842 38.6 KB
data-users-management-main.txt (10.2 KB) data-users-management-feature.txt (10.2 KB)

Thanks,
Steve

#8

Hi Steve,

Your commit diff shows 2 changes: 1 comment line and 1 export statement in… a test file. Even if those lines were coverable, by definition, test files shouldn’t be included in coverage calculations. That’s reflected in the “0 New Lines to Cover” in your branch screenshot.

Your develop homepage screenshot shows 2 new lines to cover. If you click through on that “2”, what files do you see at the top of the list with non-zero values? Is it user-management-create-user-failure.test.ts or some other file merged into develop from another branch?

 
Ann

#9

Sorry, that was showing the wrong diff. This is the file with 2 line changes:

Correct file diff

#10

Hi,

This question still stands:

And conversely, if you click through on the 0 in your branch - assuming it still exists in SonarQube - what do you see for the file in question? Does it show 0 New lines? And if you click through on the file, does it have the appropriate lines highlighted in yellow or not?

 
Ann

#11

It shows the user-management-action.ts file. On develop, it lists the 2 line changes shown in the screenshot. On the feature branch it shows 0, and those lines are not marked as new.

#12

Hi,

In the branch analysis can you double-check the contents of the file to see that SonarQube has the changed version? And if so, with what blame date?

 
Ann

#13

Hi,

Yes, it has the lines with date Nov 25 2020. I checked the main branch and it shows the same date and commit. In the feature it does not show as new, and in main it shows as new.

Thanks,
Steve

#14

Okay, thanks. I’ve referred this internally. We’re past my competence now.

 
Ann

#15

Thank you! Appreciate all of your help.

#16

Hello,

There’s already a ticket open on this. Here
Sonar has a fudge factor of 20 lines whereas it’ll ignore some quality checks if the new lines of code is less than 20. So, it will probably pass your analysis (by ignoring some quality checks).

However, it’ll possibly fail, when your analysis runs against the after-merged code (develop or master)

I hope this helps.

#17

That doesn’t explain this issue though. The feature branch shows 0 lines of new code to cover, and X when merged into the main branch, when only the changes from the feature branch are merged. Unless you’re saying that it only counts new lines to cover if they are over 20? If so, that doesn’t match the current functionality either, as again, the feature branch shows 0 lines and the main branch shows the correct number.

1 Like
#18

This also just occurred for us (new code seen as existing code) for a bug introduced in the new code, which dropped the reliability and caused the build to fail once merged to main branch.

1 Like
#19

This is happening with 8.6 as well.