New Code results differ between main and feature branches

  • SonarQube Enterprise 8.5.1.38104
  • Feature branches are not comparing new code correctly, which affects quality gates

When we are building our feature branches, we compare against our main branch for new code, and our main branch compares against the previous version. Since upgrading to 8.5, the new code metrics have been inconsistent. Our feature branches are building and passing the quality gate, but once merged to the main branch, the quality gates fail due to missing coverage, drops in reliability, or security issues. I’ve attached screenshots showing the feature branch passing, the main branch after merge failing, and our new code settings.



Thanks

Hi,

Your branch snapshot (from 4:54 p.m.) shows 100% coverage on 1 line. Your develop/main snapshot (from 8:48 p.m. - presumably post-merge) shows 43% coverage on 14 lines. So I think this is not about the merging of that one branch but possibly of several - or at least of one more.

Several years ago we introduced a control to not apply QG conditions on Coverage (overall Coverage?) when there are fewer than 20 new lines. AFAIK, this is still in force. It would explain why a PR could pass with low coverage on just a few lines, but then cumulatively multiple merged PRs could push the Project over the QG threshold.

Your next question is why you would be seeing a failed QG on 14 “New Lines to cover”. Here we come to the differences among (Docs reference):

  • Lines - carriage returns
  • Lines of code - “Number of physical lines that contain at least one character which is neither a whitespace nor a tabulation nor part of a comment.”
  • Lines to cover - [We don’t actually have a definition of this, but I believe it’s a subset of Lines of Code which omits, e.g. includes, comments &etc.]

If you check your Measures page, you’ll probably see that while you only have 14 New Lines to cover, you have more than 20 New Lines.

And yes, some of this could certainly be improved. Feel free to watch and vote for the related tickets:

  • SONAR-10231 - refine metrics used by Quality Gate fudge factor
  • SONAR-10485 - Provide a way to configure the Quality Gate fudge factor

 
HTH,
Ann

Nope, this is only merge of a single branch.

Hi,

Your branch screenshot shows 1 New Line to cover, and your Develop screenshot shows 14. I don’t know how else to explain that.

 
Ann

The new code metric is wrong on the branch. This was working in 8.4.

Hi,

I don’t understand this. Wrong how? Are you saying it mis-counts the number of lines to cover? Can you provide analysis logs for the branch?

 
Ann

Hi,

It happened again yesterday. Attached are screenshots and logs from the scans.

data-users-management-main.txt (10.2 KB) data-users-management-feature.txt (10.2 KB)

Thanks,
Steve

Hi Steve,

Your commit diff shows 2 changes: 1 comment line and 1 export statement in… a test file. Even if those lines were coverable, by definition, test files shouldn’t be included in coverage calculations. That’s reflected in the “0 New Lines to Cover” in your branch screenshot.

Your develop homepage screenshot shows 2 new lines to cover. If you click through on that “2”, what files do you see at the top of the list with non-zero values? Is it user-management-create-user-failure.test.ts or some other file merged into develop from another branch?

 
Ann

Sorry, that was showing the wrong diff. This is the file with 2 line changes:

Correct file diff

Hi,

This question still stands:

And conversely, if you click through on the 0 in your branch - assuming it still exists in SonarQube - what do you see for the file in question? Does it show 0 New lines? And if you click through on the file, does it have the appropriate lines highlighted in yellow or not?

 
Ann

It shows the user-management-action.ts file. On develop, it lists the 2 line changes shown in the screenshot. On the feature branch it shows 0, and those lines are not marked as new.

Hi,

In the branch analysis can you double-check the contents of the file to see that SonarQube has the changed version? And if so, with what blame date?

 
Ann

Hi,

Yes, it has the lines with date Nov 25 2020. I checked the main branch and it shows the same date and commit. In the feature it does not show as new, and in main it shows as new.

Thanks,
Steve

Okay, thanks. I’ve referred this internally. We’re past my competence now.

 
Ann

Thank you! Appreciate all of your help.

Hello,

There’s already a ticket open on this. Here
Sonar has a fudge factor of 20 lines whereas it’ll ignore some quality checks if the new lines of code is less than 20. So, it will probably pass your analysis (by ignoring some quality checks).

However, it’ll possibly fail, when your analysis runs against the after-merged code (develop or master)

I hope this helps.

That doesn’t explain this issue though. The feature branch shows 0 lines of new code to cover, and X when merged into the main branch, when only the changes from the feature branch are merged. Unless you’re saying that it only counts new lines to cover if they are over 20? If so, that doesn’t match the current functionality either, as again, the feature branch shows 0 lines and the main branch shows the correct number.

1 Like

This also just occurred for us (new code seen as existing code) for a bug introduced in the new code, which dropped the reliability and caused the build to fail once merged to main branch.

1 Like

This is happening with 8.6 as well.