Coverage on new code shows as "---"


(Doug) #1
  • which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)

SonarQube 7.3.0.15553
sonar-scanner-cli 3.2.0.1227
using git scm

  • what are you trying to achieve

Quality gate for “Coverage on New Code”. Sonar reports this as a “—” in the UI.

  • what have you tried so far to achieve this

My understanding is:

  1. Coverage on new code quality gate will only be checked if new lines of code > 20
  2. Coverage on new code is determined using the SCM provider blame function

What I am doing:

  1. Create a long lived branch feature/long-lived, initialize a BASELINE version of the sonar scan from the main branch (master). I do this by running sonar scanner on master but passing in sonar.branch.name=feature/long-lived and sonar.projectVersion=BASELINE – This step works

  2. Run sonar scan on feature/long-lived using the same branch name and a new projectVersion for the latest commits. – This step also works.

(Step 1 and 2 are ran in the same jenkins multipipeline. Am I possibly running into some kind of cache issue? I’ve tried using sonar.scm.forceReloadAll=true and this didn’t help)

At this point my activity looks like the following:

  1. At this point sonar is not displaying any metrics for coverage or duplications

  1. Here is the coverage graph for the time period:

As you can see lines have changed more than 20 and also coverage was correctly reported into sonar. Is there something else I am missing around this?


(G Ann Campbell) #2

Hi,

Sorry for the delay in answering, but I’ve been stumped by this, and I suspect no one else picked this up because they were all stumped too.

The question of the number of lines changed is a red herring. That only affects whether Coverage can fail the quality gate, not whether coverage is calculated. I suspect the problem is the tiny amount of time between when the New Code period started and the second analysis. Has this sorted itself out with more time?

Tangentially, your composite screenshot shows the pre-rolled Coverage graph and includes only the leak period. It’s a reasonable assumption that this is showing “on New Code” values. However, Coverage on New Code is actually a different metric.

 
Ann


(Doug) #3

Its never sorted itself out with more time. I think maybe it’s how Sonar is calculating diffs between files? Here are two flows that product different results:

Flow described in original post:

git checkout master
git checkout -b feature/some-branch
make some new commits to feature/some-branch

git checkout master
/bin/sonar-scanner -Dsonar.branch.name=feature/some-branch -Dsonar.projectVersion=BASELINE

git checkout feature/some-branch
/bin/sonar-scanner -Dsonar.branch.name=feature/some-branch -Dsonar.projectVersion=1.3.3-27-g8fdb85b

new lines of code shows as 0

Different flow that yields coverage

git checkout master
git checkout -b feature/some-branch
/bin/sonar-scanner -Dsonar.branch.name=feature/some-branch -Dsonar.projectVersion=BASELINE

make some new commits to feature/some-branch

/bin/sonar-scanner -Dsonar.branch.name=feature/some-branch -Dsonar.projectVersion=1.3.3-27-g8fdb85b

new lines of code shows correctly

Let me know if the way I’m describing the flows above are confusing. I can try to give the examples in a different way.


(G Ann Campbell) #4

Hi,

I am rather confused at this point. I thought the problem was coverage.

Further, your original screenshots showed the homepage of a long-lived branch, yet the branch names in your second post are (by default) short-lived branches.

Further, in your “flow described in original post” you appear to be seeding your branch with an analysis of master. What are you trying to accomplish with this? Short-lived branches (SLB) are automatically baselined against master so that issues already found in master aren’t reported in the SLB. Long-lived branches are synced automatically on first analysis.

 
Ann


(Doug) #5

Hi Ann,

Didn’t catch this as my email marked it as spam.

  1. The problem is coverage, but the coverage detection appears to be determined based on the scm blame for new lines of code. Whenever I get coverage reported correctly, I have new lines of code reported. When coverage reports as “–” new lines of code is reported as 0, even though the files appear in the list.

  2. All of this is against long-lived branches. My regex for “long-lived” branches includes these branch names.

  3. I am seeding the analysis based on master because as a developer your typical workflow is to push a new branch at the same time as you push new code. I am trying to seed the long-lived branch with a baseline that does not include all of their initial commits in the branch.

This is all currently a workaround because SLB’s don’t include coverage.


(G Ann Campbell) #6

Hi,

So first, you have a workflow that gets you the desired result. On reflection, I suspect that the problems in your first flow are caused by the timestamps.

Your branch changes are made before your baseline analysis, so… are they really “new” in this branch? (Answer this like you’re a dumb computer, not a clever human.)

Anyway, you have a workflow that gets you the desired results, so… I’m going to consider the rest moot at this point.

Good news! 7.4 (GA/announcement currently planned for Monday) adds coverage on PRs and short-lived branches.

 
Ann