Lines to cover on new code is 0 in PR and for branches


(Peter Rockstars) #1

Hi,

I just read this thread pull-request-branches-to-include-code-coverage and was excited. We have a policy of 100% code coverage on new code. The quality gate on master works as expected. It is red when new code is added.

To my understanding, this should now also be measured for PR’s and short-lived branches. This isn’t working for us at the moment. We do see the expected overall coverage but not the expected coverage on new code, which we require to be 100%. Screenshot below. It is the same for long- and short-lived branches, and for PRs

The PR is decorated in VSTS but not in the way I would expect. According to the JIRA issue it should be formatted like this:

No issues; n% New Code Coverage, m% Est. post-merge Coverage

But what we’re seeing is:

Code Quality check passed; 39.0% Est. post-merge coverage

Any idea what’s going wrong here? The developer console is not displaying any errors.

Edit: as you can see, the coverage % isn’t shown either.


(Mac Chapman) #2

We’re seeing the same issue with PRs opened against one of our long lived branches, but not against others. Specifically our release branches work but other long lived branches don’t - perhaps this is due to release branches being newer than our other long lived branches?

Our long lived branch reports are working perfectly on all branches - it’s just PRs that are receiving nothing.


(G Ann Campbell) #3

Hi,

Sorry this thread got lost in the shuffle.

I believe this is about your base branch not being specified during analysis. Even if your base is master and the base branch is supposed to default to master, for this to work correctly I think you still need to specify base.

 
Ann


(Mac Chapman) #4

Hmm, my base branch is definitely set correctly, as is pull request key. Running the scanner by hand, it sees all my files, uploads the results correctly, and the PR is tied to the long lived branch on the Sonarcloud UI perfectly - there’s just no content.

The report generated by the scanner appears to contain data from my files just fine. This works for us on on release branches (release/xxxx) but not on other long running branches. Very strange!

The scanner contexts attached to the analysis jobs are identical (bar the change in base branch) in both cases. The analysis succeeds in both cases.


(Mac Chapman) #5

Oh, I’ve just realised, our issue is slightly different to the one in the first post: we just get “No coverage measures” on the measures page. Again, this populates correctly on release branches.


(G Ann Campbell) #6

Hi @veryhappythings,

If you’re having a different problem, then please start a new thread and include analysis parameters, logs, and maybe screenshots to show us what’s (not) happening.

 
Thx,
Ann


(Peter Rockstars) #7

Hi Ann,

No worries, these things happen, glad you could get back to me. I’ve been on holiday, sorry for the late reply.

I tried setting

sonar.pullrequest.base=master

but the result is the same. I checked in the scanner context on the Background Tasks page and the setting is indeed present.

Do you have any other suggestions?


(G Ann Campbell) #8

Hi @PeterRockstars,

So… I’m not sure where my head was when I gave my response of 7 Nov. I’m going to start again.

Going back to the initial problem report, it was:

  • QG isn’t red for <100% coverage in PRs and SLBs
  • PR decoration isn’t formatted as expected

In fact, both of those things are expected. We’re calculating coverage now, but we’re still applying a hard-coded issues-only QG. Is that sub-optimal? Yes. Do we want to change it? Absolutely. When will we deliver it? Uhm… Hopefully in 2019Q1. 7.5 (E.T.A. early Christmas present) will add duplications in PRs & SLBs. Hopefully we’ll be able to work on QGs in 7.6 (but no promises!).

Regarding the wording of the PR decoration, it got tweaked a little during the sprint, and we just didn’t update the MMF to reflect that.

 
HTH,
Ann


(Peter Rockstars) #9

Hi Ann,

The main reason that I posted here was this: we want to be able to see which of the code added in a PR hasn’t been tested yet. The annoying thing is that this is working for us in another repo which has identical configuration. In that project, it looks like this:

As you can see, SC has correctly detected that 10 lines of new code were added and none of them are covered by tests. We can open the file and see that the new code is highlighted. In our main repo, where the initial screenshot is from, we also added new code which we did not test, but none of it is detected as being new code.

One difference I can think of between these repos is that the main repo consists of multiple projects and test projects, and this second, newer, repo only has one project and one test project so far. For both of the projects we use Coverlet to measure the coverage and ReportGenerator to combine the reports into one. Could the multiple projects be the reason for this difference?


(Ashley Ward) #10

This sounds a lot like the issue I’m having with short-lived branches at the moment: Short-lived branch analysis seems to be empty


(Duarte Meneses) #11

Hi Peter,

Could you please gives us the logs from the scanner with debug logs enabled, when scanning the problematic project?

Thanks.


(Peter Rockstars) #12

Sure! Can I send them to you privately?


(Peter Rockstars) #13

I just figured out why I can’t send you a private message, it’s because I don’t have the proper level yet, as seen here: https://blog.discourse.org/2018/06/understanding-discourse-trust-levels/. But I don’t undstand what Entering at least 5 topics means. Does it mean actualy participating in 5 topics?


(Duarte Meneses) #14

Hi Peter,

I had a quick look at the logs you sent me privately.
It looks like the project being analyzed is not at the git repository’s root, right?

We have recently found this bug which might be causing the problem: https://jira.sonarsource.com/browse/SONARSCGIT-35

If you confirm that the project is not at the root of the git repository, would it be possible to try to analyze it at the root of a git repository? That would help us confirm the problem.

Thanks.


(Peter Rockstars) #15

Hi Duarte,

You’re right, the project is in a subfolder, 1 level deep. I’ll try to analyse it from a new repo where it’s at the root.


(Peter Rockstars) #16

Just to clarify, should I analyze it using the same project key, or a new one?


(Duarte Meneses) #17

So you will need to re-analyze the long lived branch, then create a PR targeting it with some changes. These 2 branches need to be done from the same git repo (and with a common fork point).
You can do it with the same project key or with a different key (creating a new project in SC).


Code coverage not displaying on PR analysis
(Peter Rockstars) #18

Here’s what I did:

Created a new repository
Copied all the files from our nested folder to the root of the new repository
Check in to master
(I didn’t re-analyze master, so as not to pollute the master branch in SC)
Created a branch and made some changes
Checked in to the SLB
Analysed the SLB

And what do you know, it’s correctly showing the lines to cover!

Here’s the branch as analyzed in the new repo:

And here’s a branch with identical changes in the old repo:

Looks good!


(Duarte Meneses) #19

Thanks a lot for testing this out!
This confirms the problem described in https://jira.sonarsource.com/browse/SONARSCGIT-35.
We will work on a fix for the bug very soon.