Large discrepancy between "Estimated after merge" and overall coverage

SonarCloud
, ,
1

I’ve recently integrated SonarCloud into a Swift project, and I’m finding that there is a large discrepancy between a PR’s “Estimated coverage after merge” value and the overall coverage of the main branch.

My main branch is listing a 66.6% overall coverage, with 1,580 uncovered lines. A recent pull request I did strictly for illustrative purposes adds 28 new lines, and has a coverage of 32.1% - however, it shows “83.5% Estimated after merge”, and there’s obviously no way that merging this PR will change the overall coverage number in any way but down, it’s not going to jump up by nearly 17%!

There are several other topics effectively with this same question (like How does SonarQube calculate estimated code coverage after merge? and How does Sonarqube calculates "Estimated after merge" for every PR?) but none of those get to any kind of actionable conclusion - they’re typically just left hanging with a question back to the OP and no answer of any kind.

I’ve verified that my main merge analysis and PR branch analysis use the same settings for base dir, exclusions, other settings - they are identical except for the branches being specified. The only odd thing about my setup that I can think of is that I’m running the analysis from Xcode Cloud, which is less than ideal (it’s not really setup to do this). However, even though it is called a bit weird, they are both being done in the exact same way, with the only exception of calling a different Fastlane function to specify the branch.

Any suggestions for what to look at to figure out where this discrepancy is coming from?

2 Likes
2

Hey there.

The Estimated after merge figure should just be the overall coverage based on the code being analyzed, and the coverage reports provided.

My first suggestion would be to force the analysis that would normally run for the pull request to run as a long-lived branch (adding -Dsonar.branch.name=release-coverage-test) would do the test, and see how the coverage is reported there.

Can you try that and report back on the results?

If it reports the 83.5% figure, it will be much easier to dive into the Code tab and understand where that’s coming from. If it’s reporting a lower figure, we’ll have to dive deeper.

1 Like
3

After adding -Dsonar.branch.name=release-coverage-test and running the PR, I get an error of A pull request analysis cannot have the branch analysis parameter 'sonar.branch.name'.

4

Are you explicitly passing the sonar.pullrequest.* parameters? If so, try removing those for now.

If not, then setting sonar.branch.* explicitly should be overriding any autoconfiguration…

5

I’m using Fastlane’s sonar function, so those are not being added directly as a parameter, but is instead as parameters to the sonar function which I have to assume ends up mapping to doing something similar to plain runner arguments.

The current call is as below (with **** showing redacted content):

        sonar(
            project_key: "****",
            project_name: "****",
            project_version: "3.0",
            project_language: "swift",
            exclusions: "vendor/**,**/Tests/**,**/Package.swift",
            sonar_runner_args: "-Dsonar.projectBaseDir=#{options[:workspace]} -Dsonar.c.file.suffixes=- -Dsonar.cpp.file.suffixes=- -Dsonar.objc.file.suffixes=- -Dsonar.pullrequest.provider=github -Dsonar.junit.report_paths=#{options[:workspace]}/test_output -Dsonar.coverageReportPaths=#{options[:workspace]}/sonarqube-generic-coverage.xml -Dsonar.branch.name=release-coverage-test",
            sources_path: options[:workspace],
            sonar_organization: "****",
            sonar_login: options[:sonar_token],
            sonar_url: "https://sonarcloud.io",
            pull_request_branch: options[:source_branch],
            pull_request_base: options[:target_branch],
            pull_request_key: options[:pr_number] 
        )

From your comment, sounds like maybe I should change it to this instead:

    sonar(
        project_key: "****",
        project_name: "****",
        project_version: "3.0",
        project_language: "swift",
        exclusions: "vendor/**,**/Tests/**,**/Package.swift",
        sonar_runner_args: "-Dsonar.projectBaseDir=#{options[:workspace]} -Dsonar.c.file.suffixes=- -Dsonar.cpp.file.suffixes=- -Dsonar.objc.file.suffixes=- -Dsonar.pullrequest.provider=github -Dsonar.junit.report_paths=#{options[:workspace]}/test_output -Dsonar.coverageReportPaths=#{options[:workspace]}/sonarqube-generic-coverage.xml",
        sources_path: options[:workspace],
        sonar_organization: "****",
        sonar_login: options[:sonar_token],
        sonar_url: "https://sonarcloud.io",
        branch_name: "release-coverage-test"
    )

Does that sound about right?

6

Assuming that was right I went ahead and ran with that, and got the results - using the branch_name: "release-coverage-test" version, it shows an overall coverage of 66.4%, and since my overall main branch analysis shows 66.6%, that actually tracks pretty well, since this PR has several new lines that aren’t covered.

7

Thanks. It sounds like you took the right steps. And it sounds like the long-lived branch has coverage appearing correctly while the pull request doesn’t.

I’ve tried reproducing this and so far I can’t – the Estimated after merge always appears as expected on SonarCloud.

  • Do you have access to the sonarqube-generic-coverage.xml file from both runs (the branch analysis and the pull request analysis), and do you see any differences?
  • It would also be useful to check the analysis logs to compare the number of files being indexed between the PR run and the normal branch. It appears in the logs like below:

INFO: 31 files indexed

8

Unfortunately I don’t have access to the sonarqube-generic-coverage.xml file, as only certain artifacts are available for download from the build environment - however, I can see about setting up my local environment to run this directly, which honestly is probably something I should have done already. Will reply back once that is setup and I have results and files from local runs.

As far as the logs go, both the “release-coverage-test” version and the prior normal PR one show the same thing:

INFO: 298 files indexed
INFO: 130 files ignored because of inclusion/exclusion patterns
INFO: 3 files ignored because of scm ignore settings
9

Took me a bit longer to get to it than expected, but was able to run it locally and the results were the same. When run as a PR, I got the same “83.5% Estimated after merge” figure, and when run as release-coverage-test, it got 66.4%.

Doing a compare of the sonarqube-generic-coverage.xml files in Beyond Compare - it indicates that they are completely identical. I guess that makes sense, because these two test runs were done against the same build. Hard to see how they come up with different results in SonarCloud from an identical coverage file though. :man_shrugging:

I took a look at the xcresult file, and it shows some pretty high coverage - I know it’s not quite a one-to-one comparison, but it does lend some credence to the higher number shown in the PR estimate rather than the smaller one in the main analysis.

xcresult_coverage
xcresult_coverage738×1006 87.9 KB

Going to keep poking around more, there must be something else I’m missing…

10

That’s interesting – in the Code tab of your main branch, can you find any specific discrepancies compared to the raw report? Or files beign represented (perhaps non-Swift files) that aren’t represented in the coverage report at all?

11

That’s interesting – in the Code tab of your main branch, can you find any specific discrepancies compared to the raw report? Or files beign represented (perhaps non-Swift files) that aren’t represented in the coverage report at all?

I did notice a couple of things:

  • The code in SonarCloud in the release-coverage-test analysis shows the sonarqube-generic-coverage.xml file itself, as well as a test_output directory and associated files (a json and xml file). These are listed when I click the “uncovered lines” option showing the tree view, but list - as their line count.
    • I took the output of the “list” option to list all the files with uncovered lines. Anything with a number is a Swift file, and I dropped those into a spreadsheet to double check - they do add up to the expected 1,599 uncovered lines, so it doesn’t seem like they are being counted in any way.
    • When doing the list of files, they don’t even show up - only show up when doing the tree view.
    • Same goes for the “lines to cover” view - they show up in the tree view with - as the line count, and not in the list view
    • There’s also a There are 78 hidden components with a score of 0. info panel at the end of the uncovered line list view, and showing those just show a bunch of Swift files with 0 uncovered lines. Not even sure why they’d be showing up in this report at all. Spot checking a couple show they have covered lines.
    • These extra files would also be there for the PR analysis (though you can’t see things to that level in that analysis view), so I don’t think that would account for it either
    • In any case, I’ll be taking a look at getting them removed completely from analysis to make sure they aren’t muddying things up in some way
  • The analysis in Sonar is definitely looking at files that the analysis in Xcode is not
    • In a series of tests that show 98.3% coverage in Xcode, and 62.3% in Sonar, it looks like Sonar is correct here.
    • The coverage in Xcode seems to mean “of the files that are covered by the tests, the coverage is 98.3%”, while Sonar is also including files that aren’t covered by any tests in the calculation.
    • If I look at just the files that are in the tests, Sonar shows 97.3% - matching pretty close to Xcode, with a minor difference only coming from the calculation methodology differences
    • Spot checking a couple other places show the same.
12
  • In any case, I’ll be taking a look at getting them removed completely from analysis to make sure they aren’t muddying things up in some way

Removed them by adding the directory and file to the sonar.exclusions list, and they no longer show up in the code view of the report. The overall coverage number has not changed - still 66.4% in the release-coverage-test analysis, and 83.5% as the estimated one when I run the same in the PR analysis.

13

Getting a bit closer now. When digging through the lists of files and uncovered lines, I saw some Swift scripts that are part of the project, but not something that would be executed in tests (they help setup the build environment). As a result, these were correctly listed as 0% coverage. I excluded those since they really shouldn’t be tested anyway. Now my release-coverage-test analysis is at 75.8% coverage. However, unexpectedly the PR analysis estimate didn’t change - it’s still 83.5%. Based on how one number changed but other didn’t, it would seem like the PR analysis was somehow already excluding these files from its calculation but the main one was not.

With that in mind, I evaluated the rest of the lines with a bit more scrutiny. I did find a few more lines that are kind of similar to that - they exist in a different scheme than the one being tested. Removing those brought the main analysis to 78%, while again not changing the PR estimate.

With these two pieces of data in mind, I went back to the sonarqube-generic-coverage.xml file. The files excluded in the above steps are not listed in the file. This seems to be the key. It looks like to me that the estimate in the PR is only looking at the files that are listed in the sonarqube-generic-coverage.xml file, while the main analysis will also look at the files that just exist in the code/folder structure when doing it’s analysis.

15

Thank you for the in-depth investigation. I’ve pinged one of our dev teams to take a look.

1 Like
18

Hi @GeekOnIce

Thank you for raising this issue and for such detailed information! I’ll investigate now and come back to you once I know more.

Anita

19

Hi @GeekOnIce

The ‘Estimated coverage after merge’ displays the coverage calculated for the whole project with the PR changes, so indeed the value should be the same as after merge (assuming there were no other changes merged in the meantime)

SonarCloud doesn’t calculate the coverage on its own - the value comes from the results provided by the third-party tools, which would suggest that there must be a difference in the imported coverage data.

I see that you already did the comparison of the sonarqube-generic-coverage.xm files, however, could you make sure that you compare the following files:

  • file that comes from the build of the target branch
  • file that comes from the build of the PR branch

They shouldn’t be identical, due to the changes introduced in PR, and, I’d expect also additional differences that may impact the ‘Estimated coverage after merge’ metric.

Anita

20

I disagree that they shouldn’t be identical: a coverage file that comes from a build of a PR branch, and one which comes from a build of the target branch that happens after merging that PR branch in should be identical - it’s tests that are run from the exact same code. Unfortunately, the sonarqube-generic-coverage.xml is inaccessible from our build system (it’s not an artifact that we can pull down from Xcode Cloud), so I’m unable to verify that in the build system, but it makes logical sense.

When testing this out earlier with @Colin, what I did was to locally run tests to get an Xcode result bundle, then from that bundle use the xccov-to-sonarqube-generic.sh script to convert it to generate a sonarqube-generic-coverage.xml file. Without any changes to the code or this sonarqube-generic-coverage.xml file, I ran with Fastlane’s sonar action when setup as a PR and then again when setup as a non-PR main branch. The only changes to configuration between these two is the branch information - the PR branch uses -Dsonar.pullrequest.branch, -Dsonar.pullrequest.base and -Dsonar.pullrequest.key, while the main branch one uses -Dsonar.branch.name - all other settings are identical.

When setup to run as a PR branch, it gave one estimate, when setup to run as a main branch it gave a different and slightly lower number. Although this was from local runs, these numbers tracked exactly with what I was seeing from the CI builds. With identical input, I’d expect identical results, but the two ways differed in the ways described above.

21

Thanks for your reply!

I agree that the files should be identical in the scenario you describe, that’s why I wanted to know whether they are identical or not - as the observed behaviour suggests they aren’t.

Since the files are indeed identical, would it be possible to provide us with a small reproducer?

From what I see the main coverage isn’t calculated differently whether running on PR or not, hence in order to see if there is some bug somewhere, a reproducer would help a lot.

Anita

22

OK, have a test project for you that demonstrates the same issue. GitHub - GeekOnIce/SonarTest: Test project for sonar contains the code, and with the data being uploaded to this SonarCloud project. The README file in the Github project has a lot more information, with test setup, analysis and execution information, but as a quick summary:

  • The PR in the project shows a “56.7%” estimate after merge
  • A main merge analysis done on that branch shows “39.5%” coverage
  • The cause of the discrepancy seems to be the same as what it is on the production code described in the earlier part of the message thread

Let me know if you want to run this code as-is, and I can DM you the token or add you to either project, as both were setup specifically just to help debug the issue.

1 Like
23

Hi @GeekOnIce

Thanks a lot for the reproducer, it’s great!
I’ll investigate and come back to you.

Anita

1 Like