Inconsistent Go coverage reported with multiple coverage files

SonarQube Cloud
,
1
  • ALM used - GitHub
  • CI system used - Azure DevOps
  • Scanner command used when applicable - See our Azure DevOps pipeline excerpt below
  • Languages of the repository - Go
  • Error observed - Comments marked as uncovered code inconsistently on PRs
  • Steps to reproduce: We open PRs that add functions with comments and comments, closing brackets, and other non-code lines are marked as uncovered code when they should not be. We use multiple coverage files and think there may be inconsistent merging of the coverage files. We have already tried upgrading the go toolchain, using different coverage modes, and more at the source.
  • Potential workarounds - Substantially reducing quality gate percentages, force merging PRs

We use Azure DevOps to run SonarCloud analysis. Here are the relevant pipeline steps with configuration:

      - task: SonarCloudPrepare@3
        inputs:
          SonarCloud: "SonarCloud"
          organization: "ui"
          scannerMode: "CLI"
          configMode: "file"
          extraProperties: |
            sonar.go.coverage.reportPaths=**/coverage-*.out
            sonar.go.tests.reportPaths=**/report-*.xml
            sonar.exclusions=**/*_test.go,**/report.xml,**/report.json,**/dist/*,**/zz_generated*.go,pkg/clientset/**,**/hack/**,pkg/mapper/**,cmd/mazewalk/mapper/**,**/*.pb.go
            sonar.test.inclusions=**/*_test.go,test/**
            sonar.sources=.
            sonar.tests=.
            sonar.projectKey=UiPath_mazewalk
            sonar.projectName=mazewalk
            sonar.projectBaseDir=$(Build.Repository.LocalPath)

      - task: SonarCloudAnalyze@3

      - task: SonarCloudPublish@3
        inputs:
          pollingTimeoutSec: "300"

On PRs, we see comments marked as uncovered code when they should not be like so:

image
image1391Ă—150 7.98 KB

On the main branch, the comments are correctly not counted as code. This is the output we expect on branches:

image
image1409Ă—146 7.8 KB

We have also seen function signatures, closing brackets, and other comments marked as code when they should not be.

Here are our coverage files. We have a 4th but it’s over the size limit for this forum. I renamed them to txt so I could upload them, but they always have a .out suffix otherwise. I included the branch coverage files for our branch (not producing the expected sonarcloud output) and the main branch coverage files (producing the expected sonarcloud output)

MAIN - coverage-integration.out.txt (2.1 MB)

MAIN - coverage-e2e.out.txt (1.7 MB)

MAIN - coverage-cluster.out.txt (1.3 MB)

BRANCH - coverage-integration.out.txt (2.1 MB)

BRANCH - coverage-e2e.out.txt (1.7 MB)

BRANCH - coverage-cluster.out.txt (1.4 MB)

If you need any more information please let me know. Thank you in advance for the help!

2

Hi,

Welcome to the community and thanks for leading with your coverage files! :sweat_smile:

Can you help me narrow down which report(s), and where in them I should be looking at to see that the “uncovered” code in your screenshots actually is covered?

 
Thx,
Ann

3

Hi, thanks for the fast response!

I filtered the original coverages for just the lines that contained the affected file. Here are the branch and main files that only contain lines related to the privateendpoint_controller.go file which has the function/comment issue from the screenshots above.

MAIN - controller.out.txt (1.1 MB)

BRANCH - controller.out.txt (1.1 MB)

The affected lines you should look for are between lines 75-85. The filtered coverages files include the coverage for the whole privateendpoint_controller.go file just in case. Thanks for the help!

6

Hi,

Thanks for the whittled down reports.

For coverage, we go 100% by what the coverage reports tell us. If the reports say a line is not coverable, then we ignore it for coverage purposes. If they say it’s covered, then it’s covered.

Initially I was a bit confused by your presentation of multiple files each for main and branch and assumed that with each analysis you were submitting reports from multiple coverage engines and they were giving analysis conflicting information.

But if you’re submitting one report per analysis, that can’t be the case. And if you’re seeing comments marked as uncovered in your PRs, then that’s coming straight from your coverage engine.

 
Ann

7

Hi, we do submit multiple coverage files to SonarCloud. We do not submit one report per analysis. Since we’re submitting multiple reports and seeing inconsistent behavior, we thought it might be related to how the reports get merged by SonarCloud. Hope this clarifies. Thank you!

8

Hi,

Can you try this with one report at a time to see if it’s a data problem (and isolate it if so) or a merging probelm?

 
Thx,
Ann

9

Sure, I can give that a try today.

Do you know how I can manually upload a single report to SonarCloud to iterate? We usually do this through our PR process which includes all the reports.

10

Hi,

No :sweat_smile:

But maybe add to your PR commenting out where the other reports are submitted to analysis?

 
Ann

11

Ok I’ll run some scenarios through SonarCloud that way then and update here with my findings when I’m finished. Thanks!

1 Like
12

I ran some scenarios and this may be related to this issue with the Go coverage tool’s interaction with SonarCloud: cmd/cover: inconsistent treatment of comments · Issue #22545 · golang/go · GitHub.

Is the SonarCloud team tracking implementing an upstream fix? If not, what strategies are recommended for mitigating this issue? Thank you!

13

Hi,

Thanks for your investigations!

Unfortunately, I’m not aware of any effort on our part to patch the coverage tool. In other circumstances, I would advise switching coverage engines, but it looks like this native tool is the only game in town. :frowning:

I’ll flag this for the language team just so they’re aware of it. But I honestly don’t see a good path forward except to reduce your coverage threshold (not that that’s a good path :frowning: )

 
Ann

16

Hello,

Thank you for bringing this to our attention.

This issue is indeed related to how Go coverage reports are generated. We’re currently working to propose a fix, and while it involves some complex
changes, we’re making progress.

We hope to open a PR soon and appreciate your patience as we work through this.

Best,
Rudy