SonarCloud coverage changes every time

SonarCloud
, , ,
1

Hello,

We created a sonar-project.properties file with this configuration in our GO project:

sonar.organization=bkn301
sonar.projectKey=bkn301_stellar-middleware

# relative paths to source directories. More details and properties are described
# in https://sonarcloud.io/documentation/project-administration/narrowing-the-focus/
sonar.sources=.
sonar.exclusions=**/*_test.go,**/*_generated*.go,**/*_autogenerated/**,**/vendor/**
 
sonar.tests=.
sonar.test.inclusions=**/*_test.go
sonar.test.exclusions=**/*_generated*.go,**/*_autogenerated/**,**/vendor/**

# Properties specific to Go
sonar.go.golangci-lint.reportPaths=golangci-lint-report.out
sonar.go.tests.reportPaths=test-report.unit.json,test-report.integration.json
sonar.go.coverage.reportPaths=coverage.unit.out,coverage.integration.out

We have a CI that each time it is activated it runs our unit and integration tests. We use Github actions in combination with your action to push the reports.
While the test reports’ results are always the same (I am talking about lines touched and coverage percentage), the coverage shown in the SonarCloud web dashboard changes every time.

I can see in the dashboard all the voices like coverage, lines to cover, uncovered lines and line coverage continuously changes. As instance, lines that have been covered by tests are marked as uncover.

Is there any reason for that?

The only thing beside the date time that changes in our test reports from a CI workflow run to another is the test execution order, but the data logged and their results are always the same.

Thanks for anyone who can help us!

2 Likes
2

Hi,

Welcome to the community!

Can you share some analysis logs of high/low reported coverage? Also, are you absolutely sure the coverage reports are the same from run to run? And finally, are we talking about branch or PR analysis?

 
Ann

3

Hi Ann, a colleague of Massimo here.

We’ll share asap (we are on CET) the files you’re requesting for.

I can aswer the following right now.

Are you absolutely sure the coverage reports are the same from run to run?

Yes we are, we reproduce the steps that are running on GitHub Actions and we’ve checked that coverage files are the same run over run.

Are we talking about branch or PR analysis?

We encountered the problem on both cases.

Altought our repo current is private for IP reasons we can give to you or your mates on demand access to check directly.

Thanks a lot.

4

Hi,

We should set PRs aside. Coverage for a PR is only going to be reported on changed lines, so it’s quite natural that it would vary.

 
Ann

5

Hello Ann,

Thank you for your answer!
I have attached to this reply our reports from two runs of our pipeline; first and second runs are
prefixed by “1-” and “2-”.

As you can see from the coverage files, the coverage percentage is always the same, what changes is the test order execution and as a result the *coverage.out change too.
1-test-report.unit.json (14.6 KB)
1-test-report.integration.json (6.6 KB)
1-golangci-lint-report.out.log (83 Bytes)
1-coverage.unit.out.log (16.1 KB)
1-coverage.integration.out.log (293.6 KB)

6

And here there are the 2nd run report files.

Thanks again for your time

Regards,

Massimo Tamburini

2-test-report.unit.json (14.6 KB)
2-golangci-lint-report.out.log (83 Bytes)
2-test-report.integration.json (6.7 KB)
2-coverage.unit.out.log (16.1 KB)
2-coverage.integration.out.log (293.6 KB)

7

Hi, any news about this issue?

For us is a strong impact as we can’t be able to oversee our coverage stats.
As we pay for the sonarcloud license would be good to get helped otherwise we’ll evaluate other tools like https://about.codecov.io/ and https://www.codacy.com/.

Thanks,

8

Hey there.

Ann lives in the US and it was a public holiday on Monday. Please make sure to review our Community FAQ, specifically the section where it asks users to “Be patient.”

https://community.sonarsource.com/faq#patient

With regards to the files shared – while the coverage reports are interesting, it will be more meaningful to see the results of the sonar-scanner (probably the sonarsource/sonarcloud-scan-action in your GitHub Actions pipeline), where logs about the coverage import are included.

9

Hello @Colin,

Thanks you for your reply, I have attached here the sonar scanner log.
sonarcloud-scan-action.log (16.8 KB)

10

Hi,

I was hoping for two of these analysis logs: one from a high-coverage analysis and one from a low-coverage analysis. To be clear, I’m not looking for your coverage reports.

 
Ann

13

Hello,

Sorry Ann, I think I do not understand. What do you mean by high and low coverage analysis?

Massimo

14

Hi,

To diagnose the problem, I want to look at the analysis logs for multiple scenarios to see if I can detect any changes in the SonarScanner behavior between when things show up covered and when they don’t.

 
Ann

15

Hi,

So, do you need the screenshots about the SonarCloud web dashboard overviews before and after a new coverage? Or is there a way I can share this data with you extracting via logs?

Massimo

16

Hi,

I’m asking for logs. The stdout text that’s output from the analysis command.

 
Ann

17

Hi Ann,

Sorry, what command should I run to get the required logs?
It is something related to your Github Action?

Massimo

18

Hi Massimo,

Maybe the GitHub Actions docs will help.

 
Ann

19

First scan (scan1) 64.1%, second scan 13.9%.

overralcode1
overralcode11020Ă—642 22 KB

overralcode2
overralcode21039Ă—609 19.4 KB

sonarcloud scan1.txt (33.1 KB)
sonarcloud scan2.txt (24.1 KB)

20

Hi Massimo,

Thanks for the logs! That’s exactly what I was looking for.

After downloading them I stripped out the line-level timestamps and ran a diff on them. Other than minor timing differences, the files are mostly the same. The difference is this big block of coverage warnings in scan1:

INFO: Load coverage report from '/github/workspace/src/coverage.unit.out'
INFO: Load coverage report from '/github/workspace/src/coverage.integration.out'
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/create_account_urlbuilder.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_payments_by_public_key_parameters.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_by_public_key_parameters.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/account_create_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/kill/get_kill_parameters.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_payments_by_public_key.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/configure_btkl.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/server.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/paginated_account_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/list_of_operation_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/health_check/get_health_check_responses.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/paginated_asset_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/paginated_payment_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_accounts.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_operations_by_public_key.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/embedded_spec.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/get_payments_urlbuilder.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/create_payment.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/get_payments_responses.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/kill/get_kill.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/assets/create_asset_urlbuilder.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_operations_by_public_key_responses.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_payments_by_public_key_urlbuilder.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/get_payments.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/health_check/get_health_check.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/account_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/health_check/get_health_check_urlbuilder.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/create_payment_parameters.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_by_public_key_urlbuilder.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_operations_by_public_key_parameters.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/pagination_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/get_operations.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/assets/get_assets_responses.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_by_public_key_responses.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/operation_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/asset_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_accounts_responses.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/create_payment_response_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/assets/create_asset_responses.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/wire_gen.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/assets/create_asset_parameters.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/health_check/get_health_check_parameters.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/create_payment_urlbuilder.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/create_account.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/status_message.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/balance_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/accounts/get_account_operations_by_public_key_urlbuilder.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/assets/create_asset.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/btkl_api.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/response_error.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/paginated_operation_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/get_operations_parameters.go' is not included in the project, ignoring coverage
INFO: Sensor Go Cover sensor for Go coverage [go] (done) | time=65ms

Versus the much smaller block in scan2:

INFO: Load coverage report from '/github/workspace/src/coverage.unit.out'
INFO: Load coverage report from '/github/workspace/src/coverage.integration.out'
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/get_operations.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/models/balance_d_t_o.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/create_payment_parameters.go' is not included in the project, ignoring coverage
WARN: File 'botika.ai/btkl/app/webservice/routes_autogenerated/restapi/operations/operationsops/create_payment.go' is not included in the project, ignoring coverage
INFO: Sensor Go Cover sensor for Go coverage [go] (done) | time=19ms

Surprisingly, we have more warnings about ignored files in the analysis with higher coverage.

That fact made me wonder about the code passed into analysis. Your scan1 screenshot includes this line at the top:

2k Lines of Code

Unfortunately, that line is cropped out of the other screenshot…

Just to double-check, you’re absolutely sure that scan1 corresponds to the first screenshot &etc?

And if that’s the case, we may need to back up to what code is being passed into analysis. Both analyses have same configuration in terms of where to look for source files and what to exclude. But do they have the same number of files/lines of code? I.E. this could be a question of what’s checked out.

Maybe less code is available to the first analysis (as evidenced by all the “not included in the project” warnings) and the coverage % on what’s left is higher…?

Can you investigate that on your end?

 
Ann

21

Hello Ann,

Sorry for the late reply, in this period I did some test and I checked that the coverage did not change randomly.

We finally solved! But I cannot tell you with absolute certainty if we solve by changing the sonar-project.properties removing sonar.test.exclusions or for any other reasons, because, in the meanwhile, we changed a lot of things.

Anyway, I am glad that now everything is working correctly.
Thanks for your support.

Regards,

Massimo

1 Like