[Coverage] Lower coverage in SonarCloud than in Rails project

SonarQube Cloud
,
1

Hello Sonar community.

I have 2 Rails repositories for which I am noticing a different test coverage percentage reported by Simple Cov, compared to what I see in SonarCloud.

Typically, the 2 Rails projects have a relatively high code coverage:

  • Reported by RSpec/SimpleCov: 97-98%
  • Showing in SonarCloud: 89-90%

Linking 2 previous/related threads for reference on this particular topic:

From my earlier question in one of these threads, it appears that a previous bug was already fixed and deployed back in September 2024.

Relevant stack information/dependencies:

  • ruby 3.4.3
  • rails 8.0.2
  • simplecov 0.22.0
  • rspec 3.13.0
  • rspec-sonarqube-formatter 1.6.4

SonarCloud configuration (sonar-project.properties)

sonar.projectKey=*********
sonar.organization=*********
sonar.sources=app
sonar.tests=spec
sonar.test.exclusions=spec/fixtures/**
sonar.exclusions=spec/fixtures/**
sonar.inclusions=app/**,config/**
sonar.testExecutionReportPaths=coverage/sonarqube-test-report.xml
sonar.ruby.coverage.reportPaths=coverage/coverage.json

Despite last year’s bug fixes, could this issue still be an open one?

Is this possible to have SonarCloud instead read from the coverage/last_run.json file instead of attempting to compute its own percentage out of the test coverage report? This would ensure a perfect match between the 2.

{
  "result": {
    "line": 97.97
  }
}

If not, I am wondering what else could be the cause for this problem. As you possibly noted in my sonar-project.properties I have to exclude fixtures from the test paths:

sonar.test.exclusions=spec/fixtures/**
sonar.exclusions=spec/fixtures/**

Due to another SonarCloud limitation on project key lengths:

Could this be (one of) the reason(s) why the SonarCloud-assessed code coverage is lower in SonarCloud than it is in Rails/RSpec/SimpleCov?

If you aren’t familiar with Rails projects, the fixtures directory includes things like VCR cassettes and approval files, respectively allowing the replay of HTTP requests in CI, and validation that API responses are matching some previously recorded and approved content. These file names and the directory they are nested under are relatively long, as they are the concatenation of snake-cased contexts, test descriptions and expectations, and more or less, out of a developer’s control, or inevitable when you have a thorough test suite, in which every test case is well described and documented.

If that is the root cause, is there a way for you to increase this artificial character limit?

In this (linked above) project key length-related thread, Duarte Meneses mentioned a few years ago: “The limit of 400 chars is not “artificial” - it’s actually limited by the size of the database columns storing those keys. I suggest you find a way to encode a shorter key, like hashing part of it.”

Could this database column size be increased to a much higher value than 400, on your backend? At the end of the day, database column sizes are configurable and this is arguably an artificial limitation.

Thanks in advance for all of your help.

JD.

2

Hi JD,

No.

If you’re excluding tests, then I believe those tests aren’t taken into account for coverage (a debug analysis log would tell the tale there).

Sorry, but no.

 
Ann

3

Thank you Ann,

As a paying SonarCloud customer, what is our best path forward? At the end of the day, these numbers need to match.

How can I securely provide you with a debug analysis log?

And if these longer file paths/file names happen to be the issue, how do we get them included into the scan/analysis?

Other folks are also (or have also been) running into this, with no resolution in sight:

4

Hi,

Feel free to redact your log as necessary.

Regarding your path length, you’ve already cited another thread where a solution was offered.

You may also find this dusty-but-still-relevant blog post helpful.

 
Ann

1 Like