[Coverage] Different (much lower) coverage in SonarCloud for Rails project

SonarQube Cloud
,
1

Hello,

I have a problem figuring out and hopefully fixing incorrect (in my opinion) code coverage calculations in SonarQube. Below are some more details and example files from SonarQube and from SimpleCov::Formatter::HTMLFormatter. In the whole project, we have ~96% of the overall code coverage but SonarQube is reporting only 67.5% :frowning:

Details:

  • SCM used: GitHub
  • CI system used: CodeShip Pro
  • Languages of the repository: Ruby
  • Coverage reporter: SimpleCov::Formatter::JSONFormatter
  • Sonar scanner version: 4.7.0.2747

Screenshot 2023-02-10 at 08.59.11
Screenshot 2023-02-10 at 08.59.111594×1218 125 KB

Screenshot 2023-02-10 at 08.57.54
Screenshot 2023-02-10 at 08.57.542648×1222 209 KB

2

Hey there.

We had a similar report in the past but couldn’t reproduce it (and lost the user to follow-up)

I’ve flagged this for some expert attention so that fingers crossed, we can get to the bottom of this.

5

Hello! Any update on this?

7

We seem to be facing this issue with one of our projects. In our case, it might be caused by us splitting specs into several chunks that execute independently, and then merging multiple coverage reports. Apparently, SimpleCov generates zero-coverage report for (some?) files that were not covered within the chunk. E.g., for one of the files, the chunk with its tests results in a report like

    "/w/some/file.rb": {
      "lines": [
        1,
        1,
        1,
        1,
        1,
        1,
        2,
        null,
        1,
        2,
        null,
        1,
        1,
        1,
        2,
        null,
        1,
        2,
        null,
        1,
        2,
        null,
        1,
        2,
        null,
        1,
        2,
        null,
        null,
        null,
        null,
        null,
        null
      ],
      "branches": [

      ]
    },

however, other chunks include

    "/w/some/file.rb": {
      "lines": [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0
      ],
      "branches": [

      ]
    },

which seems to result in 0 coverage for lines having null coverage in the main report.

We’re using SimpleCove 0.22.0 (on top of Ruby 2.6) for that project.

8

This looks like a bug in SonarQube SimpleCov sensor, specifically in the logic that aggregates multiple reports. SimpleCov itself includes aggregation logic that explicitly says that null and 0 should be merged into null: simplecov/lib/simplecov/combine/lines_combiner.rb at v0.22.0 · simplecov-ruby/simplecov · GitHub

      # Return depends on coverage in a specific line
      #
      # @param [Integer || nil] first_val
      # @param [Integer || nil] second_val
      #
      # Logic:
      #
      # => nil + 0 = nil
      # => nil + nil = nil
      # => int + int = int
      #
      # @return [Integer || nil]

SonarQube SimpleCov sensor, however, merges null and 0 into 0 resulting in “no coverage” report for non-executable lines.

1 Like
14

Hello! Were these changes/fixes ever deployed to SonarCloud?

As mentioned under this other thread: SonarQube incorrectly aggregates SimpleCov Ruby coverage reports - #10 by Dorian_Burihabwa

I am also experiencing discrepancies between my Rails project, RSpec/SimpleCov (97-98%) and SonarCloud (89-90%) many months later.

15

Created this new thread per SonarCloud’s request: [Coverage] Lower coverage in SonarCloud than in Rails project