SonarCloud - Coverage inconsistency with multiple reports

SonarQube Cloud
,
1

Template for a good new topic, formatted with Markdown:

  • ALM used (Bitbucket Cloud)
  • CI system used (Circle CI)
  • Scanner command used when applicable (istanbul/v8)
  • Languages of the repository (JS/TS, vitest, React)

Hello all!

My team and I have a component library that has both unit tests and cypress tests. We want to have the code coverage be a merge of unit and cypress however SonarCloud’s merge is inconsistent. We use istanbul for both coverage report generation. We also tried generating the unit tests with v8 but the sonarCloud’s merge is also inconsistent. Here’s what we have and what we would expect.

We also thought that due to the inconsistent reporting, to do a manual merge but we could never get it consistent enough on the branching report.

With istanbul for both cypress and unit:

  • We found that part of the issue when using istanbul is that the unit coverage report is reporting the implicit branches. For example when there is an if statement without the else condition, the unit coverage report will show 2 branches but the cypress coverage will show 1.
  • Because of this when the 2 reports merge, the second branch which is implicitly covered as it hits code afterwards, is shown as an uncovered branch.
    • There are 2 parallel reports showing 23 branches and correctly no coverage as there are no unit tests for this component
    • There is the report for the cypress report with only 18 branches that are all covered
  • Here is what the UI reported:
    istanbul sonar ui not 100%
    istanbul sonar ui not 100%1342×735 57.5 KB

We then tried to use v8 for our unit report coverage, and even then the merge was incosistent.

  • The unit report now only had 2 branches
  • The cypress coverage is correct
  • But we still get incorrect merge, this time not using the cypress coverage at all
    v8 sonar ui not 100%
    v8 sonar ui not 100%1262×686 48.6 KB
Snippet lcov coverage report from Istanbul for Cypress 
TN:
SF:src/components/ResetAlert/ResetAlert.tsx
FN:10,(anonymous_0)
FN:29,(anonymous_1)
FN:45,(anonymous_2)
FN:60,(anonymous_3)
FNF:4
FNH:4
FNDA:1379,(anonymous_0)
FNDA:1379,(anonymous_1)
FNDA:648,(anonymous_2)
FNDA:648,(anonymous_3)
DA:10,54
DA:11,1379
DA:18,1379
DA:20,1379
DA:21,1379
DA:23,1379
DA:29,1379
DA:30,1379
DA:32,792
DA:34,704
DA:36,47
DA:38,41
DA:43,1379
DA:45,1379
DA:46,648
DA:47,638
DA:48,51
DA:55,587
DA:60,1379
DA:61,648
DA:62,10
DA:63,638
DA:64,68
DA:66,570
DA:69,1379
DA:74,731
LF:26
LH:26
BRDA:13,0,0,320
BRDA:24,1,0,792
BRDA:24,1,1,587
BRDA:30,2,0,587
BRDA:32,3,0,704
BRDA:32,3,1,47
BRDA:32,3,2,41
BRDA:46,4,0,10
BRDA:47,5,0,51
BRDA:61,6,0,10
BRDA:61,6,1,638
BRDA:63,7,0,68
BRDA:69,8,0,731
BRDA:70,9,0,1379
BRDA:70,9,1,1311
BRDA:70,9,2,741
BRDA:84,10,0,638
BRDA:84,10,1,10
BRF:18
BRH:18
end_of_record
Snippet lcov coverage report from Istanbul for Unit 
TN:
SF:src/components/ResetAlert/ResetAlert.tsx
FN:10,(anonymous_0)
FN:29,(anonymous_1)
FN:45,(anonymous_2)
FN:60,(anonymous_3)
FNF:4
FNH:0
FNDA:0,(anonymous_0)
FNDA:0,(anonymous_1)
FNDA:0,(anonymous_2)
FNDA:0,(anonymous_3)
DA:10,0
DA:11,0
DA:18,0
DA:20,0
DA:21,0
DA:23,0
DA:29,0
DA:30,0
DA:32,0
DA:34,0
DA:36,0
DA:38,0
DA:43,0
DA:45,0
DA:46,0
DA:47,0
DA:48,0
DA:55,0
DA:60,0
DA:61,0
DA:62,0
DA:63,0
DA:64,0
DA:66,0
DA:69,0
DA:74,0
LF:26
LH:0
BRDA:13,0,0,0
BRDA:24,1,0,0
BRDA:24,1,1,0
BRDA:30,2,0,0
BRDA:30,2,1,0
BRDA:32,3,0,0
BRDA:32,3,1,0
BRDA:32,3,2,0
BRDA:46,4,0,0
BRDA:46,4,1,0
BRDA:47,5,0,0
BRDA:47,5,1,0
BRDA:61,6,0,0
BRDA:61,6,1,0
BRDA:63,7,0,0
BRDA:63,7,1,0
BRDA:69,8,0,0
BRDA:69,8,1,0
BRDA:70,9,0,0
BRDA:70,9,1,0
BRDA:70,9,2,0
BRDA:84,10,0,0
BRDA:84,10,1,0
BRF:23
BRH:0
end_of_record
Additional code for coverage report configuration

This snippet is from the vitest.config.mts file under test

coverage: {
      provider: 'istanbul',
      include: ['src/**/*.{ts,tsx}'],
      exclude: [
        ...defaultExclude,
        'src/_cypress/**/*.*',
        'src/storybook/**/*.*',
        'src/api/handlers/**/*.*',
        'src/**/*.{pact.spec.ts,handlers.ts,stories.tsx,.module.scss,cy.tsx}'
      ],
      reporter: ['text', 'html', 'json', 'lcov'],
      reportsDirectory: 'coverage-unit-dir'
    }
sonar-project.properties relevant information

The selected reportPaths are the reports mentioned above

sonar.test.inclusions=**/*.spec.*,**/*.cy.*
sonar.javascript.lcov.reportPaths=coverage-cypress/worker-*/lcov.info,coverage-unit/worker-*/coverage-unit-dir/lcov.info

Any information additional help would be very useful! Thank you :smile:

2
5

Hi @Evdokia_Mina , thanks a lot for the thorough issue reporting, this is very helpful.

The branch count discrepancy is intriguing: Istanbul is correct, there is always an alternate branch for each if statement - explicit or implicit. The intriguing part is the fact that the Cypress-generated report does not include those implicit else.

Would you mind sharing how you execute the Cypress test with coverage?

About v8: coverage tools that only rely on v8 coverage data can’t be trusted, by nature. The coverage data emitted by v8 only knows about the lines that were hit during the execution of the script. A non-hit line consisting of a if is not detected as a branch since it was not interpreted at all by v8. This is why the branch count is unreliable when using v8 as provider: it depends on the lines that were actually executed.

On the other hands, tools that instrument the scripts before executing them (like Istanbul) or that mix v8 coverage data with parsing (like One-Double-Zero) are able to report the correct number of branches, regardless of the number of lines that were actually executed.

6

Hi Eric, thanks for your response. I’ll help with this info in Ev’s absence as she’s off for a couple of weeks at the moment.

That’s interesting, I didn’t realise v8 behaved that way. We’ve reverted back to using istanbul for our Vitest unit tests again.

Our script from running Cypress tests is

NODE_ENV=test npx @percy/cli@latest exec -- npx cypress run --parallel --record --component --browser=chrome --env COVERAGE=true

Then in cypress.config.ts under the component object we have

setupNodeEvents(on: PluginEvents, config: PluginConfigOptions) {
   if (process.env.CI) {
       return require('./cypress/plugins')(on, config)
   }
}

plugins/index.js has the following config

module.exports = (on, config) => {
  require('@cypress/code-coverage/task')(on, config)
  return config
}

We’re using the following relevant package versions:

  • @cypress/code-coverage: 3.10.4
  • cypress: 13.4.0
7

Thanks for the report @TomPlumMatillion. Did moving to Istanbul improve the situation?

If not could you please put a public repository in place with a minimal reproducible example so that we can investigate and provide support?

8

Hi Eric, we’ll look at getting round to creating a reproducible example when we can, appreciate your time responding. One of us will come back when we’ve got one, although it may take us a while with priorities and sign-off for the public repo etc. Thanks.

1 Like