Conditional Coverage Discrepancy From Lcov.info Generated Report

  • SonarQube Enterprise Edition v10.6
  • Helm
  • Typescript/Javascript repositories that uses jest test framework to generate coverage reports

Hello there!

When scanning a repository for code coverage I want the conditional coverage metric to closely if not match exactly what the branch coverage is reflecting from within the lcov.info coverage report. At times there are discrepancies between the lcov.info report generated by jest and the results displayed in SonarQube that results in the failure of the quality gate.

The line coverage matches up everytime, however we are noticing that condition coverage from SonarQube does not match the branch coverage that displays in the lcov.info.

One prime example is that the lcov.info files shows 100% coveraged in both line and function for a single file but in SonarQube for the conditional coverage is displaying 37.50%. Note our repositories combine multiple coverage reports that are combined into a single lcov.info file before the sonar scan begins

cat coverage/*info > lcov.info

1.) I would like to understand why we are seeing a discrepancy from the conditional coverage
2.) have an understanding of how the conditional coverage is determined when dealing with 2 or more coverage reports. Because we’ve executed a test with just submitting the unit test to sonar and didn’t see any issues, so this looks to be due to merging the reports into a single file.
3.) would it be best to submit the coverage reports separately in order receive correct results ?

Below I’ve attached the source code file that is showing the discrepency

import { AppEnvs, ContentfulEnvs } from 'lib/constants';

const contentfulEnvs = Object.values(ContentfulEnvs);
type ContentfulEnv = typeof contentfulEnvs[number];

export function getContentfulEnv(): ContentfulEnv {
  const appEnv = process.env.APP_ENV;

  if (appEnv === AppEnvs.PROD) {
    return ContentfulEnvs.MASTER;
  }

  if (appEnv === AppEnvs.STAGE) {
    return ContentfulEnvs.STAGE;
  }

  if (appEnv === AppEnvs.DEV) {
    return ContentfulEnvs.DEV;
  }

  return ContentfulEnvs.DEVELOPMENT;
}


Thank you!

Hey there.

I’m surprised you’re seeing the discrepancy even when combining the lcov.info reports into one. Could you provide it here? Sonar just reads the report, so the report should pretty clearly indicate uncovered conditions (at which point it’s not Sonar’s problem, it’s an issue with the merge)

coverage-exercise.zip (170.2 KB)

Here are the coverage reports

lcov.info.zip (169.5 KB)

Colin I also provided the combined lcov.info file that is generated withing the GHA pipeline before the sonar scan is run in CI/CD

Hey there.

Thanks for the details.

I’ve flagged this for review because indeed, it seems like the merge of branch coverage isn’t working as I would expect. Using the merged LCOV file vs. separate files didn’t make a difference.

That being said, I’m not sold that all of your coverage files are… accurate? For example, lcov-integration.info shows lines being covered that are just blank lines. Is that really intended?

Hey Colin,

Thanks for taking a deeper dive into this issue!

In regards to the lines being counted as statements for jest we use the v8 coverage provider and I did some investigation it looks like based on this thread here (Lines without statements (empty lines) on them are counted as statements · Issue #500 · bcoe/c8 · GitHub) the v8 coverage provider turns empty lines into statements rather than skipping the empty lines in the code.

Hi Colin,

do we have an estimated timeline when this issue will be reviewed by the team ?

I don’t have an ETA to share, sorry.

@Elijah_Taylor-Kuni , thanks for your patience, I’m currently looking into your issue.

When I inspect the LCOV files that you shared with us, I see that your coverage tool detects 7 branches on the src/config/get-contentful-env.ts. This seems strange to me: the content that you shared only contains 3 branches (coming from the 3 if statements), which is confirmed when I have the coverage computed using some other coverage tools.

This is the BRDA sections contained in the LCOV files that you shared, for this specific source file:

BRDA:6,0,0,6
BRDA:9,1,0,1
BRDA:12,2,0,5
BRDA:13,3,0,1
BRDA:16,4,0,4
BRDA:17,5,0,1
BRDA:20,6,0,3

Line 6 is a function, and technically not a branch.
Line 12 is an empty line.
Line 16 is an empty line.
Line 20 is an empty line.

When computing the coverage with NYC or One Double Zero, I get the following correct LCOV BRDA sections:

BRDA:9,0,0,0
BRDA:13,1,0,0
BRDA:17,2,0,0

Is the actual content of the file different from the one you shared with us?

Hi Eric, sorry for the delayed response just seeing this now so which zip are you looking at because if you look in the lcov.info.zip zip that I sent which is the combined zip file of all coverage reports and the one I am sending to sonarqube I believe the results are matching up with yours

SF:src/config/get-contentful-env.ts
FN:6,getContentfulEnv
FNF:1
FNH:0
FNDA:0,getContentfulEnv
DA:1,34
DA:3,34
DA:6,34
DA:7,0
DA:9,0
DA:10,0
DA:13,0
DA:14,0
DA:17,0
DA:18,0
DA:21,0
LF:11
LH:3
BRDA:9,0,0,0
BRDA:13,1,0,0
BRDA:17,2,0,0
BRF:3
BRH:0
end_of_record

Hi Elijah,

Actually, from line 67502 to 67541, we have another coverage report for src/config/get-contentful-env.ts::getContentfulEnv:

TN:
SF:src/config/get-contentful-env.ts
FN:6,getContentfulEnv
FNF:1
FNH:1
FNDA:6,getContentfulEnv
DA:1,1
DA:2,1
DA:3,1
DA:4,1
DA:5,1
DA:6,6
DA:7,6
DA:8,6
DA:9,6
DA:10,1
DA:11,1
DA:12,5
DA:13,6
DA:14,1
DA:15,1
DA:16,4
DA:17,6
DA:18,1
DA:19,1
DA:20,3
DA:21,3
DA:22,3
LF:22
LH:22
BRDA:6,0,0,6
BRDA:9,1,0,1
BRDA:12,2,0,5
BRDA:13,3,0,1
BRDA:16,4,0,4
BRDA:17,5,0,1
BRDA:20,6,0,3
BRF:7
BRH:7
end_of_record

This section declares 7 branches.

This is confirmed by https://lcov-viewer.netlify.app/ when I upload there the LCOV report that is contained inside the zip archive.

This 7 branches thing may not be relevant to your current issue (we are investigating merged reports regardless), but I would be interested in understanding where this discrepancy in branches come from.

Hmm… interesting I am not entirely sure why we are seeing this discrepancy what we do in our CI is that we have three independent jobs that execute in GHA and they all generate the coverage reports using jest test coverage and then upload the lcov.info files to GHA artifacts in the CI run. ( coverage-exercise.zip contains the converage reports )

Then below is the list of steps that is executed before running the GHA sonar job

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/download-artifact@v3
        with:
          name: coverage-artifacts
          path: coverage
      - name: Merge Code Coverage
        run: cat coverage/*info > lcov.info

So possibly the issues is from the step we are using to merge the coverage reports together ? Is there a preferred approach that should be used to merge the coverage reports into a single entity upon executing the sonar scan to produce better results?

@Elijah_Taylor-Kuni , sorry for the late answer.

Can you please try to not combine the files and pass them individually to the scanner configuration? I’m curious to see if the combining itself is responsible of the issue or not.

You can pass multiple files and/or globs to sonar.javascript.lcov.reportPaths, separated by commas.

About the coverage computation itself:

using jest test coverage

Are you using v8 as coverage provider?