Coverage mismatch: Istanbul vs Sonar

SonarCloud
1
  • ALM: GitHub
  • CI: Bitrise
  • Sonar plugin: sonar-scanner
  • Language: JS (react native)
  • Config:
sonar.organization=***
sonar.host.url=https://sonarcloud.io
sonar.sources=src
sonar.test.inclusions=src/**/*.test.js
sonar.javascript.lcov.reportPaths=coverage/lcov.info
sonar.sourceEncoding=UTF-8
sonar.login=$SONAR_KEY
sonar.branch.name=$branch
sonar.branch.target=***
sonar.exclusions=**/__tests__/**/* , **/__mocks__/**/* , **/*.png
  • Issue: Delta between istanbul coverage and sonar’s one is around 20%, which is incredibly huge amount. What do I need to do to make coverage is almost the same in both tools?
  • Running info:
INFO: Project root configuration file: NONE
INFO: SonarQube Scanner 3.1.0.1141
INFO: Java 1.8.0_202 Oracle Corporation (64-bit)
INFO: Mac OS X 10.15 x86_64
INFO: User cache: ***/.sonar/cache
INFO: SonarQube server 8.0.0
INFO: Default locale: "en_BY", source code encoding: "UTF-8"
INFO: Load global settings
INFO: Load global settings (done) | time=314ms
INFO: Server id: ***
INFO: User cache: ***/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=68ms
INFO: Load/download plugins (done) | time=1153ms
INFO: Loaded core extensions: developer-scanner
INFO: Process project properties
INFO: Execute project builders
INFO: Execute project builders (done) | time=4ms
INFO: Project key: ***
INFO: Base dir: ***
INFO: Working dir: ***
INFO: Load project settings for component key: '***'
INFO: Load project settings for component key: '***' (done) | time=89ms
INFO: Load project branches
INFO: Load project branches (done) | time=162ms
INFO: Load projects for organization '***'
INFO: Load projects for organization '***' (done) | time=874ms
INFO: Load project pull requests
INFO: Load project pull requests (done) | time=78ms
INFO: Load branch configuration
INFO: Load branch configuration (done) | time=2ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=87ms
INFO: Load active rules
INFO: Load active rules (done) | time=1732ms
INFO: Organization key: ***
INFO: Branch name: feat/ACM-186, type: short living
INFO: SCM collecting changed files in the branch
INFO: SCM collecting changed files in the branch (done) | time=146ms
INFO: Indexing files...
INFO: Project configuration:
INFO:   Excluded sources: **/__tests__/**/*, **/__mocks__/**/*, **/*.png, src/**/*.test.js
INFO:   Included tests: src/**/*.test.js
INFO: 240 files indexed
INFO: 0 files ignored because of inclusion/exclusion patterns
INFO: 0 files ignored because of scm ignore settings
INFO: Quality profile for js: Sonar way Recommended
INFO: ------------- Run sensors on module mobile-activation-app
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=53ms
INFO: Sensor JavaXmlSensor [java]
INFO: Sensor JavaXmlSensor [java] (done) | time=2ms
INFO: Sensor HTML [web]
INFO: Sensor HTML [web] (done) | time=5ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=4ms
INFO: Sensor ESLint-based JavaScript analysis [javascript]
INFO: 186 source files to be analyzed
INFO: Sensor ESLint-based JavaScript analysis [javascript] (done) | time=5561ms
INFO: Sensor SonarJS Coverage [javascript]
INFO: 186/186 source files have been analyzed
INFO: Analysing [***/coverage/lcov.info]
WARN: Could not resolve 9 file paths in [***/coverage/lcov.info], first unresolved path:***/src/app/assets/images/barcode-inverted.png
INFO: Sensor SonarJS Coverage [javascript] (done) | time=79ms
INFO: Sensor SonarJS [javascript]
INFO: 186 source files to be analyzed
INFO: Sensor SonarJS [javascript] (done) | time=1747ms
INFO: Sensor JavaSecuritySensor [security]
INFO: 186/186 source files have been analyzed
INFO: Reading type hierarchy from: ***
INFO: Read 0 type definitions
INFO: Reading UCFGs from: ***
INFO: No UCFGs have been included for analysis.
INFO: Sensor JavaSecuritySensor [security] (done) | time=4ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading type hierarchy from: ***
INFO: Read 0 type definitions
INFO: Reading UCFGs from: ***
INFO: No UCFGs have been included for analysis.
INFO: Sensor CSharpSecuritySensor [security] (done) | time=0ms
INFO: Sensor PhpSecuritySensor [security]
INFO: Reading type hierarchy from: ***
INFO: Read 0 type definitions
INFO: Reading UCFGs from: ***
INFO: No UCFGs have been included for analysis.
INFO: Sensor PhpSecuritySensor [security] (done) | time=0ms
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=6ms
INFO: 49 files had no CPD blocks
INFO: Calculating CPD for 137 files
INFO: CPD calculation finished
INFO: SCM writing changed lines
INFO: SCM writing changed lines (done) | time=4ms
INFO: Analysis report generated in 103ms, dir size=234 KB
INFO: Analysis report compressed in 242ms, zip size=169 KB
INFO: Analysis report uploaded in 172ms
INFO: ANALYSIS SUCCESSFUL, you can browse ***
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at ***
INFO: Analysis total time: 15.655 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 20.837s
INFO: Final Memory: 37M/574M
INFO: ------------------------------------------------------------------------
✨  Done in 42.81s.```
3

Hello!

And welcome to the community forum!

You could check in the SonarCloud UI which files have a coverage metric and that don’t in Istanbul. You might have a different configurations between the two and maybe you want to ignore some files in SonarCloud with the sonar.coverage.exclusions setting.

1 Like
4

Thank you for reply, but unfortunately (and I have no idea why) sonar says that react-native libs are not covered by tests. I tried to exclude node_modules, but not succeed with that attempt.

5

So you did change your config to add the exclusion in sonar.coverage.exclusions ? And the node_modules files are still showing in SonarCloud after a new analysis ? Can you double check that the exclusion path is correct ?

And something weird is that normally the node_modules are ignored by default by our js analyzer.

Also could it be that your coverage tool is computing the coverage for your node_modules folder ?

1 Like
6

I tried to add node_modules to exclusions in config, result the same.

Istanbul ignores node_modules as well.
As example:
Istanbul: import View from ‘react-native’ -> no problems
Sonar cloud: same import -> 20/34 covered

And such issue on most of files.

7

And what does the lcov file says about the same file? The coverage values that appears in SonarCloud comes directly from reading this lcov file. So for a given file you should not see any difference between what is in the lcov file and what’s reported in SonarCloud.

One other thing, there is different coverage values displayed in SonarCloud :

  • line coverage: coverage based on line, as reported in the lcov file
  • condition coverage: coverage based on the conditions, as reported in the lcov file
  • coverage: is the result of a calcul based on line and condition coverage
1 Like
8

So the data in lcov related to the file:

FN:5,_interopRequireDefault
FN:7,clearAllStores
FNF:2
FNH:1
FNDA:33,_interopRequireDefault
FNDA:0,clearAllStores
DA:1,33
DA:5,33
DA:8,0
DA:13,33
DA:14,33
LF:5
LH:4
BRDA:5,0,0,33
BRDA:5,0,1,0
BRDA:5,1,0,33
BRDA:5,1,1,33
BRF:4
BRH:3
end_of_record

How it looks in istanbul vs sonar:

44
Screen Shot 2019-11-11 at 15.05.44.png2396×863 176 KB

The question is why coverage is different here, and sonar points cases which are excluded in istanbul.

9

Did you end up finding out why? I’m new to this tool and I’m stuck with this problem.