Coverage reporting stopped working for multiple projects on CI

florian_ec · September 10, 2021, 9:47am

We have Sonarcloud setup with our CI (CircleCI) to report coverage for two TypeScript-based projects. At some point in mid August the coverage numbers stopped showing up in Sonarcloud and while debugging it I found that the coverage is showing up in Sonarcloud if I run the scanner command locally (Mac OS). It also works if I download the LCOV file that is generated by CI and use it in the local scanner call. However, if the Scanner is running on CI (Linux) it does not work.

At the time the coverage reporting stopped working we didn’t change anything related to how we generate coverage or run the scanner during the time. In addition this is happening for two different projects that worked perfectly for over a year. Before starting the investigation we were using Sonarscanner 4.5.0.2216, but the same is happening with 4.6.2.2472.

I’m adding the command we use to run the Scanner and the output of the Scanner at the end of this post. Any help would be greatly appreciated.

$SCANNER_DIRECTORY/sonar-scanner-$VERSION-$OS/bin/sonar-scanner \
  -Dsonar.host.url=https://sonarcloud.io \
  -Dsonar.projectBaseDir=$(pwd) \
  -Dsonar.organization=[[ORG_KEY]] \
  -Dsonar.projectKey=[[PROJECT_KEY]] \
  -Dsonar.projectVersion=$PROJECT_VERSION \
  -Dsonar.sources=src \
  -Dsonar.tests=src \
  -Dsonar.exclusions=src/setupTests.ts \
  -Dsonar.test.inclusions=src/**/*.test.js,src/**/*.test.jsx,src/**/*.test.ts,src/**/*.test.tsx \
  -Dsonar.testExecutionReportPaths=/tmp/workspace/test-results/jest/sonar-results.xml \
  -Dsonar.javascript.lcov.reportPaths=/tmp/workspace/coverage/lcov.info \
  $( [[ -n $PR_NUMBER ]] && echo "-Dsonar.pullrequest.branch=$CIRCLE_BRANCH" ) \
  $( [[ -n $PR_NUMBER ]] && echo "-Dsonar.pullrequest.base=develop" ) \
  $( [[ -n $PR_NUMBER ]] && echo "-Dsonar.pullrequest.key=$PR_NUMBER" ) \
  $( [[ -n $COVERAGE_EXCLUSIONS ]] && echo "-Dsonar.coverage.exclusions=$COVERAGE_EXCLUSIONS,src/*/test/**/*" ) \
  $( [[ -n $COVERAGE_EXCLUSIONS ]] && echo "-Dsonar.cpd.exclusions=$COVERAGE_EXCLUSIONS,src/*/test/**/*" )

/tmp/cache/scanner/.sonar
Intranet version: 1.15.0
Pull Request number: 209
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  2 41.1M    2  864k    0     0   793k      0  0:00:53  0:00:01  0:00:52  793k
 39 41.1M   39 16.1M    0     0  8004k      0  0:00:05  0:00:02  0:00:03 8003k
 81 41.1M   81 33.3M    0     0  10.8M      0  0:00:03  0:00:03 --:--:-- 10.8M
100 41.1M  100 41.1M    0     0  11.6M      0  0:00:03  0:00:03 --:--:-- 11.6M
INFO: Scanner configuration file: /tmp/cache/scanner/sonar-scanner-4.6.2.2472-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarScanner 4.6.2.2472
INFO: Java 11.0.11 AdoptOpenJDK (64-bit)
INFO: Linux 4.15.0-1106-aws amd64
INFO: User cache: /tmp/cache/scanner/.sonar/cache
INFO: Scanner configuration file: /tmp/cache/scanner/sonar-scanner-4.6.2.2472-linux/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: Analyzing on SonarCloud
INFO: Default locale: "en", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Load global settings
INFO: Load global settings (done) | time=577ms
INFO: Server id: 1BD809FA-AWHW8ct9-T_TB3XqouNu
INFO: User cache: /tmp/cache/scanner/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=138ms
INFO: Load/download plugins (done) | time=427ms
INFO: Loaded core extensions: developer-scanner
INFO: Found an active CI vendor: 'CircleCI'
INFO: Load project settings for component key: '[[PROJECT_KEY]]'
INFO: Load project settings for component key: '[[PROJECT_KEY]]' (done) | time=114ms
INFO: Process project properties
INFO: Execute project builders
INFO: Execute project builders (done) | time=1ms
INFO: Project key: [[PROJECT_KEY]]
INFO: Base dir: /home/circleci/project
INFO: Working dir: /home/circleci/project/.scannerwork
INFO: Load project branches
INFO: Load project branches (done) | time=117ms
INFO: Check ALM binding of project '[[PROJECT_KEY]]'
INFO: Detected project binding: BOUND
INFO: Check ALM binding of project '[[PROJECT_KEY]]' (done) | time=99ms
INFO: Load project pull requests
INFO: Load project pull requests (done) | time=146ms
INFO: Load branch configuration
INFO: Load branch configuration (done) | time=583ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=150ms
INFO: Load active rules
INFO: Load active rules (done) | time=3885ms
INFO: Organization key: [[ORG_KEY]]
INFO: Pull request 209 for merge into develop from feature/IN-321
INFO: SCM collecting changed files in the branch
INFO: SCM collecting changed files in the branch (done) | time=220ms
INFO: Indexing files...
INFO: Project configuration:
INFO:   Excluded sources: src/setupTests.ts, src/**/*.test.js, src/**/*.test.jsx, src/**/*.test.ts, src/**/*.test.tsx
INFO:   Included tests: src/**/*.test.js, src/**/*.test.jsx, src/**/*.test.ts, src/**/*.test.tsx
INFO:   Excluded sources for coverage: src/common/slices/index.ts, src/index.tsx, src/reduxStore.ts, src/*/test/**/*
INFO: 1233 files indexed
INFO: 1238 files ignored because of inclusion/exclusion patterns
INFO: 0 files ignored because of scm ignore settings
INFO: Quality profile for ts: Sonar way
INFO: ------------- Run sensors on module [[PROJECT_KEY]]
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=105ms
INFO: Sensor CSS Rules [cssfamily]
INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
INFO: Sensor CSS Rules [cssfamily] (done) | time=1ms
INFO: Sensor C# Project Type Information [csharp]
INFO: Sensor C# Project Type Information [csharp] (done) | time=1ms
INFO: Sensor C# Analysis Log [csharp]
INFO: Sensor C# Analysis Log [csharp] (done) | time=10ms
INFO: Sensor C# Properties [csharp]
INFO: Sensor C# Properties [csharp] (done) | time=0ms
INFO: Sensor JavaXmlSensor [java]
INFO: Sensor JavaXmlSensor [java] (done) | time=3ms
INFO: Sensor HTML [web]
INFO: Sensor HTML [web] (done) | time=5ms
INFO: Sensor VB.NET Project Type Information [vbnet]
INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=2ms
INFO: Sensor VB.NET Analysis Log [vbnet]
INFO: Sensor VB.NET Analysis Log [vbnet] (done) | time=10ms
INFO: Sensor VB.NET Properties [vbnet]
INFO: Sensor VB.NET Properties [vbnet] (done) | time=0ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=4ms
INFO: Sensor TypeScript analysis [javascript]
INFO: Deploying custom rules bundle jar:file:/tmp/cache/scanner/.sonar/cache/d7082afdb883d277963963dc71b96465/sonar-securityjsfrontend-plugin.jar!/js-vulnerabilities-rules-1.0.0.tgz to /home/circleci/project/.scannerwork/.sonartmp/eslint-bridge-bundle/package/custom-rules6949725728457162470
INFO: Found 2 tsconfig.json file(s): [/home/circleci/project/e2e-tests/tsconfig.json, /home/circleci/project/tsconfig.json]
INFO: Analyzing 693 files using tsconfig: /home/circleci/project/tsconfig.json
INFO: 693 source files to be analyzed
INFO: 7/693 files analyzed, current file: src/common[...]
INFO: 239/693 files analyzed, current file: src/backend/components/BackendLayout/BackendLayout.tsx
INFO: 457/693 files analyzed, current file: src/commerce/components/BundleDetailsPageActions/EditProgramDialogBody.tsx
INFO: 693/693 source files have been analyzed
INFO: Sensor TypeScript analysis [javascript] (done) | time=40953ms
INFO: Sensor JavaScript/TypeScript Coverage [javascript]
INFO: Analysing [/tmp/workspace/coverage/lcov.info]
INFO: Sensor JavaScript/TypeScript Coverage [javascript] (done) | time=614ms
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend]
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend] (done) | time=2ms
INFO: Sensor Serverless configuration file sensor [securityjsfrontend]
INFO: 0 Serverless function entries were found in the project
INFO: 0 Serverless function handlers were kept as entrypoints
INFO: Sensor Serverless configuration file sensor [securityjsfrontend] (done) | time=80ms
INFO: Sensor AWS SAM template file sensor [securityjsfrontend]
INFO: Sensor AWS SAM template file sensor [securityjsfrontend] (done) | time=1ms
INFO: Sensor Generic Test Executions Report
INFO: Parsing /tmp/workspace/test-results/jest/sonar-results.xml
INFO: Imported test execution data for 513 files
INFO: Sensor Generic Test Executions Report (done) | time=81ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Reading type hierarchy from: /home/circleci/project/.scannerwork/ucfg2/java
INFO: Read 0 type definitions
INFO: Reading UCFGs from: /home/circleci/project/.scannerwork/ucfg2/java
INFO: No UCFGs have been included for analysis.
INFO: Sensor JavaSecuritySensor [security] (done) | time=2ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading type hierarchy from: /home/circleci/project/ucfg_cs2
INFO: Read 0 type definitions
INFO: Reading UCFGs from: /home/circleci/project/ucfg_cs2
INFO: No UCFGs have been included for analysis.
INFO: Sensor CSharpSecuritySensor [security] (done) | time=0ms
INFO: Sensor PhpSecuritySensor [security]
INFO: Reading type hierarchy from: /home/circleci/project/.scannerwork/ucfg2/php
INFO: Read 0 type definitions
INFO: Reading UCFGs from: /home/circleci/project/.scannerwork/ucfg2/php
INFO: No UCFGs have been included for analysis.
INFO: Sensor PhpSecuritySensor [security] (done) | time=0ms
INFO: Sensor PythonSecuritySensor [security]
INFO: Reading type hierarchy from: /home/circleci/project/.scannerwork/ucfg2/python
INFO: Read 0 type definitions
INFO: Reading UCFGs from: /home/circleci/project/.scannerwork/ucfg2/python
INFO: No UCFGs have been included for analysis.
INFO: Sensor PythonSecuritySensor [security] (done) | time=0ms
INFO: Sensor JsSecuritySensor [security]
INFO: Reading type hierarchy from: /home/circleci/project/.scannerwork/ucfg2/js
INFO: Read 0 type definitions
INFO: Reading UCFGs from: /home/circleci/project/.scannerwork/ucfg2/js
INFO: 11:50:46.829099 Building Runtime Type propagation graph
INFO: 11:50:46.910359 Running Tarjan on 18119 nodes
INFO: 11:50:46.925983 Tarjan found 18119 components
INFO: 11:50:46.955815 Variable type analysis: done
INFO: 11:50:46.957291 Building Runtime Type propagation graph
INFO: 11:50:47.032056 Running Tarjan on 18119 nodes
INFO: 11:50:47.047686 Tarjan found 18119 components
INFO: 11:50:47.076773 Variable type analysis: done
INFO: Analyzing 2354 ucfgs to detect vulnerabilities.
INFO: rule: S5146, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S5146 done
INFO: rule: S2083, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S2083 done
INFO: rule: S6096, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S6096 done
INFO: rule: S5147, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S5147 done
INFO: rule: S3649, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S3649 done
INFO: rule: S5334, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S5334 done
INFO: rule: S2076, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S2076 done
INFO: rule: S2631, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S2631 done
INFO: rule: S6105, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S6105 done
INFO: rule: S5883, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S5883 done
INFO: rule: S5696, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S5696 done
INFO: rule: S5144, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S5144 done
INFO: rule: S5131, entrypoints: 696
INFO: Running symbolic analysis
INFO: rule: S5131 done
INFO: Sensor JsSecuritySensor [security] (done) | time=11492ms
INFO: ------------- Run sensors on project
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=4ms
INFO: CPD Executor 98 files had no CPD blocks
INFO: CPD Executor Calculating CPD for 537 files
INFO: CPD Executor CPD calculation finished (done) | time=71ms
INFO: SCM writing changed lines
INFO: SCM writing changed lines (done) | time=7ms
INFO: Analysis report generated in 203ms, dir size=610 KB
INFO: Analysis report compressed in 657ms, zip size=752 KB
INFO: Analysis report uploaded in 671ms
INFO: ANALYSIS SUCCESSFUL, you can find the results at: https://sonarcloud.io/dashboard?id=[[PROJECT_KEY]]&pullRequest=209
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at https://sonarcloud.io/api/ce/task?id=[[TAS_ID]]
INFO: Analysis total time: 1:09.510 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 1:12.212s
INFO: Final Memory: 93M/324M
INFO: ------------------------------------------------------------------------

Antonio_Garcia · September 16, 2021, 7:21am

Hi @florian_ec , thank you for your patience. Your problem might be related to this other thread. I see in your sonnar-scanner command you are using the correct coverage parameter but, could you check that in your CI you are not using the deprecated sonar.typescript.lcov.reportPaths and instead the sonar.javascript.lcov.reportPaths?

You can read more in our documentation.

Please, let us know if this is helpful for you or if you would need anything else!

Best regards,
Antonio

florian_ec · September 16, 2021, 8:16am

Thanks for the response, @Antonio_Garcia

Yes, we are using sonar.javascript.lcov.reportPaths also on CI. This deprecation was the first thing I noticed when starting to debug this. Is there anything else that could cause this?

florian_ec · September 21, 2021, 2:32pm

@Antonio_Garcia Do you have any other ideas what we can try to fix this?

florian_ec · October 14, 2021, 2:02pm

I was trying again to fix this issue and looked into a couple of things. I tried replacing the relative path in the lcov.info file with absolute paths, but I get the same result: 0% code coverage.

I don’t know if this helps, but the DEBUG logs don’t show any errors or warnings or the JS/TS coverage:

16:00:04.185 INFO: Sensor JavaScript/TypeScript Coverage [javascript]
16:00:04.185 DEBUG: Using 'coverage/lcov.info' to resolve LCOV files
16:00:06.318 INFO: Analysing [/Users/florianeckerstorfer/Projects/intranet/coverage/lcov.info]
16:00:06.333 INFO: Sensor JavaScript/TypeScript Coverage [javascript] (done) | time=2148ms

Lena · January 28, 2022, 4:48pm

@florian_ec did you manage to fix your issue?
That’s really weird that you get no coverage and there is no errors or warning in the log. There should be a log if some files from report are not matched. The only scenario I can imagine if that report file is somehow empty during analysis on CI.