Sonar doesn't see all files for analyses (both PR and branch)

SonarQube Cloud
1

Basic Infos

We’re using:

  • Bitbucket Cloud, with Bitbucket pipelines
  • The project is a Typescript codebase on a private SonarCloud project.
  • Logs below the post!
  • scan step:
    - step:
        &sonar # this step needs to run after tests to collect coverage information
        name: SonarCloud Analysis
        image: sonarsource/sonar-scanner-cli:11.3
        clone:
          depth: full
        script:
          - sonar-scanner -Dsonar.token=$SONAR_TOKEN
        caches: [sonar]
  • sonar config
sonar.organization=cl********
sonar.projectKey=cl**********************
sonar.host.url=https://sonarcloud.io
sonar.javascript.lcov.reportPaths=\
  app/coverage/unit/lcov.info, \
  app/coverage/integration/lcov.info
sonar.sources=\
  app/src,\
  lambda/src
sonar.exclusions=\
  app/src/**/test/**,\
  app/src/infrastructure/db/migrations/**,\
  app/src/infrastructure/db/migrations-legacy/**,\
  app/src/infrastructure/db/dataSource*,\
  app/src/interfaces/http/graphql/schema/**,\
  app/src/interfaces/errorHandling/**, \
  app/src/infrastructure/db/model/**/*Migrated.ts

The issue

I’m observing that Sonar missing some code files/paths in analyes (both PR and branch analyses). It became obvious when I isolated the issue on a PR Analysis, where the PR in Bitbucket (MAN-7015 > main) shows 15 edited files, while the Sonar report (after running successfully) gives:

“PR Summary
0 New Lines”

and has no code listed at all. So, this issue is not per se about coverage, but that the source files first of all are not included.

Then, I also observed, that under “Code” on main branch analyses (that runs after merge) some folders and files are missing. These have been merged through previous PR’s and are present in main (code in Bitbucket) but not in code of Sonar. I cannot see an obvious pattern of what paths/files are excluced.

Some examples:

/app/src/infrastructure/external/api has three folders in mainin Bitbucket

  • amazon/sp-api
  • http
  • hubspot

while Sonar has just the hubspot folder. But also not completely, the new PR diff would add to the hubspot folder 13 files, that’s when the problem surfaced.

What I tried so far…

  • obviously, checked the exclusions, with no match (see above)
  • checked the logs (see below)
    • after that, tried flag -Dsonar.scm.exclusions.disabled=true to no avail
    • removed some .gitignore files (couldn’t be the reason, as PR’s in Bitbucket show the diffs
  • earlier, I was thinking maybe my git history is messed up, so I ran git diff main…HEAD locally, while having the feature branch checked out → shows PR diffs correctly
  • tried to find a logical pattern in what files are missing → to no avail
  • I also thought, this is because the code is not actually used (imported) by anything from the project entry point (app/index.ts). I thought, maybe Sonar doesn’t analyze stuff that is not in the build for some reason:
    • I ran:
node -e "
const ts = require('typescript');

const cfg = ts.readConfigFile(
  'tsconfig.json',
  ts.sys.readFile
);

const p = ts.parseJsonConfigFileContent(
  cfg.config,
  ts.sys,
  process.cwd()
);

const tsFiles = p.fileNames.filter(f => f.endsWith('.ts') || f.endsWith('.tsx'));

console.log('TS/TSX files:', tsFiles.length);
console.log('Sample files:');
console.log(tsFiles.join('\n'));
"

    • that command lists indeed the files that are ignored by Sonar, meaning typescript config sees them!

    • for the hypothesis, that Sonar needs them in the build I didn’t find any proof, but just to mention it, those ignored files are not always in the build. I just checked for the added files of the PR and they are in the build. Just wanted to say, I considered this, but rejected. I also created a new PR MAN-7016 to main, which contains the code of MAN-7015 plus code that uses the code of MAN-7015 in the app, and the same files are missing, while the new code appears on the PR

    • I also bumped sonar now in the PR to sonarsource/sonar-scanner-cli:12.0, to no avail on the PR files in Sonar. Still 0 lines of code.

Now, I’m really out of options. Please help.

Logs

+ sonar-scanner -Dsonar.token=$SONAR_TOKEN
12:10:33.648 INFO  Scanner configuration file: /opt/sonar-scanner/conf/sonar-scanner.properties
12:10:33.654 INFO  Project root configuration file: /opt/atlassian/pipelines/agent/build/sonar-project.properties
12:10:33.673 INFO  SonarScanner CLI 7.1.0.4889
12:10:33.675 INFO  Java 17.0.14 Amazon.com Inc. (64-bit)
12:10:33.676 INFO  Linux 6.1.79 amd64
12:10:33.728 INFO  User cache: /opt/sonar-scanner/.sonar/cache
12:10:34.828 INFO  Communicating with SonarQube Cloud
12:10:34.829 INFO  JRE provisioning: os[linux], arch[x86_64]
12:10:38.177 INFO  Starting SonarScanner Engine...
12:10:38.178 INFO  Java 17.0.11 Eclipse Adoptium (64-bit)
12:10:39.389 INFO  Load global settings
12:10:40.049 INFO  Load global settings (done) | time=661ms
12:10:40.115 INFO  Server id: 1BD809FA-AWHW8ct9-T_TB3XqouNu
12:10:40.383 INFO  Loading required plugins
12:10:40.384 INFO  Load plugins index
12:10:40.507 INFO  Load plugins index (done) | time=124ms
12:10:40.508 INFO  Load/download plugins
12:10:41.378 INFO  Load/download plugins (done) | time=870ms
12:10:41.604 INFO  Loaded core extensions: a3s, architecture, sca
12:10:42.100 INFO  Detected project key 'cl**************************' from 'Bitbucket Cloud Pipelines'
12:10:42.100 INFO  Detected organization key 'cl*******' from 'Bitbucket Cloud Pipelines'
12:10:42.105 INFO  Process project properties
12:10:42.118 INFO  Project key: cli**************************
12:10:42.119 INFO  Base dir: /opt/atlassian/pipelines/agent/build
12:10:42.120 INFO  Working dir: /opt/atlassian/pipelines/agent/build/.scannerwork
12:10:42.132 INFO  Found an active CI vendor: 'Bitbucket Pipelines'
12:10:42.135 INFO  Load project branches
12:10:42.918 INFO  Load project branches (done) | time=784ms
12:10:42.923 INFO  Load project settings for component key: 'cl**************************'
12:10:43.359 INFO  Load project settings for component key: 'cl**************************' (done) | time=436ms
12:10:43.825 INFO  Check ALM binding of project 'cl**************************'
12:10:43.936 INFO  Detected project binding: BOUND
12:10:43.937 INFO  Check ALM binding of project 'cl**************************' (done) | time=112ms
12:10:43.939 INFO  Load project pull requests
12:10:44.381 INFO  Load project pull requests (done) | time=441ms
12:10:44.384 INFO  Load branch configuration
12:10:44.385 INFO  Detected analysis for pull request '125' targeting 'main'
12:10:44.385 INFO  Auto-configuring pull request 125
12:10:44.797 INFO  Load branch configuration (done) | time=413ms
12:10:44.806 INFO  Load quality profiles
12:10:45.717 INFO  Load quality profiles (done) | time=911ms
12:10:45.740 INFO  Create analysis
12:10:46.283 INFO  Create analysis (done) | time=544ms
12:10:46.491 INFO  Load active rules
12:10:47.985 INFO  Load active rules (done) | time=1494ms
12:10:48.129 INFO  Organization key: cl**********
12:10:48.129 INFO  Pull request 125 for merge into main from MAN-7015-hubspot-client
12:10:48.149 INFO  Preprocessing files...
12:10:48.624 INFO  2 languages detected in 349 preprocessed files (done) | time=474ms
12:10:48.624 INFO  67 files ignored because of inclusion/exclusion patterns
12:10:48.624 INFO  1987 files ignored because of scm ignore settings
12:10:48.756 INFO  Loading plugins for detected languages
12:10:48.757 INFO  Load/download plugins
12:10:49.739 INFO  Load/download plugins (done) | time=982ms
12:10:49.926 INFO  Load project repositories
12:10:50.255 INFO  Load project repositories (done) | time=329ms
12:10:50.257 INFO  SCM collecting changed files in the branch
12:10:50.403 INFO  SCM collecting changed files in the branch (done) | time=147ms
12:10:50.411 INFO  Indexing files...
12:10:50.412 INFO  Project configuration:
12:10:50.412 INFO    Excluded sources: app/src/**/test/**, app/src/infrastructure/db/migrations/**, app/src/infrastructure/db/migrations-legacy/**, app/src/infrastructure/db/dataSource*, app/src/interfaces/http/graphql/schema/**, app/src/interfaces/errorHandling/**, app/src/infrastructure/db/model/**/*Migrated.ts, **/build-wrapper-dump.json
12:10:50.444 INFO  298 files indexed (done) | time=32ms
12:10:50.453 INFO  Quality profile for json: Sonar way
12:10:50.453 INFO  Quality profile for ts: Sonar way
12:10:50.454 INFO  ------------- Run sensors on module cl**************************
12:10:50.493 INFO  Load metrics repository
12:10:50.611 INFO  Load metrics repository (done) | time=117ms
12:10:50.617 INFO  Sensor cache enabled
12:10:50.621 INFO  Load sensor cache
12:10:51.857 INFO  Load sensor cache (454 KB) | time=1236ms
12:10:52.170 WARN  Failed to initialize bash parser. This may happen on unsupported platforms or when the user home directory does not exist.
12:10:52.457 INFO  Sensor JasminFileCollectorSensor [jasmin]
12:10:52.458 INFO  Sensor JasminFileCollectorSensor [jasmin] (done) | time=2ms
12:10:52.458 INFO  Sensor IaC CloudFormation Sensor [iac]
12:10:52.489 INFO  There are no files to be analyzed for the CloudFormation language
12:10:52.490 INFO  Sensor IaC CloudFormation Sensor [iac] (done) | time=31ms
12:10:52.490 INFO  Sensor IaC cfn-lint report Sensor [iac]
12:10:52.491 INFO  Sensor IaC cfn-lint report Sensor [iac] (done) | time=2ms
12:10:52.492 INFO  Sensor IaC hadolint report Sensor [iac]
12:10:52.492 INFO  Sensor IaC hadolint report Sensor [iac] (done) | time=0ms
12:10:52.492 INFO  Sensor IaC Azure Resource Manager Sensor [iac]
12:10:52.492 INFO  Sensor IaC Azure Resource Manager Sensor is restricted to changed files only
12:10:52.498 WARN  Invalid character encountered in file /opt/atlassian/pipelines/agent/build/app/src/interfaces/sdk/asset-sdk/ams-asset-sdk-1.0.0.tgz at line 1 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
12:10:52.566 WARN  Invalid character encountered in file /opt/atlassian/pipelines/agent/build/app/src/interfaces/sdk/asset-sdk/ams-asset-sdk-1.0.3.tgz at line 1 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
12:10:52.621 WARN  Invalid character encountered in file /opt/atlassian/pipelines/agent/build/app/src/interfaces/sdk/asset-sdk/ams-asset-sdk-1.0.1.tgz at line 1 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
12:10:52.677 WARN  Invalid character encountered in file /opt/atlassian/pipelines/agent/build/app/src/interfaces/sdk/asset-sdk/ams-asset-sdk-1.0.4.tgz at line 1 for encoding UTF-8. Please fix file content or configure the encoding to be used using property 'sonar.sourceEncoding'.
12:10:52.704 INFO  There are no files to be analyzed for the Azure Resource Manager language
12:10:52.704 INFO  Sensor IaC Azure Resource Manager Sensor [iac] (done) | time=211ms
12:10:52.705 INFO  Sensor Java Config Sensor [iac]
12:10:52.713 INFO  There are no files to be analyzed for the Java language
12:10:52.713 INFO  Sensor Java Config Sensor [iac] (done) | time=8ms
12:10:52.713 INFO  Sensor IaC Docker Sensor [iac]
12:10:52.714 INFO  Sensor IaC Docker Sensor is restricted to changed files only
12:10:52.714 INFO  There are no files to be analyzed for the Docker language
12:10:52.714 INFO  Sensor IaC Docker Sensor [iac] (done) | time=1ms
12:10:52.714 INFO  Sensor IaC GitHub Actions Sensor [iac]
12:10:52.715 INFO  There are no files to be analyzed for the GitHub Actions language
12:10:52.715 INFO  Sensor IaC GitHub Actions Sensor [iac] (done) | time=0ms
12:10:52.715 INFO  Sensor IaC Shell Sensor [iac]
12:10:52.715 INFO  Shell analysis will be skipped because the parser could not be initialized.
12:10:52.715 INFO  Sensor IaC Shell Sensor [iac] (done) | time=0ms
12:10:52.716 INFO  Sensor JavaScript/TypeScript analysis [javascript]
12:10:53.130 INFO  Detected os: Linux arch: amd64 alpine: false. Platform: LINUX_X64
12:10:53.130 INFO  Deploy location /opt/sonar-scanner/.sonar/js/node-runtime, tagetRuntime: /opt/sonar-scanner/.sonar/js/node-runtime/node,  version: /opt/sonar-scanner/.sonar/js/node-runtime/version.txt
12:10:56.533 INFO  Using embedded Node.js runtime.
12:10:56.533 INFO  Using Node.js executable: '/opt/sonar-scanner/.sonar/js/node-runtime/node'.
12:10:58.980 ERROR [baseline-browser-mapping] The data in this module is over two months old.  To ensure accurate Baseline data, please update: `npm i baseline-browser-mapping@latest -D`
12:10:59.315 INFO  Memory configuration: OS (18383 MB), Docker (16384 MB), Node.js (4288 MB).
12:10:59.410 INFO  WebSocket client connected on /ws
12:10:59.413 INFO  Plugin version: [11.7.0.36965]
12:11:01.660 ERROR [baseline-browser-mapping] The data in this module is over two months old.  To ensure accurate Baseline data, please update: `npm i baseline-browser-mapping@latest -D`
12:11:02.654 INFO  Found 3 tsconfig.json file(s): [/opt/atlassian/pipelines/agent/build/cdk/tsconfig.json, /opt/atlassian/pipelines/agent/build/lambda/tsconfig.json, /opt/atlassian/pipelines/agent/build/app/tsconfig.json]
12:11:02.654 INFO  0 source files to be analyzed
12:11:02.654 INFO  0/0 source files have been analyzed
12:11:02.660 INFO  JasminAstConsumer done
12:11:02.661 INFO  Hit the cache for 287 out of 287
12:11:02.661 INFO  Miss the cache for 0 out of 287
12:11:02.661 INFO  Sensor JavaScript/TypeScript analysis [javascript] (done) | time=9946ms
12:11:02.661 INFO  Sensor CSS Rules [javascript]
12:11:02.662 INFO  Sensor CSS Rules is restricted to changed files only
12:11:02.662 INFO  No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
12:11:02.662 INFO  Sensor CSS Rules [javascript] (done) | time=1ms
12:11:02.662 INFO  Sensor JavaScript/TypeScript Coverage [javascript]
12:11:02.663 INFO  Analysing [/opt/atlassian/pipelines/agent/build/app/coverage/unit/lcov.info, /opt/atlassian/pipelines/agent/build/app/coverage/integration/lcov.info]
12:11:02.749 WARN  Could not resolve 89 file paths in [/opt/atlassian/pipelines/agent/build/app/coverage/unit/lcov.info, /opt/atlassian/pipelines/agent/build/app/coverage/integration/lcov.info]
12:11:02.749 WARN  First unresolved path: src/infrastructure/db/dataSourceLegacy.ts (Run in DEBUG mode to get full list of unresolved paths)
12:11:02.749 INFO  Sensor JavaScript/TypeScript Coverage [javascript] (done) | time=86ms
12:11:02.749 INFO  Sensor IaC JSON Sensor [iac]
12:11:02.750 INFO  Sensor for language "JSON" is enabled by a feature flag. You can disable it by setting "sonar.json.activate" to false.
12:11:02.763 INFO  3 source files to be analyzed
12:11:02.814 INFO  3/3 source files have been analyzed
12:11:02.815 INFO  Sensor IaC JSON Sensor [iac] (done) | time=65ms
12:11:02.815 INFO  Sensor EnterpriseTextAndSecretsSensor [textenterprise]
12:11:02.815 INFO  Sensor EnterpriseTextAndSecretsSensor is restricted to changed files only
12:11:02.826 INFO  Available processors: 8
12:11:02.826 INFO  Using 8 threads for analysis.
12:11:03.172 INFO  The property "sonar.tests" is not set. To improve the analysis accuracy, we categorize a file as a test file if any of the following is true:
  * The filename starts with "test"
  * The filename contains "test." or "tests."
  * Any directory in the file path is named: "doc", "docs", "test", "tests", "mock" or "mocks"
  * Any directory in the file path has a name ending in "test" or "tests"
12:11:03.243 INFO  Start fetching files for the text and secrets analysis
12:11:03.249 INFO  Using Git CLI to retrieve dirty files
12:11:03.253 WARN  Retrieving only language associated files, make sure to run the analysis inside a git repository to make use of inclusions specified via "sonar.text.inclusions"
12:11:03.254 INFO  There are no files to be analyzed for the text and secrets analysis
12:11:03.254 INFO  Start fetching files for the binary file analysis
12:11:03.254 INFO  There are no files to be analyzed for the binary file analysis
12:11:03.255 INFO  Sensor EnterpriseTextAndSecretsSensor [textenterprise] (done) | time=441ms
12:11:03.261 INFO  ------------- Run sensors on project
12:11:03.383 INFO  Sensor JsSecuritySensorV2 [jasmin]
12:11:03.444 INFO  Unchanged files can be skipped in the current context and all relevant files are unchanged. Skipping taint analysis.
12:11:03.445 INFO  Sensor JsSecuritySensorV2 [jasmin] (done) | time=56ms
12:11:03.445 INFO  Sensor TsArchitectureSensor [architecture]
12:11:03.452 INFO  Architecture analysis is enabled with the following features: legacy
12:11:03.457 INFO  * Protobuf reading starting | memory total=202 | free=124 | used=77 (MB)
12:11:03.457 INFO  * Reading SonarArchitecture UDG data from directory "/opt/atlassian/pipelines/agent/build/.scannerwork/architecture/ts"
12:11:03.459 INFO  * Directory does not exist: "/opt/atlassian/pipelines/agent/build/.scannerwork/architecture/ts"
12:11:03.459 INFO  * Protobuf reading complete | memory total=202 | free=124 | used=77 (MB)
12:11:03.594 INFO  * Build architecture.graph.ts.file_graph.default_perspective hierarchy graph complete (filtered=false) | memory total=202 | free=114 | used=87 (MB)
12:11:03.603 INFO  Sensor TsArchitectureSensor [architecture] (done) | time=162ms
12:11:03.603 INFO  Sensor Zero Coverage Sensor
12:11:03.605 INFO  Sensor Zero Coverage Sensor (done) | time=1ms
12:11:03.794 INFO  ------------- Gather SCA dependencies on project
12:11:03.794 INFO  Checking if SCA is enabled for organization cl**************
12:11:04.029 INFO  Dependency analysis skipped
12:11:04.038 INFO  CPD Executor 82 files had no CPD blocks
12:11:04.038 INFO  CPD Executor Calculating CPD for 205 files
12:11:04.088 INFO  CPD Executor CPD calculation finished (done) | time=49ms
12:11:04.162 INFO  SCM writing changed lines
12:11:04.169 INFO  SCM writing changed lines (done) | time=7ms
12:11:04.239 INFO  Analysis report generated in 141ms, dir size=364 KB
12:11:04.391 INFO  Analysis report compressed in 152ms, zip size=183 KB
12:11:05.146 INFO  Analysis report uploaded in 755ms
12:11:05.148 INFO  ANALYSIS SUCCESSFUL, you can find the results at: https://sonarcloud.io/dashboard?id=cl*********************agement-service&pullRequest=125
12:11:05.148 INFO  Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
12:11:05.148 INFO  More about the report processing at https://sonarcloud.io/api/ce/task?id=AZu******************************
12:11:05.149 INFO  ------------- Upload SCA dependency files
12:11:05.269 INFO  Analysis total time: 23.665 s
12:11:05.270 INFO  SonarScanner Engine completed successfully
12:11:05.608 INFO  EXECUTION SUCCESS
12:11:05.610 INFO  Total time: 31.967s
2

Hi,

Let’s start by seeing what properties analysis is actually getting. Can you add -Dsonar.scanner.internal.dumpToFile=[path to file] to your analysis command line and run analysis, please? The output of this will be not an analysis report submitted to the server, but a file at your designated path with all the property keys and values the analyzer sees. Please redact that as necessary and post it so we can audit it and go from there.

Also,

Do I correctly infer that you’re kicking off analysis from the directory that holds both app and lambda?

 
Thx,
Ann