Sonarqube analyzes d.ts in node_modules for each tsconfig.json

Yinan_Wang · May 18, 2023, 10:38am

Must-share information (formatted with Markdown):

which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension):
Enterprise 9.9.0.65466 SonarScanner 4.8.0.2856
how is SonarQube deployed: zip, Docker, Helm
what are you trying to achieve
scan source code
what have you tried so far to achieve this

Hello, I have a nx monorepo project which has many libs inside and each one has tsconfig.json, tsconfig.app.json and tsconfig.spec.json. After upgrading to sonarqube v9, for each config json, it tries to analyze node_modules d.ts files even though I have excluded it.
the blow log shows one of them
Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!

 Sensor TypeScript analysis [javascript]
2023-05-18T09:28:13.4332298Z 17:28:13.418 DEBUG: eslint-bridge server is up, no need to start.
2023-05-18T09:28:13.4338818Z 17:28:13.418 DEBUG: Analysis of unchanged files will not be skipped (current analysis requires all files to be analyzed)
2023-05-18T09:28:13.4379615Z 17:28:13.434 DEBUG: Initializing linter "default" with no-commented-code,sonar-no-fallthrough,duplicates-in-character-class,no-inverted-boolean-check,file-uploads,sonar-no-misleading-character-class,dns-prefetching,certificate-transparency,no-same-argument-assert,arguments-order,single-char-in-character-classes,aws-ec2-unencrypted-ebs-volume,no-unsafe-finally,prefer-while,unused-named-groups,single-character-alternation,aws-iam-public-access,no-ip-forward,session-regeneration,no-use-of-empty-return-value,no-associative-arrays,no-weak-keys,confidential-information-logging,no-throw-literal,weak-ssl,no-useless-increment,pseudo-random,no-redundant-optional,cookie-no-httponly,aws-s3-bucket-public-access,post-message,constructor-for-side-effects,no-globals-shadowing,unverified-hostname,hashing,for-loop-increment-sign,no-vue-bypass-sanitization,no-nested-conditional,no-unnecessary-type-assertion,insecure-jwt-token,no-dead-store,prefer-type-guard,use-type-alias,no-in-misuse,no-parameter-reassignment,no-unstable-nested-components,jsx-key,updated-loop-counter,link-with-target-blank,no-array-index-key,concise-regex,stateful-regex,test-check-exception,max-switch-cases,production-debug,no-undefined-argument,csrf,cognitive-complexity,no-labels,use-isnan,no-nested-template-literals,inverted-assertion-arguments,aws-s3-bucket-insecure-http,generator-without-yield,no-duplicate-in-composite,no-ignored-return,no-caller,aws-s3-bucket-server-encryption,call-argument-line,no-unenclosed-multiline-block,no-uniq-key,jsx-no-constructed-context-values,no-redundant-boolean,content-security-policy,prefer-promise-shorthand,regex-complexity,no-empty-after-reluctant,assertions-in-tests,no-intrusive-permissions,disabled-resource-integrity,unused-import,empty-string-repetition,no-nested-assignment,index-of-compare-to-positive-number,no-unsafe-unzip,aws-s3-bucket-versioning,todo-tag,strict-transport-security,no-mime-sniff,prefer-default-last,no-gratuitous-expressions,no-referrer-policy,no-empty-pattern,sonar-no-invalid-regexp,no-mixed-content,anchor-precedence,no-angular-bypass-sanitization,frame-ancestors,slow-regex,fixme-tag,aws-s3-bucket-granted-access,new-operator-misuse,non-existent-operator,no-small-switch,aws-iam-privilege-escalation,jsx-no-comment-textnodes,prefer-for-of,aws-sagemaker-unencrypted-notebook,sonar-jsx-no-leaked-render,default-param-last,xml-parser-xxe,sql-queries,no-global-this,no-array-delete,no-alphabetical-sort,require-render-return,no-sequences,no-octal,void-use,no-hardcoded-ip,comma-or-logical-or-case,label-position,existing-groups,aws-sns-unencrypted-topics,super-invocation,no-try-promise,aws-ec2-rds-dms-public,sonar-no-control-regex,no-empty-alternatives,sonar-no-regex-spaces,prefer-regex-literals,no-same-line-conditional,sonar-no-unused-class-component-methods,rules-of-hooks,no-identical-functions,no-useless-react-setstate,aws-restricted-ip-admin-access,no-hook-setter-in-body,no-element-overwrite,no-equals-in-for-termination,no-sparse-arrays,no-var,no-redundant-jump,no-duplicate-imports,no-unthrown-error,no-os-command-from-path,no-collection-size-mischeck,prefer-namespace-keyword,unverified-certificate,no-empty-collection,disabled-auto-escaping,no-empty-group,aws-sqs-unencrypted-queue,aws-apigateway-public-api,cors,aws-efs-unencrypted,no-accessor-field-mismatch,insecure-cookie,no-unused-collection,no-invariant-returns,no-case-label-in-switch,os-command,no-misleading-array-reverse,no-redundant-assignments,no-all-duplicated-branches,no-identical-conditions,no-weak-cipher,no-delete-var,encryption-secure-mode,no-useless-intersection,sonar-block-scoped-var,no-empty-function,no-code-after-done,no-unsafe-negation,deprecation,misplaced-loop-counter,no-one-iteration-loop,no-multi-str,function-inside-loop,no-duplicated-branches,disabled-timeout,bitwise-operators,class-name,code-eval,no-identical-expressions,no-unreachable,no-useless-catch,x-powered-by,file-permissions,publicly-writable-directories,no-incomplete-assertions,chai-determinate-assertion,sonar-no-dupe-keys,no-primitive-wrappers,no-exclusive-tests,aws-opensearchservice-domain,no-self-assign,no-misused-new,aws-iam-all-privileges,no-invalid-await,no-hardcoded-credentials,aws-rds-unencrypted-databases,content-length,no-clear-text-protocols,no-empty,sonar-max-params,hidden-files,no-unused-expressions,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg,ucfg
2023-05-18T09:28:13.4402148Z 17:28:13.434 DEBUG: Loaded rules ucfg from D:\cd077a\_w\1\s\.scannerwork\.sonartmp\eslint-bridge-bundle\package\custom-rules17949231816446481745\package
2023-05-18T09:28:13.4451524Z 17:28:13.434 DEBUG: Analysis of unchanged files will not be skipped (current analysis requires all files to be analyzed)
2023-05-18T09:28:13.8436599Z 17:28:13.840 INFO: Found 1 tsconfig.json file(s): [D:\cd077a\_w\1\s\tsconfig.json]
2023-05-18T09:28:13.8442587Z 17:28:13.840 INFO: 1172 source files to be analyzed
2023-05-18T09:28:13.8443232Z 17:28:13.840 INFO: Creating TypeScript program
2023-05-18T09:28:13.8445441Z 17:28:13.840 INFO: TypeScript configuration file D:\cd077a\_w\1\s\tsconfig.json
2023-05-18T09:28:21.7832767Z 17:28:21.778 DEBUG: program from D:\cd077a\_w\1\s\tsconfig.json with id 1 is created
2023-05-18T09:28:21.7942350Z 17:28:21.793 INFO: Creating TypeScript program (done) | time=7953ms
2023-05-18T09:28:21.7945058Z 17:28:21.793 INFO: Starting analysis with current program
2023-05-18T09:28:21.7951383Z 17:28:21.793 DEBUG: File not part of the project: 'D:/cd077a/_w/1/s/.scannerwork/.sonartmp/eslint-bridge-bundle/package/node_modules/typescript/lib/lib.es5.d.ts'
2023-05-18T09:28:21.7953240Z 17:28:21.793 DEBUG: File not part of the project: 'D:/cd077a/_w/1/s/.scannerwork/.sonartmp/eslint-bridge-bundle/package/node_modules/typescript/lib/lib.es2015.d.ts'
2023-05-18T09:28:21.7954159Z 17:28:21.793 DEBUG: File not part of the project: 'D:/cd077a/_w/1/s/.scannerwork/.sonartmp/eslint-bridge-bundle/package/node_modules/typescript/lib/lib.es2016.d.ts'
...

Colin · May 22, 2023, 8:06am

Hey there.

d.ts files should be excluded by default, along with the node_modules folder.

Are these files actually appearing in the Code tab of your project, or just referenced in the DEBUG logs?

Yinan_Wang · May 30, 2023, 2:25am

Just Debug log, but scanner repeats reporting those debug information for each tsconfig.json in the project. And my project is a monorepo which has tens of tsconfig

Colin · May 30, 2023, 9:41am

If all your files are being analyzed and analysis is performing well, you should be able to safely ignore these DEBUG messages. I’ll flag this for some experts anyways to see if they have any opinions.

Yinan_Wang · May 31, 2023, 6:27am

Thank you for your response. The performance is bad because of the complicated analysis. I have to use ‘tsconfigPath’ to assign a tsconfig file for all sub libraries of the project

victor.diez · June 6, 2023, 7:55am

Hi @Yinan_Wang,

indeed you can safely ignore those debug logs. Typescript will resolve those dependencies for type-checking, but analysis will not be performed, that’s why these lines are shown.

You could also try having a single tsconfig.json in the root of your monorepo which includes all subdirectories, and only use that one for the tsconfigPath sonar property. Depending of the size of the whole monorepo it may cause memory issues (you may try increasing the node memory -Dsonar.javascript.node.maxspace=8192, but performance should improve as it will only create one TS Program.

Cheers,
Victor

Yinan_Wang · June 7, 2023, 2:14am

Thank you very much, Victor! I have assigned a tsconfig file as you suggest. It does work . But you know it’s very common that there are different types of sub projects in monorepo and they may need different tsconfigs. Do you have solution for it instead of using only one config file?

victor.diez · June 7, 2023, 7:58am

Hi @Yinan_Wang,

I would advise creating the minimum tsconfigs needed, making each the common denominator configuration for the included files on each tsconfig file. And making sure they exclude directories that do not use their compilerOptions, to avoid having files included in more than one program.

In the end, program creation for big projects takes a lot of time and the analyzer cannot avoid spending the time creating them, so any help/hint provided will help with analysis times. Creating fewer programs containing more files and properly setting files, inclusions, and exclusions will greatly help the performance. Depending on the project size, memory will then be the main issue instead.

The next version of the JS/TS analyzer uses TypeScript 5, which in principle improves performance with Program creation. Hopefully, you will notice the difference after you upgrade.

Cheers,
Victor

victor.diez · June 7, 2023, 8:53am

I would just like to add that many compilerOptions do not affect the analysis results, so it should be easier to merge tsconfigs when removing those. Any option which affects the JS output is not needed (outDir, target…).

However, options affecting module detection are indeed important to properly create the program and type-check against the right dependencies (module, moduleResolution, moduleDetection, paths, lib, baseUrl…). So, if all subfolders share the same values in all of these, you may only need to use one tsconfig.

Cheers,
Victor

Yinan_Wang · June 8, 2023, 2:02am

I see. Thank you again. Your answer does help me a lot.