Sonar Scanner is really slow scanning typescript

SonarQube
, ,
1

We hadn’t sonar in use for one month, so recently we updated sonarqube to version 8.1.
With that update we migrated our database to another server (before it was on the same server).

On the old configuration a scan took about 2 minutes.

Now with the new setup scanning takes a half an hour for the typescript files. In 10 seconds he scans 13-16 files.

I also tested it locally on my computer which doesn’t change anything. The server CPU isn’t even used with its full capacity.

The command for running the scanner is:
sonar-scanner.bat -Dsonar.test.inclusions=**/*.test.*,**/*.spec.*,**.int.* -Dsonar.projectKey=frontend -Dsonar.projectName=Frontend -Dsonar.host.url=https://our.domain.com/ -Dsonar.login=******** -Dsonar.language=ts,html,css

2

Hi,

Could you post debug logs (with -X)?

We recently migrated TypeScript analysis to a new frontend and it seems to be slow on huge projects. How many TS files and tsconfigs do you have?

Try to set sonar.javascript.node.maxspace property to 4096 or 8192

4

any update @Lena ?

5

Not yet, but I hope to investigate your problem this week. I will update this thread.

One thing I noticed is that you use windows-1252 encoding. Could you experiment with using utf-8?

6

How can i change the encoding in the sonarscanner? Our files should be utf-8.

7

any update @Lena?

8

Hello,

To set the encoding, you need to add this in your sonar-project.properties or on the sonar-scanner command line:

sonar.sourceEncoding=UTF-8

I suggest to disable the Duplication engine that is consuming by definition a lot of time and is not your priority:

sonar.cpd.exclusions=**

The configuration sonar.language is having no effect. The parameter was dropped with SonarQube 7.7.

So finally, I would:

  • use SonarQube 8.2 (the latest and greatest)
  • execute that command at the root of your TypeScript project:
sonar-scanner.bat \
-Dsonar.sourceEncoding=UTF-8 \
-Dsonar.cpd.exclusions=** \
-Dsonar.test.inclusions=**/*.test.*,**/*.spec.*,**.int.* \
-Dsonar.projectKey=frontend \
-Dsonar.projectName=Frontend \
-Dsonar.host.url=https://our.domain.com/  \
-Dsonar.login=********

Regards
Alex

1 Like
9

Hello,

thank you @Alexandre_Gigleux for your help. Sadly i have still the same issues even after i applied your suggestions.
I updated SonarQube to 8.2, i updated the sonar-scanner-cli to the latest version and i set sonar.cps.exclusions=** which the scanner is caring.

But i have still problems that the scanner is too slow (28 files in 10 seconds).

I hope to hear from you.

Regards
Daniel

10

could you share logs of analysis with encoding fixed to UTF-8?

11

I have a similar problem. We are using

  • SonarQube Enterprise 7.9.1
  • SonarJS 6.1.0
  • Sonarqube-scanner 2.6.0 with the newest sonar-scanner-cli
    The project contains more than 4900 typescript source files and over 20 tsconfig.json files.

Looking at the log, most of the steps (e.g. indexing files, matching files to tsconfig.json, etc) only take few seconds to finish. But when it comes to the analysis of the >4900 typescript files, it takes ~2 hours.
Do you have an idea where this long duration comes from?
My guess would be that it has to do with sending AnalysisRequest and/or getting AnalysisResponse in TypeScriptSensor.java but I might be wrong.

Thanks in advance for your help!

12

@To1
First, are your files in UTF-8?:slight_smile:

Our current guess is that there is a lot of memory used, so a lot of time is takes by allocating. But so far we didn’t get any reproducer in our hands so we can’t debug and improve that.

You would help us a lot if you could provide us sources of your project, privately of course.

13

i would like to provide you our project. But unforntunately our project (enterprise) is not to give it to someone… my boss forbid it explicitly.
Sorry.

Maybe i can reproduce it with another small project. But that needs a lot of time.

Maybe @To1 can hand over something?

14

I just tried scanning the angular project which results in 100 files/10 seconds. Our project has 25-30 files/10 seconds which is a lot less.

15

@Lena yes, they are in UTF-8 and we also set -Dsonar.sourceEncoding=UTF-8.

I’m really sorry, I’m in the same situation as Daniel. I cannot / must not provide the sources either.

@danieldaeschle which version of SonarQube did you use before updating to 8.1? Did it still use SonarTS or already SonarJS then?

16

@To1
We used 7.9 LTS Version before. If i can remember corrently, it was SonarTS.

18

@Alexandre_Gigleux @Lena

Here is a Repository with our Project structure in obfuscated.
The typescript files doesn’t contain real code.

In this state SonarScanner is slow (6 files per second). To speed it up: Remove the typings directory either through removing it in the tsconfig, removing the directory itself or copy all files from typings/eit to typings and remove the eit directory.
After that SonarScanner scans 30 files per second which is 5 times faster than before!
With out original project the scanner scans 2-3 files per second.

I hope it helps you to fix that bug.

2 Likes
19

any feedback?

20

Hey @danieldaeschle,

I didn’t have so far any time to investigate your reproducer, but that’s certainly on my todo.
I will ping you as soon as I have any info.
Thanks for the help anyway!

21

Update?

22

Hey @danieldaeschle ,

As I wrote you in private, I’m not able to reproduce slow analysis with your example project :frowning:

Your 2000 files take ~35sec for me, removing typings makes it just couple of seconds faster.