Kotlin's Sonar analysis taking too long

juanagustin_innocv · October 26, 2021, 4:55pm

Hi!

We have an Android aplication and we are analysing it with sonarcloud. This application has a 200k lines approximately and it is developed with java and kotlin language. For some time now, we have had a huge time increase when sonar analyses the application.

Right now this is a huge problem for use since we are running Sonar alongside Jacoco and detekt with every pull request and the time becomes unmanageable. All of our gradle builds are running on macOS images with a 3 core CPU, 14 GB of RAM, and 14 GB of SSD disk space.

I run this command to show log.

gradle clean build sonarqube --no-build-cache --info

And the analysis of one single module of the app takes around 2 hours.

cat sonar_build.log | grep compileKotlin | grep "Took" | sed 's/.* Took \(.*\) secs./\1/' | awk '{sum+=$1} END {print (sum/60) " min"}'
0.61785 min


cat sonar_build.log | grep "Kotlin Sensor" | grep "done" | sed 's/.* time=\(.*\)ms/\1/' | awk '{sum+=$1} END {print (sum/1000/60) " min"}'
124.991 min

I need help with this because fast feedback on PR’s is essential for us.

ALM used Azure DevOps

Use a Microsoft-hosted agent

CI system used Azure DevOps

Scanner command used gradle clean build sonarqube --no-build-cache --info

Languages of the repository Kotlin, Java

Margarita_Nedzelska · October 28, 2021, 8:26am

Hello @juanagustin_innocv,

Thanks a lot for your input! I have a few additional questions, so we can better identify your problem.
Since what date do you observe this degradation? Was it always like this or it has suddenly become so slow? Is it so slow for the whole project or only for some specific modules or files? (You can find this information in the analysis log)

Another question is could you please disable all the rules in your quality profile and try to analyze without the rules, so we can understand if this is related to the rules or the compiler?

It will also be extremely useful if you can share the full log with us.

Kind regards,
Margarita

juanagustin_innocv · October 29, 2021, 8:55am

Hello @Margarita_Nedzelska,

Before June 28, 2021, the sonar task took 6 minutes to complete, after this date, the task began to increase up to 50 minutes. On October 20, 2021, this task increased again to the current 126 minutes it takes.

I have disabled all rules on my kotlin and Java quality profile and it takes the same time to complete the sonar task.

This is a summary of my build log. I can’t share a full file because the project is private.

2021-10-28T05:34:16.2775520Z ------------- Run sensors on module app
2021-10-28T05:34:16.2776016Z Sensor JavaSensor [java]
2021-10-28T05:34:16.2776398Z Configured Java source version (sonar.java.source): 8
2021-10-28T05:34:16.2776795Z JavaClasspath initialization
2021-10-28T05:34:16.2777162Z JavaClasspath initialization (done) | time=584ms
2021-10-28T05:34:16.2777528Z JavaTestClasspath initialization
2021-10-28T05:34:16.2777905Z JavaTestClasspath initialization (done) | time=224ms
2021-10-28T05:34:16.2778576Z Java "Main" source files AST scan
2021-10-28T05:34:25.8760784Z 
2021-10-28T05:34:25.8762257Z 58/484 files analyzed, current file: .../ComparativeFragment.java
...
2021-10-28T05:35:06.9765518Z 484/484 source files have been analyzed
2021-10-28T05:35:06.9766482Z 27 source files to be analyzed
2021-10-28T05:35:06.9766846Z 
2021-10-28T05:35:06.9767240Z > Task :sonarqube
2021-10-28T05:35:06.9767699Z Slowest analyzed files:
2021-10-28T05:35:06.9768249Z     .../Formatter.java (1980ms, 31250B)
2021-10-28T05:35:06.9768886Z     .../ProfitabilityFragment.java (1676ms, 10159B)
2021-10-28T05:35:06.9769740Z     .../WebViewActivity.java (1333ms, 16417B)
2021-10-28T05:35:06.9776860Z Java "Main" source files AST scan (done) | time=51183ms
2021-10-28T05:35:06.9777249Z Java "Test" source files AST scan
2021-10-28T05:35:09.5761038Z 
2021-10-28T05:35:09.5762348Z 27/27 source files have been analyzed
2021-10-28T05:35:09.5762756Z 
2021-10-28T05:35:09.5763224Z > Task :sonarqube
2021-10-28T05:35:09.5764345Z Java "Test" source files AST scan (done) | time=2604ms
2021-10-28T05:35:09.5765440Z No "Generated" source files to scan.
2021-10-28T05:35:09.5765946Z Sensor JavaSensor [java] (done) | time=54607ms
2021-10-28T05:35:09.5766416Z Sensor CSS Rules [cssfamily]
2021-10-28T05:35:09.6762487Z No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
2021-10-28T05:35:09.6764309Z Sensor CSS Rules [cssfamily] (done) | time=18ms
2021-10-28T05:35:09.6765048Z Sensor C# Project Type Information [csharp]
2021-10-28T05:35:09.6765570Z Sensor C# Project Type Information [csharp] (done) | time=2ms
2021-10-28T05:35:09.6766078Z Sensor C# Analysis Log [csharp]
2021-10-28T05:35:09.6766597Z Sensor C# Analysis Log [csharp] (done) | time=0ms
2021-10-28T05:35:09.6767057Z Sensor C# Properties [csharp]
2021-10-28T05:35:09.6767530Z Sensor C# Properties [csharp] (done) | time=0ms
2021-10-28T05:35:09.6768000Z Sensor SurefireSensor [java]
2021-10-28T05:35:09.6784703Z parsing [/home/vsts/work/1/s/BMI-droid/app/target/surefire-reports]
2021-10-28T05:35:09.6785238Z Sensor SurefireSensor [java] (done) | time=0ms
2021-10-28T05:35:09.6785543Z Sensor JavaXmlSensor [java]
2021-10-28T05:35:09.6785814Z Sensor JavaXmlSensor [java] (done) | time=53ms
2021-10-28T05:35:09.6786092Z Sensor HTML [web]
2021-10-28T05:35:09.6786355Z Sensor HTML [web] (done) | time=9ms
2021-10-28T05:35:09.6786997Z Sensor VB.NET Project Type Information [vbnet]
2021-10-28T05:35:09.6787307Z Sensor VB.NET Project Type Information [vbnet] (done) | time=2ms
2021-10-28T05:35:09.6787581Z Sensor VB.NET Analysis Log [vbnet]
2021-10-28T05:35:09.6787863Z Sensor VB.NET Analysis Log [vbnet] (done) | time=0ms
2021-10-28T05:35:09.6788147Z Sensor VB.NET Properties [vbnet]
2021-10-28T05:35:09.6788404Z Sensor VB.NET Properties [vbnet] (done) | time=0ms
2021-10-28T05:35:09.6788690Z Sensor JaCoCo XML Report Importer [jacoco]
2021-10-28T05:35:09.7766325Z Importing 1 report(s). Turn your logs in debug mode in order to see the exhaustive list.
2021-10-28T05:35:12.4763678Z Sensor JaCoCo XML Report Importer [jacoco] (done) | time=2810ms
2021-10-28T05:35:12.4765013Z Sensor Kotlin Sensor [kotlin]
2021-10-28T05:35:12.4765599Z 
2021-10-28T05:35:12.4766210Z 4515 source files to be analyzed
2021-10-28T05:35:22.5761599Z 38/4515 files analyzed, current file: .../CalendarZoomPresenterImpl.kt
...
2021-10-28T06:14:02.6762000Z 3177/4515 files analyzed, current file: .../ExpenseForecastRepository.kt
...
2021-10-28T07:03:42.9761856Z 4507/4515 files analyzed, current file: .../OperationalCardSimpleRepository.kt
2021-10-28T07:04:11.9760623Z 4515/4515 source files have been analyzed
2021-10-28T07:04:11.9762709Z 
2021-10-28T07:04:11.9781005Z > Task :sonarqube
...
2021-10-28T07:04:42.6761163Z Analysis total time: 1:31:42.108 s
2021-10-28T07:04:42.6762305Z :sonarqube (Thread[Daemon worker,5,main]) completed. Took 1 hrs 32 mins 13.349 secs.
2021-10-28T07:04:43.4762171Z

Margarita_Nedzelska · October 29, 2021, 1:09pm

Thanks for the details.

The initial increase probably happened when we released the new version of Kotlin Analyzer, that computes semantic. This makes the analysis more precise, but makes analysis longer. To check if the analysis is faster without semantic you can populate properties sonar.java.binaries sonar.java.libraries with empty line. This will disable the generation of semantic information so the analysis should be faster.
Note: we’re asking for it to investigate the issue, but it’s not the recommended way to perform the analysis, as the result might contain FPs and FNs.

For the second increase it sounds weird, but this could be related to the upgrade of Kotlin compiler to 1.5.31, that we’ve done recently I am not completely sure, so will get back to you, once we check it.

From your side, I would like you to double check the true compilation time for the project without any cache and incremental compilation.

So we can see what’s our lower bound. And the second thing would be to check if there’re specific files that slows down the whole analysis. These files usually appear in the log of analysis like this several times:

Timestamp 3177/4515 files analyzed, current file: .../SomeFile.kt
Timestamp 3177/4515 files analyzed, current file: .../SomeFile.kt
Timestamp 3177/4515 files analyzed, current file: .../SomeFile.kt
Timestamp 3177/4515 files analyzed, current file: .../SomeFile.kt

If you can see something like this in the log, it will be nice to share these files with us (can do it in a private conversation). By doing this we can identify, if there are any source code patterns that in some cases make compiler slow.

Best,
Margarita

Margarita_Nedzelska · November 18, 2021, 11:57am

@juanagustin_innocv There should be a new version of the Kotlin analyzer on SonarCloud since today. It contains a fix for some performance issues. Could you, please, check, if you see any performance improvements on your side?

Regards,
Margarita

juanagustin_innocv · November 19, 2021, 10:47am

What a great news!

In the latest pull requests, we have seen that the time has dropped from 94 minutes to 5.

Thanks a lot @Margarita_Nedzelska! It’s working like a charm now.

system · November 26, 2021, 1:44pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.