Suddenly analysis takes a long time to finish


(Tobias Hein) #1

Hello,

I’m having this weird issue and I can’t figure out how to fix it. We switched from a self-hosted SonarQube instance to SonarCloud three weeks ago. Yesterday I noticed a sudden increase in the runtime of our builds. Runtime jumped from around 40 minutes to over 100 minutes.

Our SonarCloud project consists of 200K lines of C# code. There was neither a big change in our codebase nor any changes in the configuration of our SonarCloud project.

Having a look at our log files from our build server I can spot a small difference.

Before:

INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=63ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: Skipping CPD calculation for short living branch and pull request
INFO: Analysis report generated in 140ms, dir size=115 KB
INFO: Analysis reports compressed in 16ms, zip size=22 KB
INFO: Analysis report uploaded in 141ms

After:

INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=110ms
INFO: Sensor CPD Block Indexer
INFO: Sensor CPD Block Indexer (done) | time=0ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Analyzing 0 ucfgs to detect vulnerabilities.
INFO: Sensor JavaSecuritySensor [security] (done) | time=31ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Analyzing 12435 ucfgs to detect vulnerabilities.
INFO: Sensor CSharpSecuritySensor [security] (done) | time=4517314ms
INFO: Skipping CPD calculation for short living branch and pull request
INFO: Analysis report generated in 203ms, dir size=130 KB
INFO: Analysis reports compressed in 31ms, zip size=28 KB
INFO: Analysis report uploaded in 313ms

Where are those *SecuritySensor messages coming from and what are ‘ucfgs’? Is there a way to disable whatever is causing this long running analysis step?

Any help would be appreciated.

Best regards,
Tobias


Java project scan hangs on "Analyzing XXX ucfgs to detect vulnerabilities"
(Nicolas Peru) #5

Hi,
Thanks for reporting this issue.

Where are those *SecuritySensor messages coming from and what are ‘ucfgs’? Is there a way to disable whatever is causing this long running analysis step?

Those security sensor are coming from the new taint analysis engine for security rules. ucfgs means Universal Control Flow GraphS which are the input of the taint analysis engine.

For some reason the taint analysis is taking a long time : is your project public so we could investigate what is going on our side ? if that is the case could you provide a link to its sources ?

Unfortunately you don’t really have a way to disable this besides deactivating vulnerabilities rules.

All in all : this is a serious matter for us and thanks for reporting it but we lack a bit of information to take further action and improve things.

Thanks for your help.


(Tobias Hein) #6

Hello,

unfortunately our project is private. I had a closer look at the documentation last night and came to the same conclusion: somehow the issue is related to the vulnerabilities rules.

I’ve created a new C# ruleset and deactivated all vulnerabilities rules but the following:

  • Fields should not have public accessibility
  • Generic exceptions should not be ignored
  • Mutable fields should not be “public static”

Runtime of the analysis step is down to 3 minutes again. I’ll reactivate the remaining rules in small sets and see if I can identify the ones that are causing this problem.


(Michael Gumowski) #7

Hey Tobias,

No need to spend too much time on finding the related rules. There is currently only 6 vulnerability rules which are now relying on the Taint Analysis engine:

As soon as one of these 6 rules is enabled, the UCFGs will be generated and the Taint Analysis engine triggered during analysis. So simply disabling these rules should restore the usual analysis time.

Note that is is major concern for us, and we are currently investigating this performance issue.

Regards,
Michael


(Nicolas Peru) #8

Hey Tobias,

In order for us to investigate this properly : would it be possible for you to get your hand on the ucfg files (which can be found in the root dir of your project : rootdir/.sonarqube/out/ucfg_cs
If you would be able to zip all those .pb file and make them available to us (privately) that would be really helpful.

How are you running your analysis ? (which CI tool are you using ?)

Those files only contains method calls and not your code.


(Tobias Hein) #9

Thanks for the info. I’ll keep those rules disabled for now.

This might be of help. I’ve run an analysis last night in verbose mode. This is an excerpt of the log

2018-06-19T20:08:40.5899956Z 22:08:40.575 INFO: Sensor CSharpSecuritySensor [security]
2018-06-19T20:08:42.0625453Z 22:08:42.058 INFO: Analyzing 12435 ucfgs to detect vulnerabilities.
2018-06-19T20:08:42.0690268Z 22:08:42.058 DEBUG: loaded 30 sinks for rule S3649
2018-06-19T20:08:42.0690895Z 22:08:42.058 DEBUG: Running rule csharpsquid:S3649
2018-06-19T20:08:42.2537710Z 22:08:42.246 DEBUG: Matching passthrough : string.Format(string, object, object)
2018-06-19T20:08:42.2615593Z 22:08:42.246 DEBUG: Matching passthrough : __concat
2018-06-19T20:08:42.2621249Z 22:08:42.246 DEBUG: Matching passthrough : __concat
2018-06-19T20:08:42.2627514Z 22:08:42.262 DEBUG: Matching passthrough : string.Format(string, object, object)
2018-06-19T20:08:42.2628310Z 22:08:42.262 DEBUG: Matching passthrough : string.Format(string, object, object)
2018-06-19T20:08:42.2640404Z 22:08:42.262 DEBUG: Matching passthrough : __concat
2018-06-19T20:08:42.2641499Z 22:08:42.262 DEBUG: Matching passthrough : __concat
2018-06-19T20:08:42.2644687Z 22:08:42.262 DEBUG: Matching passthrough : string.Format(string, object, object)
.
.
.
2018-06-19T20:42:20.0506761Z 22:42:20.040 INFO: Sensor CSharpSecuritySensor [security] (done) | time=2019465ms

This step took almost 35 minutes and produced 12.000 log entries (see attachment).

log.txt (1.0 MB)

Best regards,
Tobias


(Tobias Hein) #10

Sure, I can provide those files. Do I need to run an analysis with all rules enabled again first?

We are using Visual Studio Team Services and our build definition integrates the SonarCloud task from the Visual Studio Marketplace. Our solution consists of ~40 C# projects (205k LoC), the SonarCloud project is configured to just scan the .cs files (Source File Inclusions: **/*.cs).

Best regards,
Tobias


(Nicolas Peru) #11

Yes, you would need to activate the rules.
The mechanism is that if there is one of those rule activated, ucfgs are generated and then picked up by the security sensor if present.

Thank you very much for your help.


(Tobias Hein) #12

I’ve collected the files from the last run. To which email address can I send the zip file?


(Nicolas Bontoux) #13

Can you tell us which size it is ? If below 3MB we can arrange something on this very forum (provided you’re okay with sharing publicly, not sure if it’s a public project or not)


(Tobias Hein) #16

It’s a private project. File size is only 5MB. I’ve uploaded it to my OneDrive, I can send you the download link via e-mail.


(Nicolas Peru) #17

I’ve sent you my email privately, please use it. Thanks a lot again,


(Nicolas Peru) #18

Hi Tobias,

Just to keep you updated : we investigated what you sent us. We found out two areas of improvement :

  • one is that we are doing a lot of computation of hashes that can be cached (this brought roughly a 2x improvement)

  • The second is that this analysis technique requires to have a starting point. Our algorithm to determine those starting point was, let’s say, pretty naive (let’s start from all methods). We change this to limit the number of method to start with and it seems to give some very good results (we reduce the analysis time to ~3seconds on my machine).

We still have work to do to make this production ready and avalaible on SonarCloud but hopefully we are talking about a few days.

Thanks again for your help.


(Tobias Hein) #19

Hello Nicolas,

that sounds promising. Thanks for the status update.


(Dinesh Bolkensteyn) #20

Hello Tobias,

We have just deployed a fix on SonarCloud. Could you please relaunch an analysis on your side and confirm that the performance problem has been fixed?

Thank you again

Dinesh


(Tobias Hein) #21

Hello Dinesh,

everything’s back to normal. I reactivated the rules and ran builds on two of our branches. The runtime for the analysis step is down to 90 seconds again.

Best regards,
Tobias