What do "Load projects for organization" ? how to speed up this step on a CI?

SonarCloud
1

I was creating a small training application and I included some best practice in it and especially a sonar scan.

My application is a really small maven project, but it lives in a big github organization.

When I launch the scan on my CI (every time, the environment is cleaned) there is one step that take a lot of time:

[INFO] Load projects for organization 'xxxxxx'
[INFO] Load projects for organization 'xxxxxx' (done) | time=60009ms

What does this step ? is there a way to cache something to speed this up ?

Thank you

3

This step makes a web service call to SonarCloud to download the list of bound projects for the organization. I’m not aware of performance problems with this web service. It should normally not be longer than a second. During an analysis, the scanner makes multiple web service calls, many much slower than this, on average.

Are you sure that the time in your sample log snippet is typical? Are you getting such slow response times consistently? Would it be possible to include a larger snippet from your logs, so we can get a better idea of the relative speed of the various web service calls.

Is your application / organization public? If not, we can start a private thread to dig deeper.

5

I can confirm that this time is typical. Note it is run on a CI with temporary workspace, so it is like the build run on an completely empty new server every time.
I put an example of complete sonar step below. It come from a build of a really small library.

Why do the scan need the list of project for the organization ? it seems completly useless

Now it is a private organization with a pretty large number of repository yes.

And yes we can start a private thread if needed

Thank you

[INFO] --- sonar-maven-plugin:3.7.0.1746:sonar (default-cli) @ api ---
[INFO] User cache: /home/ci/.sonar/cache
[INFO] SonarQube version: 8.0.0
[INFO] Default locale: "en", source code encoding: "UTF-8"
[INFO] Load global settings
[INFO] Load global settings (done) | time=312ms
[INFO] Server id: 1BD809FA-AWHW8ct9-T_TB3XqouNu
[INFO] User cache: /home/ci/.sonar/cache
[INFO] Load/download plugins
[INFO] Load plugins index
[INFO] Load plugins index (done) | time=71ms
[INFO] Load/download plugins (done) | time=245ms
[INFO] Loaded core extensions: developer-scanner
[INFO] Process project properties
[INFO] Execute project builders
[INFO] Execute project builders (done) | time=4ms
[INFO] Project key: my-appli
[INFO] Base dir: /job
[INFO] Working dir: /job/target/sonar
[INFO] Load project settings for component key: 'my-appli'
[INFO] Load project settings for component key: 'my-appli' (done) | time=96ms
[INFO] Found an active CI vendor: 'Jenkins'
[INFO] Load project branches
[INFO] Load project branches (done) | time=35ms
[INFO] Load projects for organization 'xxxxxx'
[INFO] Load projects for organization 'xxxxxx' (done) | time=60008ms
[INFO] Load project pull requests
[INFO] Load project pull requests (done) | time=288ms
[INFO] Load branch configuration
[INFO] Load branch configuration (done) | time=2ms
[INFO] Load quality profiles
[INFO] Load quality profiles (done) | time=83ms
[INFO] Load active rules
[INFO] Load active rules (done) | time=1719ms
[INFO] Organization key: dktunited
[INFO] Indexing files...
[INFO] Project configuration:
[INFO] 11 files indexed
[INFO] 0 files ignored because of scm ignore settings
[INFO] Quality profile for java: Sonar way
[INFO] Quality profile for xml: Sonar way
[INFO] ------------- Run sensors on module api
[INFO] Load metrics repository
[INFO] Load metrics repository (done) | time=39ms
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by net.sf.cglib.core.ReflectUtils$1 (file:/home/ci/.sonar/cache/a89f1943fc75b65becd9fb4ecab8d913/sonar-tsql-plugin.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of net.sf.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Sensor JavaSquidSensor [java]
[INFO] Configured Java source version (sonar.java.source): 13
[INFO] JavaClasspath initialization
[INFO] JavaClasspath initialization (done) | time=13ms
[INFO] JavaTestClasspath initialization
[INFO] JavaTestClasspath initialization (done) | time=4ms
[INFO] Java Main Files AST scan
[INFO] 9 source files to be analyzed
[INFO] Load project repositories
[INFO] Load project repositories (done) | time=97ms
[INFO] 9/9 source files have been analyzed
[INFO] Java Main Files AST scan (done) | time=2652ms
[INFO] Java Test Files AST scan
[INFO] 1 source files to be analyzed
[INFO] 1/1 source files have been analyzed
[INFO] Java Test Files AST scan (done) | time=31ms
[INFO] Sensor JavaSquidSensor [java] (done) | time=2852ms
[INFO] Sensor SonarCSS Rules [cssfamily]
[INFO] No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
[INFO] Sensor SonarCSS Rules [cssfamily] (done) | time=1ms
[INFO] Sensor SurefireSensor [java]
[INFO] parsing [/job/target/surefire-reports]
[INFO] Sensor SurefireSensor [java] (done) | time=23ms
[INFO] Sensor JavaXmlSensor [java]
[INFO] 1 source files to be analyzed
[INFO] Sensor JavaXmlSensor [java] (done) | time=162ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor HTML [web]
[INFO] Sensor HTML [web] (done) | time=4ms
[INFO] Sensor XML Sensor [xml]
[INFO] 1 source files to be analyzed
[INFO] Sensor XML Sensor [xml] (done) | time=104ms
[INFO] 1/1 source files have been analyzed
[INFO] Sensor JaCoCo XML Report Importer [jacoco]
[INFO] Sensor JaCoCo XML Report Importer [jacoco] (done) | time=2ms
[INFO] Sensor JavaSecuritySensor [security]
[INFO] Reading type hierarchy from: /job/target/sonar/ucfg2/java
[INFO] Read 49 type definitions
[INFO] Reading UCFGs from: /job/target/sonar/ucfg2/java
[INFO] 07:57:29.34302 Building Type propagation graph
[INFO] 07:57:29.3492 Running Tarjan on 106 nodes
[INFO] 07:57:29.350712 Tarjan found 106 components
[INFO] 07:57:29.353131 Variable type analysis: done
[INFO] 07:57:29.35514 Building Type propagation graph
[INFO] 07:57:29.357526 Running Tarjan on 106 nodes
[INFO] 07:57:29.358076 Tarjan found 106 components
[INFO] 07:57:29.358631 Variable type analysis: done
[INFO] Analyzing 22 ucfgs to detect vulnerabilities.
[INFO] All rules entrypoints : 0 Retained UCFGs : 0
[INFO] rule: S5131, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S5131 done
[INFO] rule: S3649, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S3649 done
[INFO] rule: S2076, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S2076 done
[INFO] rule: S2091, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S2091 done
[INFO] rule: S2078, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S2078 done
[INFO] rule: S2631, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S2631 done
[INFO] rule: S5135, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S5135 done
[INFO] rule: S2083, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S2083 done
[INFO] rule: S5167, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S5167 done
[INFO] rule: S5144, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S5144 done
[INFO] rule: S5145, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S5145 done
[INFO] rule: S5146, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S5146 done
[INFO] rule: S5334, entrypoints: 0
[INFO] Visited 0 ucfgs in 0 ms, 0 steps
[INFO] rule: S5334 done
[INFO] Sensor JavaSecuritySensor [security] (done) | time=296ms
[INFO] Sensor CSharpSecuritySensor [security]
[INFO] Reading type hierarchy from: /job/target/ucfg_cs2
[INFO] Read 0 type definitions
[INFO] Reading UCFGs from: /job/target/ucfg_cs2
[INFO] No UCFGs have been included for analysis.
[INFO] Sensor CSharpSecuritySensor [security] (done) | time=1ms
[INFO] Sensor PhpSecuritySensor [security]
[INFO] Reading type hierarchy from: /job/target/sonar/ucfg2/php
[INFO] Read 0 type definitions
[INFO] Reading UCFGs from: /job/target/sonar/ucfg2/php
[INFO] No UCFGs have been included for analysis.
[INFO] Sensor PhpSecuritySensor [security] (done) | time=1ms
[INFO] Sensor PythonSecuritySensor [security]
[INFO] Reading type hierarchy from: /job/target/sonar/ucfg2/python
[INFO] Read 0 type definitions
[INFO] Reading UCFGs from: /job/target/sonar/ucfg2/python
[INFO] No UCFGs have been included for analysis.
[INFO] Sensor PythonSecuritySensor [security] (done) | time=1ms
[INFO] ------------- Run sensors on project
[INFO] Sensor Zero Coverage Sensor
[INFO] Sensor Zero Coverage Sensor (done) | time=10ms
[INFO] Sensor Java CPD Block Indexer
[INFO] Sensor Java CPD Block Indexer (done) | time=20ms
[INFO] SCM provider for this project is: git
[INFO] 1 files to be analyzed
[INFO] 1/1 files analyzed
[INFO] 5 files had no CPD blocks
[INFO] Calculating CPD for 4 files
[INFO] CPD calculation finished
[INFO] Analysis report generated in 59ms, dir size=183 KB
[INFO] Analysis report compressed in 31ms, zip size=53 KB
[INFO] Analysis report uploaded in 87ms
[INFO] ANALYSIS SUCCESSFUL, you can browse https://sonarcloud.io/dashboard?id=my-appli
[INFO] Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
[INFO] More about the report processing at https://sonarcloud.io/api/ce/task?id=abcdefghijklmn
[INFO] Analysis total time: 1:09.133 s
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:14 min
[INFO] Finished at: 2020-02-05T07:57:29Z
[INFO] ------------------------------------------------------------------------
8

Ok that’s really not normal. I created a ticket to track this: SCCOMM-18.

Thanks @GregoireW for reporting! I hope to improve this very soon!

1 Like
9

Thank you

Speaking of large organization, I will also point out the “analyze new project” process in sonarcloud.
You got a mandatory “Analyze projects - Select repositories” step which load all github repositories, so it can take time.

If it was possible to have either the repositories list or to enter a repository manually (then sonarcloud can check if it is valid for the current organization) it would be better.

10

Hi @GregoireW,

We’ve improved this a while ago and forgot to update this thread.

Can you please check if your scans got faster? The step will not be there anymore, we replaced it with a better solution.

11

Hello,

On the CI it is much better yes. Thank you.

Now I dream you can do something on the sonarcloud “new project” page

Thank you