Restriction of the download of scanner plugins to those that are essentially needed

[Rebse (Gilbert Rebhan)]

Hi,
I have tried before, but IMO this needs to be pushed further and the real urgency has not been sufficiently appreciated - the forum has a bunch of related postings.

Restriction of the download of scanner plugins to those essentially needed

Hi, we’re running some special Jenkins builds for Android in Docker containers that are destroyed afterwards. This means every new build downloads all available scanner plugins from our Enterprise instance again, but in fact we would only need the Kotlin scanner beside the core plugins. Would be fine, if we could restrict the amount of downloaded plugins to only those that are really needed. Or is there already a solution for that !? Gilbert

We run a central Sonarqube instance that is accessed from multiple network areas in a very restrictive environment, which means there are a lot of proxies, web gateways and content security scans involved.

We have a LOT of Jenkins setups (one controller and several agents) for special purposes, i.e.
some Jenkins unit does only .Net builds, another only Android builds, another iOS builds … etc.
Also some special Jenkins units running in Docker containers that are destroyed afterwards.

I know there is a caching meachnism, but i just don’t get it why ALL plugins are downloaded to
EVERY scanner, i.e. also plugins like the monitoring Java Melody plugin

GitHub - javamelody/sonar-javamelody: JavaMelody plugin for Sonar

We’re hit regularly by socket timeouts at multiple Jenkins units because of downloading ALL plugins when only the Sonarqube core plugins and i.e. the Kotlin plugin is needed, as we’ve decided to always go with Sonarqube latest to be able to use the latest features and suppport of the latest syntax … we must also edit multiple docker files.

The scanner has access to the Sonarqube server settings, so if only java sources to be scanned, it should be sufficient to only download the Sonar core and Java plugin !?

I don’t know if this Jira ticket already goes in the right direction, as Jira insight got murky
since switching from prem to cloud, and the ticket has no details

https://sonarsource.atlassian.net/browse/SONAR-16633

The plugin api must provide a flag scanner download yes | no.
This is a design decision that has unfortunately been neglected so far.
At first it would already helpful to only download the core and language related plugins.

Gilbert

[Alexandre_Gigleux]

Hello @Rebse,

Thanks for the feedback.

In the context of improving the performance of the scanner and not spending time downloading plugins for nothing, we did attempt to make the Scanner clever enough to guess which plugins are required or not according to the files that will be analyzed. Unfortunately, we stopped that initiative because, from a performance perspective, it was not better or worst.

The scanner has access to the Sonarqube server settings, so if only java sources to be scanned, it should be sufficient to only download the Sonar core and Java plugin !?

The thing is that we can’t trust the previous analysis of a given project to determine which plugins are required. Imagine that you are playing with sonar.inclusions|exclusions parameters and you decide now to include more files, and more languages. Looking at SQ server settings won’t work. That will be the same if you were doing just Java in your project and then you decide to have in your repo some Python or TypeScript files.

The underlying need here is to be in control of what will be downloaded/executed on Scanner side and maybe one solution could be to give the freedom to project administrators to activate/deactivate analyzers/3rd party plugins at a project level.

What do you think about this idea? Would that work for you to go into your 100+ projects and customize the “plugins” activated?

Alex

[Rebse (Gilbert Rebhan)]

Hi Alex,

the first step would be to provide a label to mark a plugin as scanner-relevant yes | no.
This would already prevent e.g. monitoring or reporting plugins from being downloaded.

Is this the goal of [SONAR-16633] - Jira ’ Split plugins in scanner context’ ?

The decision which language plugins are needed for a specific analysis is of course more complex.
With “scanner has access to the Sonarqube server …” i mainly meant the language settings like i.e. sonar.java.file.suffixes on project and global level to be able to decide via indexed file extensions !?
But you’re right, the aspect of every new analysis might bring in changes in settings like inclusions / exclusions must be considered in addition.

AFAIK as i see from the logs, the plugins are downloaded before the analysis is started.
Maybe postpone the plugin download and start it only after some evaluation of the analysis parameters and the language relevant settings ?
Don’t know if this is sufficient together with the determined fileextensions though.
This may also help to run only the related language sensors !?

A project level setting for activate/deactivate analyzers/3rd party plugins is better than nothing.
That would be convenient with a project template (will create another feature request for this project template).

But i may also use the rest api in the pipeline to adjust this project settings,
so yes this idea would be helpful!

Have a nice weekend,
Gilbert

[daniel]

trying to correlate this idea with things i experienced in the past:

• i experienced failing analysis because a scanner(/plugin?) could not work because of $reason (exception/formatting/charset(?)/idontknowanymore)

would this envisioned serverside scanner-configurability also empower me to disable a problematic scanner/plugin when i experience such a hindrance?

i would like that

(sorry for convoluted english )

[Rebse (Gilbert Rebhan)]

Instead of a label / flag for the plugin api

@Rebse

the first step would be to provide a label to mark a plugin as scanner-relevant yes | no.
This would already prevent e.g. monitoring or reporting plugins from being downloaded.

a global setting beside activate/deactivate analyzers/3rd party plugins at a project level
would also be sufficient - if activate/deactivate means it should be used by the scanner.
When installing a monitoring plugin like Java Melody of course I want to use it.

[Rebse (Gilbert Rebhan)]

Hi Alex,

yesterday i’ve updated to Sonarqube Enterprise 9.7.1 - as we go with the latest version
to use the latest/greatest features - and the update itself went smooth, but it was
‘socket timeout day’ again

A lot of calls from the dev teams:
“It seems Sonarqube is down”
“Does Jenkins has any problems ?”
“The Sonar scan fails in our pipelines!”
[…]

My response is always “please replay until it works”.
Means every Sonarqube update is a PITA for us admins and the dev teams.
Any attempt to fix or improve this would be progress.
While i understand that you’ve already investigated this problem and it is complex,
preventing the download of plugins like Java Melody must be quite simple with some flag
or a project / global setting.

It’s almost Christmas and I wish it would be given higher priority.

Gilbert