Restriction of the download of scanner plugins to those that are essentially needed

[anon67236913]

Hi,
I have tried before, but IMO this needs to be pushed further and the real urgency has not been sufficiently appreciated - the forum has a bunch of related postings.

Restriction of the download of scanner plugins to those essentially needed

Hi, we’re running some special Jenkins builds for Android in Docker containers that are destroyed afterwards. This means every new build downloads all available scanner plugins from our Enterprise instance again, but in fact we would only need the Kotlin scanner beside the core plugins. Would be fine, if we could restrict the amount of downloaded plugins to only those that are really needed. Or is there already a solution for that !? Gilbert

We run a central Sonarqube instance that is accessed from multiple network areas in a very restrictive environment, which means there are a lot of proxies, web gateways and content security scans involved.

We have a LOT of Jenkins setups (one controller and several agents) for special purposes, i.e.
some Jenkins unit does only .Net builds, another only Android builds, another iOS builds … etc.
Also some special Jenkins units running in Docker containers that are destroyed afterwards.

I know there is a caching meachnism, but i just don’t get it why ALL plugins are downloaded to
EVERY scanner, i.e. also plugins like the monitoring Java Melody plugin

GitHub - javamelody/sonar-javamelody: JavaMelody plugin for Sonar

We’re hit regularly by socket timeouts at multiple Jenkins units because of downloading ALL plugins when only the Sonarqube core plugins and i.e. the Kotlin plugin is needed, as we’ve decided to always go with Sonarqube latest to be able to use the latest features and suppport of the latest syntax … we must also edit multiple docker files.

The scanner has access to the Sonarqube server settings, so if only java sources to be scanned, it should be sufficient to only download the Sonar core and Java plugin !?

I don’t know if this Jira ticket already goes in the right direction, as Jira insight got murky
since switching from prem to cloud, and the ticket has no details

https://sonarsource.atlassian.net/browse/SONAR-16633

The plugin api must provide a flag scanner download yes | no.
This is a design decision that has unfortunately been neglected so far.
At first it would already helpful to only download the core and language related plugins.

Gilbert

[Alexandre_Gigleux]

Hello @anon67236913,

Thanks for the feedback.

In the context of improving the performance of the scanner and not spending time downloading plugins for nothing, we did attempt to make the Scanner clever enough to guess which plugins are required or not according to the files that will be analyzed. Unfortunately, we stopped that initiative because, from a performance perspective, it was not better or worst.

The scanner has access to the Sonarqube server settings, so if only java sources to be scanned, it should be sufficient to only download the Sonar core and Java plugin !?

The thing is that we can’t trust the previous analysis of a given project to determine which plugins are required. Imagine that you are playing with sonar.inclusions|exclusions parameters and you decide now to include more files, and more languages. Looking at SQ server settings won’t work. That will be the same if you were doing just Java in your project and then you decide to have in your repo some Python or TypeScript files.

The underlying need here is to be in control of what will be downloaded/executed on Scanner side and maybe one solution could be to give the freedom to project administrators to activate/deactivate analyzers/3rd party plugins at a project level.

What do you think about this idea? Would that work for you to go into your 100+ projects and customize the “plugins” activated?

Alex

[anon67236913]

Hi Alex,

the first step would be to provide a label to mark a plugin as scanner-relevant yes | no.
This would already prevent e.g. monitoring or reporting plugins from being downloaded.

Is this the goal of [SONAR-16633] - Jira ’ Split plugins in scanner context’ ?

The decision which language plugins are needed for a specific analysis is of course more complex.
With “scanner has access to the Sonarqube server …” i mainly meant the language settings like i.e. sonar.java.file.suffixes on project and global level to be able to decide via indexed file extensions !?
But you’re right, the aspect of every new analysis might bring in changes in settings like inclusions / exclusions must be considered in addition.

AFAIK as i see from the logs, the plugins are downloaded before the analysis is started.
Maybe postpone the plugin download and start it only after some evaluation of the analysis parameters and the language relevant settings ?
Don’t know if this is sufficient together with the determined fileextensions though.
This may also help to run only the related language sensors !?

A project level setting for activate/deactivate analyzers/3rd party plugins is better than nothing.
That would be convenient with a project template (will create another feature request for this project template).

But i may also use the rest api in the pipeline to adjust this project settings,
so yes this idea would be helpful!

Have a nice weekend,
Gilbert

[daniel]

trying to correlate this idea with things i experienced in the past:

• i experienced failing analysis because a scanner(/plugin?) could not work because of $reason (exception/formatting/charset(?)/idontknowanymore)

would this envisioned serverside scanner-configurability also empower me to disable a problematic scanner/plugin when i experience such a hindrance?

i would like that

(sorry for convoluted english )

[anon67236913]

Instead of a label / flag for the plugin api

@anon67236913

the first step would be to provide a label to mark a plugin as scanner-relevant yes | no.
This would already prevent e.g. monitoring or reporting plugins from being downloaded.

a global setting beside activate/deactivate analyzers/3rd party plugins at a project level
would also be sufficient - if activate/deactivate means it should be used by the scanner.
When installing a monitoring plugin like Java Melody of course I want to use it.

[anon67236913]

Hi Alex,

yesterday i’ve updated to Sonarqube Enterprise 9.7.1 - as we go with the latest version
to use the latest/greatest features - and the update itself went smooth, but it was
‘socket timeout day’ again

A lot of calls from the dev teams:
“It seems Sonarqube is down”
“Does Jenkins has any problems ?”
“The Sonar scan fails in our pipelines!”
[…]

My response is always “please replay until it works”.
Means every Sonarqube update is a PITA for us admins and the dev teams.
Any attempt to fix or improve this would be progress.
While i understand that you’ve already investigated this problem and it is complex,
preventing the download of plugins like Java Melody must be quite simple with some flag
or a project / global setting.

It’s almost Christmas and I wish it would be given higher priority.

Gilbert

[anon67236913]

Hi @Alexandre_Gigleux ,

@Alexandre_Gigleux

In the context of improving the performance of the scanner and not spending time downloading plugins for nothing, we did attempt to make the Scanner clever enough to guess which plugins are required or not according to the files that will be analyzed. Unfortunately, we stopped that initiative because, from a performance perspective, it was not better or worst.

6 months later and still nothing has been done to improve this really annoying problem in any way
I wanted to update a reporting plugin today, but again problems with socket timeouts.
Finally i couldn’t update this plugin because of that.

Introducing some boolean that every plugin developer must implement, i.e.
isScannerPlugin with default = false would already be sufficient IMO.
There is a similar feature for custom rules to make them available in Sonarlint too,
one has to use <sonarLintSupported>true</sonarLintSupported> in pom.xml

I’m ok with downloading all scanner plugins, but i’ll never use reporting or monitoring plugins with the scanner.

PLEASE do something about this ASAP, introducing a boolean and take it into consideration for plugin download by scanner - shouldn’t be that complicated !?
Would already be possible in one of the next Sonarqube 10.x versions combined with a new scanner version.

Gilbert

[anon67236913]

Hi,

trying to push that again.
That’s my greatest pain point right now, in fact it’s a PITA.
Is this problem being actively pursued or am I on my own?
Do I really need to implement some weird workarounds by manually adding the jars to the
scanner cache ?!

I’m really disappointed that there is no further feedback for this problem

Gilbert

np Kingfish - Hard times

Christone "Kingfish" Ingram - Hard Times (acoustic) - (10/10/19) Vancouver, Fox Cabaret

[daniel]

Hi, i do not want to take anything away or derail from Gilberts postings. I also see his points as valid reasoning.

Concerning your request for feedback in the end of your posting:

@Alexandre_Gigleux

The underlying need here is to be in control of what will be downloaded/executed on Scanner side and maybe one solution could be to give the freedom to project administrators to activate/deactivate analyzers/3rd party plugins at a project level.

yes. this and what gilbert mentioned: if a plugin is not relevant to the scanner side, enable filtering for these and do not download them (if i understood Gilberts words correctly that would reduce download, too)

@Alexandre_Gigleux

What do you think about this idea? Would that work for you to go into your 100+ projects and customize the “plugins” activated?

i think being able to restrict (blacklist?) what can be downloaded would work. yes.

If i understand the documented hierarchy of setting things correctly, one should also be able to apply a “more global customization than project level”, right?

from here

… i hope you do envision this being set via settings/properties and not to be set via UI clickyclacky, tho

cheers
Daniel

[pierreguillot]

Hello folks,

I think the need is clarified enough and makes sense. As looking for a specific solution (using global/project/scanner settings, or flag at plugin level, or another solution), I think it can be handled by the SonarQube team to decide what makes sense.

I created SONAR-19996 to track this

[daniel]

Hello Pierre, always good to get prompt feedback. Thank you for that.

I skimmed through the jira ticket and would like to offer you one more scenario that would benefit from bespoken configurability of plugin usage:

• Say there is something which makes a used plugin behave erratic (say, a bug in UTF-8 parsing ).
  • Reproducibly erratic
  • 100s of jobs failing erratic

So the goal of this is probably more than impact on performance. It [c|w]ould also increase resilience.

Being able to firefight said scenario globally as an instance admin might be beneficial … but actually, while writing all this, i am pondering about “something, something, quality gate”

So: Implementationwise: good luck! If you like, keep us updated about the reasoning behind your (e.g. the teams) chosen implementation

[dw-om (David Wong)]

Just to add my 2 cents. It seems alot of unsuccessful analysis has gone into trying to automatically detect what plugins need to be downloaded. IMO in the short term the simplest step is to provide an option to allow development teams to override the plugins to download manually as we are in the best position to know what language our application is written in.

In the long term Sonar can work on the automatic plugin detect feature.

[Alexandre_Gigleux]

We are deploying a solution to this problem on SonarCloud these days. So far so good if you exclude the fact that all C++ analysis failed at some point but mistakes happen.
Once the solution is fully deployed on SonarCloud, we will think about applying the same principles to SonarQube.

[andird-gdncomm (Andi)]

Hi,

is this already deployed on sonarcloud yet ?

[Alexandre_Gigleux]

Hello,

I confirmed it’s already deployed in SonarCloud for a couple of months.
BTW, we are working to port the behavior to SonarQube 10.4

Regards
Alex

[daniel]

@Alexandre_Gigleux

we are working to port the behavior

could you maybe already link to an URL that describes “the behaviour”? (or is such documentation part of “the porting” ;))

[Alexandre_Gigleux]

Hello,

This feature will come with SonarQube 10.4.

More details here: The SonarScanners download only required 3rd-party plugins

Alex