Remove analysis to reduce LOC

[Przemek_Sech]

Must-share information (formatted with Markdown):

• which versions are you using
  • Enterprise Edition v9.9.6 (build 92038)
• how is SonarQube deployed:
  • Docker
• what are you trying to achieve
• what have you tried so far to achieve this

Dear Support,

We ran into a problem after upgrading the dotnet-sonarscanner to the latest v8.
It introduces fantastic new features. However, after we unconsciously upgraded, we realised we were not ready. (dotnet tool install -g dotnet-sonarscanner with no --version )

It started to scan XML in our dependencies and consumed the license.

We have downgraded to 7.1.1, and now we need to clean up the analysis consuming the license.

The LOCs that are counted are those found on the largest branch of the project.
– Calculating lines of code

I have read this thread How to delete the latest activity from SonarQube project?. I understand I need to run a fresh analysis to remove the older one.
However, we run sonar on branches and pull requests, and cherry-picking seems tedious.

What would be the most efficient way to remove analysis consuming our license?
It can be manual UI steps or API.

Thank you,
Przemek

UPDATE
I ran {{baseUrl}}/api/ce/activity?ps=1000, and it looks like I have 475 analyses to review. There must be a better way.

[ganncamp (G Ann Campbell)]

Hi Przemek,

I’m sorry you’re having to deal with this.

Unfortunately, this is not a scenario we anticipated, so we’ve never built anything to handle it.

The easiest thing to do, I think, is to look at the CI side, and simply re-trigger everything that ran since the upgrade. This has a few benefits:

• removes the tedious work of manually examining and deleting
• drops the unintended LOC from each branch and PR
• leaves the big analyses in your project/branch histories, so that you have an idea of what you’re facing LOC-wise if you decide to selectively re-enable multi-language analysis

 
HTH,
Ann

[Przemek_Sech]

Thank you, @ganncamp, for getting back to me.

It is resolved now, and I figured out a similar approach.

Here’s the story if anyone faces a similar issue.

1. I have downgraded the dotnet-sonarscanner to 7.1.1 to restore desired functionality.
2. I ran the {{baseUrl}}/api/ce/activity?ps=1000 endpoint to list all the analyses and filter ones matching the timeframe.
3. Since the latest analysis can’t be removed, I had to run CI for all the impacted branches.
   • I noticed that running analysis with lower LOC does not free the license. It is expected and pointed out in the documentation. To my best understanding, the options were removing analyses or whole branches.
4. I reviewed all the analyses we ran since dotnet-sonarscanner upgrade, re-ran CI for these and removed the impacted scans.

The Activity > Custom metric > Lines of code was very useful for finding aberrations.

Lesson learned: Never upgrade the major version automatically and without testing, as it’s expected for the major version to bring significant changes.
The dotnet tool install -g dotnet-sonarscanner --version 7.* is The Way.

Cheers,
Przemek

[ganncamp (G Ann Campbell)]

Hi Przemek,

@Przemek_Sech

I noticed that running analysis with lower LOC does not free the license

Ehm… No. But license LOC are counted from the largest branch (or PR) for each project. So you would need to reanalyzed each inflated branch of a project to make the reduction fully kick in.

@Przemek_Sech

It is expected and pointed out in the documentation.

Could you point me to that, please?

@Przemek_Sech

The Activity > Custom metric > Lines of code was very useful for finding aberrations.

There’s also the api/projects/license_usage web service, which will tell you which branch is being counted toward for each project. Sorry I didn’t think to mention it before.

@Przemek_Sech

The dotnet tool install -g dotnet-sonarscanner --version 7.* is The Way.

Or… you could use the opt-out
/d:sonar.scanner.scanAll=false in the begin step.

 
Ann

[denis.troller]

Hi @Przemek_Sech,

Thank you for your fantastic feedback and follow-through on this issue that will most probably help other users. I am truly sorry this caused such pain. We are actively tracking such issues and fixing them as we discover them.

On our side, lesson learned as well, such a major change will be announced here in advance next time so you all have a heads-up on what is coming and when.

May I ask what files exactly were analyzed and where in the repository they were located? This will help us figure out how to ignore them if they do not make sense for an analysis.

Denis

[Przemek_Sech]

Hi Ann and Denis,

@ganncamp

Documentation

Could you point me to that [place in the document], please?

I assumed that was the sentence

The LOCs that are counted are those found on the largest branch of the project.

but now I read the following one

the LOC will be counted from the most recent analysis of the project with the largest branch or PR.

which clarifies that the latest analysis is considered. Meaning I did not have to delete impacted analyses.
However, (and it’s only my impression) the license was not released after scanning the project again with a downgraded scanner. The license manager updated LOCs after removing analyses. It is possible it was due to some async background tasks running. (Long live eventual consistency )

License usage

Thank you for sharing api/projects/license_usage web service. It is fantastic.

dotnet-sonarscanner

Or… you could use the opt-out

It is an option . However, I learned about the Nuget RSS feed NuGet Gallery Feed for dotnet-sonarscanner, which will notify me about changes so that I can test in isolation and release with confidence.

I have seen the. /d:sonar.scanner.scanAll=false in the. Thank you for reminding me.

I like the new capabilities added to version 8. We need to exclude dependencies better.

@denis.troller

Thank you for your kind words and willingness to protest customers from themselves .

Since I have remover impacted analysis, I must rely on my memory. Apology if I’m not very helpful. I’ll be testing v8 soon (in a few weeks), so I may update this thread.

We have a custom setup where we put packages in .globalpackages/. We do not remember why it’s this way. It may be inherited from when we used TeamCity.

The files added to the scan were mainly XML. Some were ~1.2 million lines.

Many teams added the .globalpackages/ to .gitignore, resolving the problem. However, we are looking for a more centralised solution since we use a custom Docker image.

[Przemek_Sech]

I missed cleaning up one project, so I can confirm the following.

1. Rerunning the pipeline reduces license consumption without removing previous analyses.
2. Additional languages scanned
   
   
   
   
   

[denis.troller]

Hi @Przemek_Sech

Indeed, the Scanner honors .gitignore settings, which is the best solution for such a case, as this directory really needs to be ignored by git anyway.

Also, we are working on a new version that will correct issues with using sonar.exclusions to exclude some files from the analysis. This version should come out very soon.

Denis

[Przemek_Sech]

For anyone facing a similar issue, here’s the solution

Release 8.0.3 · SonarSource/sonar-scanner-msbuild

• Multi-Language analysis: exclude XML files from the analysis.

Thank you!