Azure DevOps Exclusions Not Working

Good afternoon!

Some background. We have SonarCloud integrated into Azure DevOps (YAML Pipeline) with our PR builds. Here is a snippet of the SonarPrepare task:

- task: SonarCloudPrepare@1
  inputs:
    SonarCloud: 'redacted'
    organization: 'redacted'
    scannerMode: 'MSBuild'
    projectKey: 'redacted'
    projectName: 'redacted'
    extraProperties: |
      # Additional properties that will be passed to the scanner, 
      # Put one key=value per line, example:
      # sonar.exclusions=**/*.bin
      sonar.exclusions=**/Migrations/**, **/*.html
      sonar.coverage.exclusions=**/Program.cs

As you can see why have a couple of file patterns that we want complete excluded. The problem is that the html exclusions appear to be ignored.

During the SonarCloudAnalyze Tasks I see the following in our logs:

INFO: Project configuration:
INFO: Excluded sources: **/build-wrapper-dump.json, **/Migrations/**, **/*.html

This leads me to believe that the configuration is being picked up correctly. However further down in the logs we get this:

INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=0ms
INFO: Sensor JavaScript inside HTML analysis [javascript]
INFO: 61 source files to be analyzed
##[error]ERROR: Failed to parse file [someredactedpath/html/somefilename.html] at line 610: Missing semicolon. (610:92)

This leads me to believe that the scanner is still scanning these files which we do not want or need scanned.

We have also tried setting the exclusions directly in the project settings in SonarCloud itself and the issue still persists. (Administration → General Settings → Analysis Scope → Source File Exclusions

I’m not sure if we just have something misconfigured or if this is a bug. Any help would be greatly appreciated!

Hello Nick,

I’ve tried to reproduce your issue without success so far. Can you confirm that your exclusions work in other cases, like for your **/Migrations/** regex?

Can you share additional logs? like the lines below

INFO: Excluded sources: **/build-wrapper-dump.json, **/Migrations/**, **/*.html

If exclusions work, you should have something like:

INFO: Project configuration:
INFO:   Excluded sources: **/Migrations/**, **/*.html
INFO: 1 file indexed
INFO: 1 file ignored because of inclusion/exclusion patterns

Also, what is the environment on which the scanner is run? Is it interpreting the exclusion patterns correctly?

Best,
Ilia

Thanks for the reply Ilia

I do see these lines as well:

INFO:   Base dir: D:\a\1\s
INFO:   Excluded sources: **/build-wrapper-dump.json, **/Migrations/**, **/*.html, **/*.sql
INFO:   Excluded sources for coverage: **/Program.cs
INFO: 5360 files indexed
INFO: 563 files ignored because of inclusion/exclusion patterns
INFO: 70 files ignored because of scm ignore settings

The scanner is running in Azure DevOps and this particular one is actually running on a windows-latest agent.

Do we have our pattern wrong? It looks like these files may not be being excluded because I see the following in our logs as well:

WARN: File 'D:\a\1\s\path\Migrations\filename.cs' referenced by the protobuf 'MetricsInfo' does not exist in the analysis context

However, I’d expect a lot more code analysis errors in these files if they actually are being analyzed.

Hey Nick,

Thank you for the additional information.

I would just like to check if the issue comes from the file selection or a sensor.

Could you share the whole logs from the scanner, adding the --debug option?

If not, Could you send me the lines around Sensor HTML [web], like:

09:36:09.858 INFO: Sensor HTML [web]
09:36:09.862 INFO: Sensor HTML [web] (done) | time=4ms

and the ones around JavaScript inside HTML analysis, like:

09:36:04.550 DEBUG: 'JavaScript inside HTML analysis' skipped because there is no related file in current project

Best,
Ilia

Ilia,

I had to take a snippet and just redact some of the keys that debug spits out but looks like it’s triggering on the Javascript inside HTML analysis but I guess my assumption is that this shouldn’t trigger if this filers are in the excluded path.

I attached a debug snippet below that I think has the information you are looking for. Please let me know if I can provide anything else.

sonardebug.txt (97.6 KB)

Hey Nick,

Thank you for the logs, could you check in the part of the logs that you’ve redacted if you find the following string:

Sensor HTML [web]

and print 10 lines before and after, if they don’t contain any secrets of course.

Best,
Ilia

No problem sorry I missed those previously!

2023-03-10T16:58:46.7095991Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.Designer.cs' was recognized as generated
2023-03-10T16:58:46.7098743Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.Designer.cs' was recognized as generated
2023-03-10T16:58:46.7101327Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.Designer.cs' was recognized as generated
2023-03-10T16:58:46.7103843Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.Designer.cs' was recognized as generated
2023-03-10T16:58:46.7127158Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.Designer.cs' was recognized as generated
2023-03-10T16:58:46.7129975Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.Designer.cs' was recognized as generated
2023-03-10T16:58:46.7132364Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.Designer.cs' was recognized as generated
2023-03-10T16:58:46.7134590Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.cs' was recognized as generated
2023-03-10T16:58:46.7136820Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.cs' was recognized as generated
2023-03-10T16:58:46.7139020Z 16:58:46.674 DEBUG: File 'D:\a\1\s\src\path\Migrations\file.cs' was recognized as generated
2023-03-10T16:58:46.7140797Z 16:58:46.674 INFO: Sensor C# Analysis Log [csharp] (done) | time=48ms
2023-03-10T16:58:46.7142079Z 16:58:46.674 INFO: Sensor C# Properties [csharp]
2023-03-10T16:58:46.7143882Z 16:58:46.674 DEBUG: Project 'redacted:B78C69FE-34A2-44DB-B13E-FBFC9D2B9D67': Analyzer working directory 'D:\a\1\.sonarqube\out\18\output-cs' contains 6 .pb file(s)
2023-03-10T16:58:46.7146232Z 16:58:46.674 DEBUG: Project 'redacted:B78C69FE-34A2-44DB-B13E-FBFC9D2B9D67': The Roslyn JSON report path has 'D:\a\1\.sonarqube\out\18\Issues.json'
2023-03-10T16:58:46.7148026Z 16:58:46.674 INFO: Sensor C# Properties [csharp] (done) | time=0ms
2023-03-10T16:58:46.7149319Z 16:58:46.674 INFO: Sensor HTML [web]
2023-03-10T16:58:46.7150574Z 16:58:46.674 INFO: Sensor HTML is restricted to changed files only
2023-03-10T16:58:46.7152190Z 16:58:46.674 INFO: Sensor HTML [web] (done) | time=0ms
2023-03-10T16:58:46.7153439Z 16:58:46.674 INFO: Sensor TextAndSecretsSensor [text]
2023-03-10T16:58:46.7154873Z 16:58:46.674 INFO: Sensor TextAndSecretsSensor is restricted to changed files only
2023-03-10T16:58:46.7156383Z 16:58:46.689 INFO: Sensor TextAndSecretsSensor [text] (done) | time=15ms
2023-03-10T16:58:46.7157755Z 16:58:46.689 INFO: Sensor VB.NET Project Type Information [vbnet]
2023-03-10T16:58:46.7159169Z 16:58:46.689 INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=0ms
2023-03-10T16:58:46.7160526Z 16:58:46.689 INFO: Sensor VB.NET Analysis Log [vbnet]
2023-03-10T16:58:46.7163886Z 16:58:46.689 DEBUG: Project 'redacted:B78C69FE-34A2-44DB-B13E-FBFC9D2B9D67': Property missing: 'sonar.vbnet.analyzer.projectOutPaths'. No protobuf files will be loaded for this project.
2023-03-10T16:58:46.7166843Z 16:58:46.689 INFO: Sensor VB.NET Analysis Log [vbnet] (done) | time=0ms
2023-03-10T16:58:46.7168357Z 16:58:46.689 INFO: Sensor VB.NET Properties [vbnet]
2023-03-10T16:58:46.7170905Z 16:58:46.689 DEBUG: Project 'redacted:B78C69FE-34A2-44DB-B13E-FBFC9D2B9D67': Property missing: 'sonar.vbnet.analyzer.projectOutPaths'. No protobuf files will be loaded for this project.
2023-03-10T16:58:46.7173806Z 16:58:46.689 DEBUG: Project 'redacted:B78C69FE-34A2-44DB-B13E-FBFC9D2B9D67': No Roslyn issues reports have been found.
2023-03-10T16:58:46.7175743Z 16:58:46.689 INFO: Sensor VB.NET Properties [vbnet] (done) | time=0ms

Could you check that the 563 files that are supposedly ignored by the scanner do indeed correspond to the files your project contains?
On a Unix, you can obtain this by running:

  • find ${base-dir} -type f -name "build-wrapper-dump.json" | wc -l
  • find ${base-dir}/**/Migrations/** -type f | wc -l
  • find ${base-dir} -type f -name "*.html" | wc -l
  • find ${base-dir} -type f -name "*.sql" | wc -l

Best,
Ilia

Hey Nick,

Could you also provide the scanner version that your CI is using. It should be one of the first lines of logs like:

INFO: Scanner configuration file: /Users/ilia/Dev/Sonar/scanner-4.7.0.2747-macosx/conf/sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarScanner 4.7.0.2747

Here ya go!

2023-03-10T16:56:48.2623307Z ##[section]Starting: SonarCloudAnalyze
2023-03-10T16:56:48.4476905Z ==============================================================================
2023-03-10T16:56:48.4477815Z Task         : Run Code Analysis
2023-03-10T16:56:48.4478012Z Description  : Run scanner and upload the results to the SonarCloud server.
2023-03-10T16:56:48.4478304Z Version      : 1.36.0
2023-03-10T16:56:48.4478486Z Author       : sonarsource
2023-03-10T16:56:48.4479354Z Help         : Version: 1.36.0. This task is not needed for Maven and Gradle projects since the scanner should be run as part of the build.

[More Information](https://docs.sonarcloud.io/advanced-setup/ci-based-analysis/sonarcloud-extension-for-azure-devops/)
2023-03-10T16:56:48.4480029Z ==============================================================================
2023-03-10T16:56:50.5665065Z [command]D:\a\_tasks\SonarCloudPrepare_14d9cde6-c1da-4d55-aa01-2965cd301255\1.33.0\classic-sonar-scanner-msbuild\SonarScanner.MSBuild.exe end
2023-03-10T16:56:50.6772814Z SonarScanner for MSBuild 5.11
2023-03-10T16:56:50.6775074Z Using the .NET Framework version of the Scanner for MSBuild

Here’s the output for the two that we really care about (Migrations and html) - I’m expecting the other two to not have anything for these projects:

find -path "**/Migrations/**" -type f | wc -l
567

find -path "**/*.html" -type f | wc -l
167

Thank you, Nick.

Could you also print the number of HTML files in Migration folders, so we can get a total of files that should be ignored by the scanner?

find -path "**/Migrations/**/*.html" -type f | wc -l

Best,
Ilia

Sure thing! It should report 0 and looks like it does!

find -path "**/Migrations/**/*.html" -type f | wc -l
0

Hello @NickBranstein ,

as a parallel investigation while my colleague @Ilia_Kebets checks the exclusion issue, is it possible for you to provide us (you can use private messages if needed) with the contents of the file that was failing?

We would like to know if there is any issue with our JS inside HTML parsing.

Thanks!
Victor

Hey Victor - I can’t send the full HTML without redacting a bunch of stuff even in a private message (sorry!) but I can tell send the part that was getting triggered for missing a semi-colon.

I’ll try and summarize the html a bit below. I’ve included the part that triggered it. Funny thing is that this html is just consumed by a unit test (which is why we want to ignore it from coverage in the first place).

<body>
<div>{a bunch more html that is most definitely malformed and missing a bunch of closing divs}</div>

  <!-- Visual Studio Browser Link -->
    <script type="application/json" id="__browserLink_initializationData">
        {"appName":"Chrome","requestId":"47c799b70a244a701e41cdd2e834ad42"}
    </script>
    <script type="text/javascript" src="http://localhost:1897/6d87de3fe224441b8e8b54c7245d9acc/browserLink" async="async"></script>
    <!-- End Browser Link -->
</body>

Hopefully this helps and I’m sorry if it doesn’t. Another thing to note is that we’ve been working on upgrading our pipelines to use dotnet build and dotnet test instead of the older VSBuild/Test Tasks and I’ve noticed that the scanner definitely still appears to be scanning these html files. (see below)

There’s exactly 61 html files in this source folder that we use in unit tests. As a work around we’ve removed the js from the file that was causing the error to pop up but we really don’t need these files scanned at all.

INFO: Sensor JavaScript inside HTML analysis [javascript]
INFO: 61 source files to be analyzed
INFO: 61/61 source files have been analyzed
INFO: Hit the cache for 0 out of 61
INFO: Miss the cache for 61 out of 61: FILE_CHANGED [61/61]
INFO: Sensor JavaScript inside HTML analysis [javascript] (done) | time=5045ms
1 Like

Thanks @NickBranstein,

yeah, that helps, thanks for the feedback. Indeed we were aware of an issue with non-js scripts which were also analyzed by the JS plugin. This will be fixed in the next release.

Cheers,
Victor

1 Like