Problem with excluding files from scan

Hi,

I’m new to SQ and trying to setup scans for our C++ project.
SQ server: 9.9.0.65466
Sonar scanner: 4.7.0.2747
Deployed as a zip

The setup of the projects is:

c:\source\Libs
c:\source\Application1
c:\source\Application2
c:\source\Application3

All 3 applications are using the same source code from Libs.
Cmake configuration and build parameters are setup in c:\source\Libs. All applications are built from c:\source\Libs directory

Currently I have 4 scanning jobs: ‘Libs’ only and then one for each application.

Each application has test files which are mixed in with source code. I’ve created a sonar.exclusion pattern to exclude those files but
when the scan is executed the files are not being excluded. When I look at the scan results on the server I get Duplicate rate 29.9% all based on
the files that are suppose to be excluded.

Here is the sample of property file

sonar.sources=c:/source/application1/
sonar.projectBaseDir=C:/source/application1/
sonar.exclusions=C:/source/application1/test/*,C:/source/application1/test/approved/*,C:/source/application1/mock/*,C:/source/application1/**/**/**/*.html

The following information is logged:

INFO: Project configuration:
INFO:   Excluded sources: C:/source/application1/test/*, C:/source/application1/test/approved/*, C:/source/application1/mock/*, C:/source/application1/**/**/**/*.html
INFO: 924 files indexed
INFO: 0 files ignored because of inclusion/exclusion patterns
INFO: 0 files ignored because of scm ignore settings

Are there any additional parameters I need to include to exclude test files from being scanned?

I’ve tried using the command line rather than property file, and simplifying the exclude to a single directory. I’ve also confirmed the path is correct, and it’s the same format as my projectBaseDir, which is getting picked up correctly.

Thanks

Hi,

Welcome to the community!

I find your need to specify a full path for sonar.sources and your use of sonar.projectBaseDir suspicious.

But can you share your full analysis log?

The analysis / scanner log is what’s output from the analysis command. Hopefully, the log you provide - redacted as necessary - will include that command as well.

This guide will help you find them.

 
Ann

Hi Ann,

Here are the logs that were outputted during running of sonar scan.

INFO: Scanner configuration file: C:\sonar-scanner-4.7.0.2747-windows\bin\..\conf\sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: SonarScanner 4.7.0.2747
INFO: Java 11.0.14.1 Eclipse Adoptium (64-bit)
INFO: Windows Server 2012 R2 6.3 amd64
INFO: User cache: C:\Users\xxxxxx\.sonar\cache
INFO: Scanner configuration file: C:\sonar-scanner-4.7.0.2747-windows\bin\..\conf\sonar-scanner.properties
INFO: Project root configuration file: NONE
INFO: Analyzing on SonarQube server 9.9.0.65466
INFO: Default locale: "en_US", source code encoding: "windows-1252" (analysis is platform dependent)
INFO: Load global settings
INFO: Load global settings (done) | time=98ms
INFO: Server id: Application1_20221208
INFO: User cache: C:\Users\xxxxxxx\.sonar\cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=55ms
INFO: Load/download plugins (done) | time=556ms
INFO: Loaded core extensions: developer-scanner
INFO: Process project properties
INFO: Process project properties (done) | time=15ms
INFO: Execute project builders
INFO: Execute project builders (done) | time=5ms
INFO: Project key: xxxxxxx
INFO: Base dir: C:\source\application1\source
INFO: Working dir: C:\source\Application1\source\.scannerwork
INFO: Load project settings for component key: 'Application1_20221208'
INFO: Load project settings for component key: 'Application1_20221208' (done) | time=33ms
INFO: Load project branches
INFO: Load project branches (done) | time=21ms
INFO: Load branch configuration
INFO: Load branch configuration (done) | time=2ms
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=92ms
INFO: Load active rules
INFO: Load active rules (done) | time=2323ms
INFO: Load analysis cache
INFO: Load analysis cache | time=37ms
INFO: Load project repositories
INFO: Load project repositories (done) | time=52ms
INFO: Indexing files...
INFO: Project configuration:
INFO:   Excluded sources: C:/autoscan_source/Application1/source/test/*, C:/autoscan_source/source_VOS2/source/test/approved/*, C:/autoscan_source/source_VOS2/source/mock/*, C:/autoscan_source/source_VOS2/source/**/**/**/*.html
INFO: 924 files indexed
INFO: 0 files ignored because of inclusion/exclusion patterns
INFO: 0 files ignored because of scm ignore settings
INFO: Quality profile for c: Sonar way
INFO: Quality profile for cpp: Sonar way
INFO: Quality profile for web: Sonar way
INFO: Quality profile for xml: Sonar way
INFO: Quality profile for yaml: Sonar way
INFO: ------------- Run sensors on module Application1_20221208
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=25ms
INFO: Sensor IaC CloudFormation Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC CloudFormation Sensor [iac] (done) | time=71ms
INFO: Sensor IaC Kubernetes Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC Kubernetes Sensor [iac] (done) | time=22ms
INFO: Sensor C# Project Type Information [csharp]
INFO: Sensor C# Project Type Information [csharp] (done) | time=8ms
INFO: Sensor C# Analysis Log [csharp]
INFO: Sensor C# Analysis Log [csharp] (done) | time=20ms
INFO: Sensor C# Properties [csharp]
INFO: Sensor C# Properties [csharp] (done) | time=0ms
INFO: Sensor HTML [web]
INFO: Sensor HTML [web] (done) | time=17283ms
INFO: Sensor XML Sensor [xml]
INFO: 4 source files to be analyzed
INFO: 4/4 source files have been analyzed
INFO: Sensor XML Sensor [xml] (done) | time=592ms
INFO: Sensor TextAndSecretsSensor [text]
INFO: 860 source files to be analyzed
INFO: 860/860 source files have been analyzed
INFO: Sensor TextAndSecretsSensor [text] (done) | time=3039ms
INFO: Sensor VB.NET Project Type Information [vbnet]
INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=1ms
INFO: Sensor VB.NET Analysis Log [vbnet]
INFO: Sensor VB.NET Analysis Log [vbnet] (done) | time=20ms
INFO: Sensor VB.NET Properties [vbnet]
INFO: Sensor VB.NET Properties [vbnet] (done) | time=0ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=2ms
INFO: Sensor JavaScript inside YAML analysis [javascript]
INFO: No input files found for analysis
INFO: Hit the cache for 0 out of 0
INFO: Miss the cache for 0 out of 0
INFO: Sensor JavaScript inside YAML analysis [javascript] (done) | time=13ms
INFO: Sensor CSS Rules [javascript]
WARN: Node.js not found in PATH. PATH value was: C:\Program Files\Common Files\Oracle\Java\javapath;C:\Python38\Scripts\;C:\Python38\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files\Amazon\cfn-bootstrap\;C:\ProgramData\chocolatey\bin;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files\Microsoft VS Code\bin;C:\Program Files\Git\cmd;c:\OpenSSH-Win64;C:\Program Files\SafeNet\Authentication\SAC\x64;C:\Program Files\SafeNet\Authentication\SAC\x32;C:\Program Files\Gemalto\Classic Client\BIN;C:\Program Files (x86)\Gemalto\Classic Client\BIN;C:\Program Files (x86)\VeriFone\FST\
INFO: Hit the cache for 0 out of 0
INFO: Miss the cache for 0 out of 0
INFO: Sensor CSS Rules [javascript] (done) | time=60161ms
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend]
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend] (done) | time=210ms
INFO: Sensor IaC Docker Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC Docker Sensor [iac] (done) | time=110ms
INFO: Sensor Serverless configuration file sensor [security]
INFO: 0 Serverless function entries were found in the project
INFO: 0 Serverless function handlers were kept as entrypoints
INFO: Sensor Serverless configuration file sensor [security] (done) | time=8ms
INFO: Sensor AWS SAM template file sensor [security]
INFO: Sensor AWS SAM template file sensor [security] (done) | time=6ms
INFO: Sensor AWS SAM Inline template file sensor [security]
INFO: Sensor AWS SAM Inline template file sensor [security] (done) | time=15ms
INFO: Sensor CFamily [cpp]
INFO: CFamily plugin version: 6.41.0.60884
INFO: Using build-wrapper output: c:\autoscan_source\Libs_VOS2\bw_output\build-wrapper-dump.json
INFO: Available processors: 2
INFO: Using 2 threads for analysis.
INFO: Loading cache from: server
INFO: Cache hit for: C:\autoscan_source\Application1\source\Main.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\AccountVerification.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\AuthFunctions.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\AdminMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\Batch.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\Auth.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\AmountEntry.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\CardEntry.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\CommFactory.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\BatchFunctions.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\Checks.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\CloudPayment.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\DisplayTrans.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\FinalAuth.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\EMVTrans.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\EMVUI.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\EMVUtils.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\EventHandler.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\GiftMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\ForceRefer.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\HelpMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\IntegratedHandler.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\GiftFunctions.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\IntegratedThread.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\MicrosFuncs.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\MainMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\ManagementMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\IPMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\MACMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\sourceRetCodes.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\Operations.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\PayAtTable.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\PRBatch.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\PayRouterPkt.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\PreAuth.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\Refund.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\PRComms.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\Receipt.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\PRReports.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\PRUtils.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\ShiftMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\Sale.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\ReportsMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\TerminalInfo.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\SettingsMenu.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\TransFns.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\Transaction.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\Thrifty.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\TillInterface.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\OracleCloud.cpp
INFO: Cache hit for: C:\autoscan_source\Application1\source\AlternativePayment.cpp
INFO: PCH: unique=0 use=0 (forceInclude=0,throughHeader=0,firstInclude=0) out of 51 (forceInclude=0,throughHeader=0)
INFO: SE: 51 out of 51
INFO: Z3 refutation rate: 0 out of 8
INFO: Subprocess(es) done in 7414ms
INFO: 136/136 files marked as unchanged
INFO: Cache: 51/51 hits, 609330 bytes
INFO: 51 compilation units analyzed
INFO: Sensor CFamily [cpp] (done) | time=14700ms
INFO: Sensor javabugs [dbd]
INFO: Reading IR files from: C:\autoscan_source\Application1\source\.scannerwork\ir\java
INFO: No IR files have been included for analysis.
INFO: Sensor javabugs [dbd] (done) | time=2ms
INFO: Sensor pythonbugs [dbd]
INFO: Reading IR files from: C:\autoscan_source\Application1\source\.scannerwork\ir\python
INFO: No IR files have been included for analysis.
INFO: Sensor pythonbugs [dbd] (done) | time=2ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Reading type hierarchy from: C:\autoscan_source\Application1\source\.scannerwork\ucfg2\java
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor JavaSecuritySensor [security] (done) | time=4ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading type hierarchy from: C:\autoscan_source\Application1\source\ucfg_cs2
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor CSharpSecuritySensor [security] (done) | time=1ms
INFO: Sensor PhpSecuritySensor [security]
INFO: Reading type hierarchy from: C:\autoscan_source\Application1\source\.scannerwork\ucfg2\php
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor PhpSecuritySensor [security] (done) | time=0ms
INFO: Sensor PythonSecuritySensor [security]
INFO: Reading type hierarchy from: C:\autoscan_source\Application1\source\.scannerwork\ucfg2\python
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor PythonSecuritySensor [security] (done) | time=0ms
INFO: Sensor JsSecuritySensor [security]
INFO: Reading type hierarchy from: C:\autoscan_source\Application1\source\.scannerwork\ucfg2\js
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor JsSecuritySensor [security] (done) | time=1ms
INFO: ------------- Run sensors on project
INFO: Sensor Analysis Warnings import [csharp]
INFO: Sensor Analysis Warnings import [csharp] (done) | time=1ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=455ms
INFO: CPD Executor 64 files had no CPD blocks
INFO: CPD Executor Calculating CPD for 574 files
WARN: Too many duplication references on file test/GUITest.cpp for block at line 421. Keep only the first 100 references.
WARN: Too many duplication references on file test/GUITest.cpp for block at line 366. Keep only the first 100 references.
WARN: Too many duplication references on file test/GUITest.cpp for block at line 323. Keep only the first 100 references.
WARN: Too many duplication references on file test/GUITest.cpp for block at line 325. Keep only the first 100 references.
WARN: Too many duplication references on file test/GUITest.cpp for block at line 323. Keep only the first 100 references.
INFO: CPD Executor CPD calculation finished (done) | time=2086ms
INFO: Load New Code definition
INFO: Load New Code definition (done) | time=21ms
INFO: Analysis report generated in 4515ms, dir size=7.2 MB
INFO: Analysis report compressed in 2718ms, zip size=2.7 MB
INFO: Analysis report uploaded in 144ms
INFO: ANALYSIS SUCCESSFUL, you can find the results at: http://1.1.1.1:9999/dashboard?id=Application1_20221208
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at http://1.1.1.1:9999/api/ce/task?id=AYcvEapaTzQfOTE5zajo
INFO: Analysis total time: 2:17.296 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 2:20.354s
INFO: Final Memory: 35M/120M

Hi,

Thanks for the log. A few things look strange to me:

Unless this is the result of an uneven redaction, your exclusions don’t seem to match up with where analysis is running:

base dir:
C:\source\application1\source
vs exclusions
C:/autoscan_source/Application1/source/
C:/autoscan_source/source_VOS2/source

Both the path segments and the casing(!) don’t match. And especially since you’re specifying full paths, if they don’t match then it’s natural that nothing is excluded.

Also, I took a 2nd look at this pattern: C:/autoscan_source/source_VOS2/source/**/**/**/*.html

The ** wildcard matches 0-n directories. So **/**/** is not helping.

I suggest you rewrite your exclusions (at minimum) to something like:
**/test/**/*, **/mock/*, **/source_VOS2/source/**/*.html

Note that you may want to make that ‘mock’ one **/mock/**/* unless all the files to be ignored are directly under the mock directory.

 
HTH,
Ann