However, after the scan completes, the only files scanned are python files. The json files are not listed in the report. In addition, there are html files as well that are not scanned. The json files are most important though as we need to scan CloudFormation templates written in JSON.
Hi,
Welcome to the community!
The first place to start looking here is at your sonar.sources definition and at any exclusions or inclusions you’ve set. All your files should be analyzed automatically, and if they’re not, then it’s because they’re not included in the source file set.
Can you maybe share your project structure, plus those^ settings?
Ann
Ann,
Thank you so much for answering my post. We are all novice users of SonarQube we are scanning our code with SonarQube defaults. We are using the SonarQube Enterprise UI. However, we are kicking off scans with Jenkins. We did not create or modify any definitions. The directory we are scanning is top level down with a mix of python files and json CloudFormation templates. The pythons files are scanned and the json files are skipped. We added the CloudFormation quality profile in the project settings and that’s pretty much it.
In Jenkins, both of these settings are empty for the SonarQube plugin settings:
Additional arguments
Additional command line arguments to be passed to the SonarQube scanner. For example, -X.
Additional analysis properties
Additional analysis properties in the form of key-value pairs. For example, sonar.analysis.mode=issues.
Are you saying there is a way to force SonarQube to scan CloudFormation json templates other than the quality profile?
Hi,
Can you share your analysis log?
The analysis / scanner log is what’s output from the analysis command. Hopefully, the log you provide - redacted as necessary - will include that command as well.
This guide will help you find them.
Ann
Ann,
I think these are the logs you need.
+ /home/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarScanner/bin/sonar-scanner
INFO: Scanner configuration file: /home/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarScanner/conf/sonar-scanner.properties
INFO: Project root configuration file: /home/jenkins/workspace/MyProject/sonar-project.properties
INFO: SonarScanner 5.0.1.3006
INFO: Java 17.0.8 Eclipse Adoptium (64-bit)
INFO: Linux 5.19.0-1029-aws amd64
INFO: User cache: /home/jenkins/.sonar/cache
INFO: Analyzing on SonarQube server 9.9.1.69595
INFO: Default locale: "en_US", source code encoding: "UTF-8" (analysis is platform dependent)
INFO: Load global settings
INFO: Load global settings (done) | time=107ms
INFO: Server id: MyServerID
INFO: User cache: /home/jenkins/.sonar/cache
INFO: Load/download plugins
INFO: Load plugins index
INFO: Load plugins index (done) | time=59ms
INFO: Load/download plugins (done) | time=471ms
INFO: Loaded core extensions: developer-scanner
INFO: Process project properties
INFO: Process project properties (done) | time=1ms
INFO: Execute project builders
INFO: Execute project builders (done) | time=2ms
INFO: Project key: MyProjectKey
INFO: Base dir: /home/jenkins/workspace/MyProject
INFO: Working dir: /home/jenkins/workspace/MyProject/.scannerwork
INFO: Load project settings for component key: 'MyProjectKey'
INFO: Load project settings for component key: 'MyProjectKey' (done) | time=32ms
INFO: Load project branches
INFO: Load project branches (done) | time=15ms
INFO: Load branch configuration
INFO: Load branch configuration (done) | time=3ms
INFO: Auto-configuring with CI 'Jenkins'
INFO: Load quality profiles
INFO: Load quality profiles (done) | time=72ms
INFO: Load active rules
INFO: Load active rules (done) | time=1985ms
INFO: Load analysis cache
INFO: Load analysis cache | time=85ms
INFO: Load project repositories
INFO: Load project repositories (done) | time=20ms
INFO: Indexing files...
INFO: Project configuration:
INFO: 106 files indexed
INFO: 0 files ignored because of scm ignore settings
INFO: Quality profile for js: Sonar way
INFO: Quality profile for json: Sonar way
INFO: Quality profile for py: Sonar way
INFO: ------------- Run sensors on module MyProjectKey
INFO: Load metrics repository
INFO: Load metrics repository (done) | time=37ms
INFO: Sensor IaC CloudFormation Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC CloudFormation Sensor [iac] (done) | time=36ms
INFO: Sensor IaC Kubernetes Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC Kubernetes Sensor [iac] (done) | time=13ms
INFO: Sensor C# Project Type Information [csharp]
INFO: Sensor C# Project Type Information [csharp] (done) | time=1ms
INFO: Sensor C# Analysis Log [csharp]
INFO: Sensor C# Analysis Log [csharp] (done) | time=28ms
INFO: Sensor C# Properties [csharp]
INFO: Sensor C# Properties [csharp] (done) | time=0ms
INFO: Sensor HTML [web]
INFO: Sensor HTML [web] (done) | time=5ms
INFO: Sensor TextAndSecretsSensor [text]
INFO: 56 source files to be analyzed
INFO: 56/56 source files have been analyzed
INFO: Sensor TextAndSecretsSensor [text] (done) | time=633ms
INFO: Sensor VB.NET Project Type Information [vbnet]
INFO: Sensor VB.NET Project Type Information [vbnet] (done) | time=2ms
INFO: Sensor VB.NET Analysis Log [vbnet]
INFO: Sensor VB.NET Analysis Log [vbnet] (done) | time=16ms
INFO: Sensor VB.NET Properties [vbnet]
INFO: Sensor VB.NET Properties [vbnet] (done) | time=1ms
INFO: Sensor Python Sensor [python]
WARN: Your code is analyzed as compatible with python 2 and 3 by default. This will prevent the detection of issues specific to python 2 or python 3. You can get a more precise analysis by setting a python version in your configuration via the parameter "sonar.python.version"
INFO: Starting global symbols computation
INFO: 52 source files to be analyzed
INFO: 52/52 source files have been analyzed
INFO: Starting rules execution
INFO: 52 source files to be analyzed
INFO: 52/52 source files have been analyzed
INFO: The Python analyzer was able to leverage cached data from previous analyses for 0 out of 52 files. These files were not parsed.
INFO: Sensor Python Sensor [python] (done) | time=8372ms
INFO: Sensor Cobertura Sensor for Python coverage [python]
INFO: Sensor Cobertura Sensor for Python coverage [python] (done) | time=43ms
INFO: Sensor PythonXUnitSensor [python]
INFO: Sensor PythonXUnitSensor [python] (done) | time=45ms
INFO: Sensor JaCoCo XML Report Importer [jacoco]
INFO: 'sonar.coverage.jacoco.xmlReportPaths' is not defined. Using default locations: target/site/jacoco/jacoco.xml,target/site/jacoco-it/jacoco.xml,build/reports/jacoco/test/jacocoTestReport.xml
INFO: No report imported, no coverage information will be imported by JaCoCo XML Report Importer
INFO: Sensor JaCoCo XML Report Importer [jacoco] (done) | time=2ms
INFO: Sensor JavaScript analysis [javascript]
ERROR: Error when running: 'node -v'. Is Node.js available during analysis?
org.sonarsource.nodejs.NodeCommandException: Error when running: 'node -v'. Is Node.js available during analysis?
at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:79)
at org.sonarsource.nodejs.NodeCommandBuilderImpl.getVersion(NodeCommandBuilderImpl.java:203)
at org.sonarsource.nodejs.NodeCommandBuilderImpl.checkNodeCompatibility(NodeCommandBuilderImpl.java:169)
at org.sonarsource.nodejs.NodeCommandBuilderImpl.build(NodeCommandBuilderImpl.java:143)
at org.sonar.plugins.javascript.eslint.EslintBridgeServerImpl.initNodeCommand(EslintBridgeServerImpl.java:201)
at org.sonar.plugins.javascript.eslint.EslintBridgeServerImpl.startServer(EslintBridgeServerImpl.java:142)
at org.sonar.plugins.javascript.eslint.EslintBridgeServerImpl.startServerLazily(EslintBridgeServerImpl.java:233)
at org.sonar.plugins.javascript.eslint.AbstractEslintSensor.execute(AbstractEslintSensor.java:68)
at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:64)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:88)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.lambda$execute$1(ModuleSensorsExecutor.java:61)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.withModuleStrategy(ModuleSensorsExecutor.java:79)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:61)
at org.sonar.scanner.scan.SpringModuleScanContainer.doAfterStart(SpringModuleScanContainer.java:82)
at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
at org.sonar.scanner.scan.SpringProjectScanContainer.scan(SpringProjectScanContainer.java:403)
at org.sonar.scanner.scan.SpringProjectScanContainer.scanRecursively(SpringProjectScanContainer.java:399)
at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:368)
at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:137)
at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72)
at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66)
at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
at jdk.proxy1/jdk.proxy1.$Proxy0.execute(Unknown Source)
at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
at org.sonarsource.scanner.cli.Main.execute(Main.java:126)
at org.sonarsource.scanner.cli.Main.execute(Main.java:81)
at org.sonarsource.scanner.cli.Main.main(Main.java:62)
Caused by: java.io.IOException: Cannot run program "node": error=2, No such file or directory
at java.base/java.lang.ProcessBuilder.start(Unknown Source)
at java.base/java.lang.ProcessBuilder.start(Unknown Source)
at org.sonarsource.nodejs.ProcessWrapperImpl.startProcess(ProcessWrapperImpl.java:39)
at org.sonarsource.nodejs.NodeCommand.start(NodeCommand.java:77)
... 37 common frames omitted
Caused by: java.io.IOException: error=2, No such file or directory
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
at java.base/java.lang.ProcessImpl.start(Unknown Source)
... 41 common frames omitted
INFO: Hit the cache for 0 out of 0
INFO: Miss the cache for 0 out of 0
INFO: Sensor JavaScript analysis [javascript] (done) | time=3038ms
INFO: Sensor TypeScript analysis [javascript]
INFO: No input files found for analysis
INFO: Hit the cache for 0 out of 0
INFO: Miss the cache for 0 out of 0
INFO: Sensor TypeScript analysis [javascript] (done) | time=2ms
INFO: Sensor CSS Rules [javascript]
INFO: No CSS, PHP, HTML or VueJS files are found in the project. CSS analysis is skipped.
INFO: Sensor CSS Rules [javascript] (done) | time=1ms
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend]
INFO: Sensor ThymeLeaf template sensor [securityjavafrontend] (done) | time=1ms
INFO: Sensor Python HTML templates processing [securitypythonfrontend]
INFO: HTML files are not indexed : you may want to add them in the scanned files of this project to detect Python XSS vulnerabilities
INFO: Sensor Python HTML templates processing [securitypythonfrontend] (done) | time=257ms
INFO: Sensor IaC Docker Sensor [iac]
INFO: 0 source files to be analyzed
INFO: 0/0 source files have been analyzed
INFO: Sensor IaC Docker Sensor [iac] (done) | time=89ms
INFO: Sensor Serverless configuration file sensor [security]
INFO: 0 Serverless function entries were found in the project
INFO: 0 Serverless function handlers were kept as entrypoints
INFO: Sensor Serverless configuration file sensor [security] (done) | time=5ms
INFO: Sensor AWS SAM template file sensor [security]
INFO: Sensor AWS SAM template file sensor [security] (done) | time=0ms
INFO: Sensor AWS SAM Inline template file sensor [security]
INFO: Sensor AWS SAM Inline template file sensor [security] (done) | time=0ms
INFO: Sensor javabugs [dbd]
INFO: Reading IR files from: /home/jenkins/workspace/MyProject/.scannerwork/ir/java
INFO: No IR files have been included for analysis.
INFO: Sensor javabugs [dbd] (done) | time=2ms
INFO: Sensor pythonbugs [dbd]
INFO: Reading IR files from: /home/jenkins/workspace/MyProject/.scannerwork/ir/python
INFO: Analyzing 84 functions to detect bugs.
INFO: Sensor pythonbugs [dbd] (done) | time=914ms
INFO: Sensor JavaSecuritySensor [security]
INFO: Reading type hierarchy from: /home/jenkins/workspace/MyProject/.scannerwork/ucfg2/java
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor JavaSecuritySensor [security] (done) | time=4ms
INFO: Sensor CSharpSecuritySensor [security]
INFO: Reading type hierarchy from: /home/jenkins/workspace/MyProject/ucfg_cs2
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor CSharpSecuritySensor [security] (done) | time=1ms
INFO: Sensor PhpSecuritySensor [security]
INFO: Reading type hierarchy from: /home/jenkins/workspace/MyProject/.scannerwork/ucfg2/php
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor PhpSecuritySensor [security] (done) | time=1ms
INFO: Sensor PythonSecuritySensor [security]
INFO: Reading type hierarchy from: /home/jenkins/workspace/MyProject/.scannerwork/ucfg2/python
INFO: Read 239 type definitions
INFO: Reading UCFGs from: /home/jenkins/workspace/MyProject/.scannerwork/ucfg2/python
INFO: 13:59:54.086984875 Building Runtime Type propagation graph
INFO: 13:59:54.127436671 Running Tarjan on 2853 nodes
INFO: 13:59:54.144785279 Tarjan found 2853 components
INFO: 13:59:54.172420489 Variable type analysis: done
INFO: 13:59:54.176544326 Building Runtime Type propagation graph
INFO: 13:59:54.208293358 Running Tarjan on 2823 nodes
INFO: 13:59:54.215483067 Tarjan found 2823 components
INFO: 13:59:54.228595266 Variable type analysis: done
INFO: Analyzing 1090 ucfgs to detect vulnerabilities.
INFO: All rules entrypoints : 0
INFO: Retained UCFGs : 0
INFO: Taint analysis starting. Entrypoints: 0
INFO: Taint analysis: done.
INFO: Sensor PythonSecuritySensor [security] (done) | time=1285ms
INFO: Sensor JsSecuritySensor [security]
INFO: Reading type hierarchy from: /home/jenkins/workspace/MyProject/.scannerwork/ucfg2/js
INFO: Read 0 type definitions
INFO: No UCFGs have been included for analysis.
INFO: Sensor JsSecuritySensor [security] (done) | time=1ms
INFO: ------------- Run sensors on project
INFO: Sensor Analysis Warnings import [csharp]
INFO: Sensor Analysis Warnings import [csharp] (done) | time=2ms
INFO: Sensor Zero Coverage Sensor
INFO: Sensor Zero Coverage Sensor (done) | time=24ms
INFO: CPD Executor 4 files had no CPD blocks
INFO: CPD Executor Calculating CPD for 48 files
INFO: CPD Executor CPD calculation finished (done) | time=139ms
INFO: Load New Code definition
INFO: Load New Code definition (done) | time=10ms
INFO: Analysis report generated in 242ms, dir size=995.5 kB
INFO: Analysis report compressed in 300ms, zip size=613.1 kB
INFO: Analysis report uploaded in 121ms
INFO: ANALYSIS SUCCESSFUL, you can find the results at: MySonarQUbeServer/dashboard?id=MyProjectKey
INFO: Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
INFO: More about the report processing at MySonarQUbeServer/api/ce/task?id=MyTaskId
INFO: Analysis total time: 25.816 s
INFO: ------------------------------------------------------------------------
INFO: EXECUTION SUCCESS
INFO: ------------------------------------------------------------------------
INFO: Total time: 28.402s
INFO: Final Memory: 48M/167M
INFO: ------------------------------------------------------------------------
Hi,
Thanks for the log. Nothing much sticks out to me here. Can you share the contents of Project Settings → Background Tasks → [top row cog menu] → Show Scanner Context? This is the property values used in the analysis.
Also, can you give an idea of your project structure?
And to belatedly answer your question:
No. I’m not. 
Ann
Ann,
Here is the file you requested. Can you elaborate more on project structure? In the meantime, I will try to scan just one individual json file. If by structure, do you mean directory structure?
Thanks.
Plugins:
Bundled analyzers:
- IaC Code Quality and Security 1.11.0.2847 (iac)
- PL/SQL Code Quality and Security 3.8.0.4948 (plsql)
- Scala Code Quality and Security 1.11.0.3905 (sonarscala)
- C# Code Quality and Security 8.51.0.59060 (csharp)
- Vulnerability Analysis 9.9.0.19083 (security)
- Java Code Quality and Security 7.16.0.30901 (java)
- HTML Code Quality and Security 3.7.1.3306 (web)
- Flex Code Quality and Security 2.8.0.3166 (flex)
- XML Code Quality and Security 2.7.0.3820 (xml)
- Text Code Quality and Security 2.0.2.1090 (text)
- VB.NET Code Quality and Security 8.51.0.59060 (vbnet)
- Swift Code Quality and Security 4.8.0.5759 (swift)
- CFamily Code Quality and Security 6.41.0.60884 (cpp)
- Python Code Quality and Security 3.24.0.10784 (python)
- Dataflow Bug Detection Rules for Python 1.10.0.3046 (dbdpythonfrontend)
- Dataflow Bug Detection 1.10.0.3046 (dbd)
- Go Code Quality and Security 1.11.0.3905 (go)
- JaCoCo 1.3.0.1538 (jacoco)
- Kotlin Code Quality and Security 2.12.0.1956 (kotlin)
- RPG Code Quality 3.3.0.3147 (rpg)
- Dataflow Bug Detection Rules for Java 1.10.0.3046 (dbdjavafrontend)
- PL/I Code Quality and Security 1.12.0.3443 (pli)
- T-SQL Code Quality and Security 1.7.0.5449 (tsql)
- VB6 Code Quality and Security 2.9.0.3341 (vb)
- Apex Code Quality and Security 1.11.0.3905 (sonarapex)
- JavaScript/TypeScript/CSS Code Quality and Security 9.13.0.20537 (javascript)
- Ruby Code Quality and Security 1.11.0.3905 (ruby)
- Vulnerability Rules for C# 9.9.0.19083 (securitycsharpfrontend)
- Vulnerability Rules for Java 9.9.0.19083 (securityjavafrontend)
- Vulnerability Rules for JS 9.9.0.19083 (securityjsfrontend)
- COBOL Code Quality 5.2.0.5949 (cobol)
- Vulnerability Rules for Python 9.9.0.19083 (securitypythonfrontend)
- PHP Code Quality and Security 3.27.1.9352 (php)
- ABAP Code Quality and Security 3.11.0.4030 (abap)
- Configuration detection fot Code Quality and Security 1.2.0.267 (config)
- Vulnerability Rules for PHP 9.9.0.19083 (securityphpfrontend)
Global server settings:
- sonar.cloudformation.activate=true
- sonar.core.id=MySonarCoreID
- sonar.core.startTime=2023-08-14T17:02:00+0000
- sonar.forceAuthentication=true
- sonar.plugins.risk.consent=ACCEPTED
- sonar.qualitygate.ignoreSmallChanges=false
Project server settings:
Project scanner properties:
- sonar.host.url=MySonarHost
- sonar.login=******
- sonar.projectBaseDir=/home/jenkins/workspace/MyProject
- sonar.projectKey=MyProjectKey
- sonar.scanner.app=ScannerCLI
- sonar.scanner.appVersion=5.0.1.3006
- sonar.sourceEncoding=UTF-8
- sonar.working.directory=/home/jenkins/workspace/MyProject/.scannerwork
Close
After running the test on two json files, this is a snippet of the output, I did notice that CloudFormation has no language pattern as well as Kubernetes and docker. Is there a way to add a json language pattern to CloudFormation?
09:15:28.927 DEBUG: Declared extensions of language Terraform were converted to sonar.lang.patterns.terraform : **/*.tf
09:15:28.927 DEBUG: Declared extensions of language CloudFormation were converted to sonar.lang.patterns.cloudformation :
09:15:28.927 DEBUG: Declared extensions of language Kubernetes were converted to sonar.lang.patterns.kubernetes :
09:15:28.927 DEBUG: Declared extensions of language Docker were converted to sonar.lang.patterns.docker :
09:15:28.927 DEBUG: Declared extensions of language PL/SQL were converted to sonar.lang.patterns.plsql : **/*.sql,**/*.pks,**/*.pkb
09:15:28.928 DEBUG: Declared extensions of language Scala were converted to sonar.lang.patterns.scala : **/*.scala
09:15:28.928 DEBUG: Declared extensions of language C# were converted to sonar.lang.patterns.cs : **/*.cs
My test revealed that apparently you need the following somewhere in the file for the CloudFormation file to be scanned:
{
"AWSTemplateFormatVersion" : "2010-09-09",
This would require that a ton of files be updated. I do think that if we just said parse json files, the specific CloudFormation plugin would be rendered useless.
After looking into it more, I discovered that you can set whatever identifier you want for the files in Administration → Configuration → Languages → CloudFormation → File Identifier
Hi,
Yes, I was asking about your project’s file/directory structure.
Did this fix it for you?
Ann
Ann,
Yes it did.
Thank you for your help.
Hi @jcao,
Welcome to the community.
This thread was resolved but every reply spams the OP with a new message. Please create a new thread with all your details.
Thx,
Ann