Kubernetes scanning is not working

pjasinski · October 31, 2023, 2:00pm

Hello,

We are using SonarQube version 10.2.1.78527, Developer Edition. Task versions in Azure Pipelines: SonarQubePrepare and SonarQubeAnalyze 5.17.2, SonarQubePublish 5.4.2.
SonarQube is deployed inside a K8s cluster without Helm, just a yaml manifest with deployment.
We are trying to scan a k8s manifest using SonarQube and it doesn’t seem to catch any issues with it. Here is the manifest, you can see some obvious security issues in this vulns.yaml file.

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: vulnerable-pod
  name: vulnerable-pod
spec:
  containers:
  - image: nginx:1.19.9
    name: vulnerable-pod
    securityContext:
      runAsUser: 0
      privileged: true
      allowPrivilegeEscalation: true
      capabilities:
        add: ["SYS_ADMIN"]

In the logs we can see that the yaml files have been indexed successfully, however right after that K8s scanner does nothing as it does not detect the yaml files as K8s manifests (?), for Dockerfiles the scanning works well.

2023-10-31T13:24:32.6186921Z 13:24:32.618 DEBUG: 8 non excluded files in this Git repository
2023-10-31T13:24:32.6290603Z 13:24:32.628 DEBUG: 'secret.yml' indexed with language 'kubernetes'
2023-10-31T13:24:32.6331846Z 13:24:32.632 DEBUG: 'Dockerfile' indexed with language 'docker'
2023-10-31T13:24:32.6342236Z 13:24:32.634 DEBUG: 'vulns.yaml' indexed with language 'kubernetes'
2023-10-31T13:24:32.6361956Z 13:24:32.636 INFO: 3 files indexed
2023-10-31T13:24:32.6363184Z 13:24:32.636 INFO: 3 files ignored because of inclusion/exclusion patterns
2023-10-31T13:24:32.6363831Z 13:24:32.636 INFO: 0 files ignored because of scm ignore settings
2023-10-31T13:24:32.6374356Z 13:24:32.637 INFO: Quality profile for docker: Sonar way
2023-10-31T13:24:32.6375035Z 13:24:32.637 INFO: Quality profile for kubernetes: Sonar way
(…)
2023-10-31T13:24:34.3078134Z 13:24:34.307 DEBUG: 'IaC Kubernetes Sensor' skipped because there is no related file in current project
(…)
2023-10-31T13:24:34.6060541Z 13:24:34.605 INFO: Sensor IaC Docker Sensor [iac]
2023-10-31T13:24:34.6080035Z 13:24:34.607 INFO: 1 source file to be analyzed
2023-10-31T13:24:34.6866588Z 13:24:34.686 INFO: 1/1 source file has been analyzed

This is the relevant part of our pipeline:

- task: SonarQubePrepare@5
  inputs:
    SonarQube: 'SonarQube'
    scannerMode: 'CLI'
    configMode: 'manual'
    cliProjectKey: '<REDACTED>'
    cliSources: '.'
    extraProperties: |
      sonar.projectName=<REDACTED>
      sonar.verbose=true
      sonar.qualitygate.wait=true
      # sonar.qualitygate.timeout=60
      sonar.inclusions=**/*.yaml,**/*.yml,**/Dockerfile
      sonar.exclusions=**/azure-pipelines.yml,**/azure-pipelines.yaml
      sonar.lang.patterns.kubernetes=**/*.yaml,**/*.yml
      sonar.lang.patterns.docker=**/Dockerfile
      sonar.lang.patterns.yaml=""
      # sonar.scm.disabled=true
- task: SonarQubeAnalyze@5
- task: SonarQubePublish@5
  inputs:
    pollingTimeoutSec: '300'

Apart from that, quality profiles for the project are configured as in the screenshot

the Kubernetes rules are active

and we’ve even recreated the project hoping it would help. It didn’t change anything.
Anything we’re missing here?

Kind regards,
Patryk Jasinski

Marcin_Stachniuk · November 7, 2023, 9:49am

Cześć Patryk!

Welcome to Sonar Community — thanks for contributing!

Could you try to comment sonar.lang.patterns.kubernetes and sonar.lang.patterns.yaml please? In my opinion, those settings are contradictory.

If it doesn’t work, could you share more info about the project, please?

Do you have other files than Kubernetes and Docker in repository, e.g. some C# or any other code?
Where vulns.yaml is located, project root or some subfolder?

Best
Marcin Stachniuk

pjasinski · November 8, 2023, 8:06am

Cześć

If I comment out these 2 properties, then K8s manifests are analyzed by yaml sensor, and we have no active rules for yaml so it doesn’t really do anything.
Although this time it looks like at least the file with secret was recognized as a Kubernetes manifest (?)

07:35:43.650 DEBUG: 'secret.yml' indexed with language 'yaml'
07:35:43.655 DEBUG: 'Dockerfile' indexed with language 'docker'
07:35:43.657 DEBUG: 'vulns.yaml' indexed with language 'yaml'
07:35:43.661 INFO: 3 files indexed
07:35:43.661 INFO: 1 file ignored because of inclusion/exclusion patterns
07:35:43.662 INFO: 0 files ignored because of scm ignore settings
07:35:43.663 INFO: Quality profile for docker: Sonar way
07:35:43.663 INFO: Quality profile for yaml: Sonar way
(...)
07:45:33.395 DEBUG: Sensors : IaC CloudFormation Sensor -> IaC Kubernetes Sensor -> C# Project Type Information -> C# Analysis Log -> C# Properties -> HTML -> TextAndSecretsSensor -> VB.NET Project Type Information -> VB.NET Analysis Log -> VB.NET Properties -> JaCoCo XML Report Importer -> JavaScript inside YAML analysis -> CSS Rules -> ThymeLeaf template sensor -> IaC Docker Sensor -> Serverless configuration file sensor -> AWS SAM template file sensor -> AWS SAM Inline template file sensor -> javabugs -> pythonbugs -> JavaSecuritySensor -> CSharpSecuritySensor -> PhpSecuritySensor -> PythonSecuritySensor -> JsSecuritySensor
07:45:33.396 INFO: Sensor IaC CloudFormation Sensor [iac]
07:45:33.403 DEBUG: 'secret.yml' generated metadata with charset 'US-ASCII'
07:45:33.404 DEBUG: File without identifier 'AWSTemplateFormatVersion': file:///azp/_work/1/s/secret.yml
07:45:33.405 DEBUG: 'vulns.yaml' generated metadata with charset 'US-ASCII'
07:45:33.406 DEBUG: File without identifier 'AWSTemplateFormatVersion': file:///azp/_work/1/s/vulns.yaml
07:45:33.411 INFO: 0 source files to be analyzed
07:45:33.421 INFO: 0/0 source files have been analyzed
07:45:33.421 INFO: Sensor IaC CloudFormation Sensor [iac] (done) | time=25ms
07:45:33.421 INFO: Sensor IaC Kubernetes Sensor [iac]
07:45:33.422 DEBUG: File without Kubernetes identifier: file:///azp/_work/1/s/secret.yml
07:45:33.423 INFO: 1 source file to be analyzed
07:45:33.498 INFO: 1/1 source file has been analyzed
07:45:33.502 INFO: Sensor IaC Kubernetes Sensor [iac] (done) | time=81ms

There were some C# files in this repo but not anymore. Everything is located in the project root, it’s just a small repo to test K8s scanning in SQ.

tree
.
├── Dockerfile
├── azure-pipelines.yml
├── secret.yml
└── vulns.yaml

1 directory, 4 files

Kind regards,
Patryk Jasiński

Marcin_Stachniuk · November 8, 2023, 3:30pm

It is good and expected that the ‘vulns.yaml’ is indexed with language ‘yaml’. The yaml files can be also a CloudFormation file, so there is an artificial ‘yaml’ language.

This line in the log comes from Kubernetes sensor, which is good. At the very beginning in K8s sensor reads the first 8k bytes, and checks if the file contains lines that start with: “apiVersion”, “kind”, “metadata” and “spec”. If yes, the sensor assumes that it is a Kubernetes file and starts analyzing it. The log says that the file doesn’t contain such identifiers, which is not true. I was unable to reproduce your case locally, it works for me (I’m using sonar-scanner instead of Azure tasks). The K8s detection logic, maybe it will help you to find an issue.

The ideas that come to my mind:

Maybe you have something at the beginning of the file, e.g. a long comment, or BiDi character.
Please check the file encodings, non-visible characters in the file, etc.
Try to use sonar-scanner locally to see if the the problem exist. If not then someone else responsible for Tasks in Azure needs to look into this issue.

Best
Marcin Stachniuk