Sonarqube logs shipping to Logstash using elastic filebeat sidecar container

[Sumit_Singh1 (Sumit Singh)]

Must-share information (formatted with Markdown):

• which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension) Sonarqube 9.5

• what are you trying to achieve
  I’m trying to send log to Logstash using filebeat. I have installed Sonarube enterprise in our AKS cluster using the helm chart. It is working fine. I want to ship the sonarqube log to logstash by sidecar logging method. I run filebeat side container

• what have you tried so far to achieve this
  I have created filebeat Configmap and a filebeat sidecar container along with Sonarqube container.
  These are my configuration:
  ----------------------------------- values.yaml --------------------------------
  sonarqube:
  edition: enterprise

  postgresql:
  enabled: false

  jdbcOverwrite:
  enable: true
  jdbcUsername: “username”
  jdbcSecretName: “secretname”
  jdbcSecretPasswordKey: “password”
  jdbcUrl: “jdbc:postgresql://servername:5432”

  ingress:
  enabled: true
  ingressClassName: nginx
  tls:

  • secretName: soanrqube-certificate
    hosts:
    • sonarqube-dev.xyz.com
      hosts:
  • name: sonarqube-dev.xyz.com
    extraContainers:
  • name: filebeat-sonar
    image: docker.elastic.co/beats/filebeat:8.3.3
    imagePullPolicy: Always
    volumeMounts:
    • name: sonarqube
      mountPath: /opt/sonarqube/logs
      subPath: logs
    • name: filebeat-config
      mountPath: /etc/filebeat.yml
      subPath: filebeat.yml
      readOnly: true
      persistence:
      volumes:
    • name: filebeat-config
      configMap:
      name: filebeat-configmap
      defaultMode: 0600
      --------------------------------------------------Filebeat-configmap.yaml-------------------------
      apiVersion: v1
      kind: ConfigMap
      metadata:
      name: filebeat-configmap
      data:
      filebeat.yml: |-
      filebeat.inputs:
    • type: container
      enabled: true
      paths:
      • /opt/sonarqube/logs/*
        output.logstash:
        hosts: [“logstashhost:5044”]

---

Both container are running fine but no logs is being send to logstash Kibana portal. When I check the log of filebeat container, it reading the filename file.name":"log/log.go

{“log.level”:“info”,“@timestamp”:“2022-08-29T09:56:32.283Z”,“log.logger”:“monitoring”,“log.origin”:{“file.name”:“log/log.go”,“file.line”:185},“message”:“Non-zero metrics in the last 30s”,“service.name”:“filebeat”,“monitoring”:{“metrics”:{“beat”:{“cgroup”:{“cpuacct”:{“total”:{“ns”:3525864}}},“cpu”:{“system”:{“ticks”:37090},“total”:{“ticks”:62800,“value”:0},“user”:{“ticks”:25710}},“handles”:{“limit”:{“hard”:1048576,“soft”:1048576},“open”:9},“info”:{“ephemeral_id”:“15912e26-403a-4e34-9d96-cf49a4b73c14”,“uptime”:{“ms”:439140064},“version”:“8.3.3”},“memstats”:{“gc_next”:18343656,“memory_alloc”:9595456,“memory_total”:4065261872,“rss”:106061824},“runtime”:{“goroutines”:27}},“filebeat”:{“harvester”:{“open_files”:0,“running”:0}},“libbeat”:{“config”:{“module”:{“running”:0}},“output”:{“events”:{“active”:0}},“pipeline”:{“clients”:0,“events”:{“active”:0}}},“registrar”:{“states”:{“current”:0}},“system”:{“load”:{“1”:0.08,“15”:0.09,“5”:0.11,“norm”:{“1”:0.01,“15”:0.0113,“5”:0.0138}}}},“ecs.version”:“1.6.0”}}

Please suggest.

[jeremy.cotineau]

Hello @Sumit_Singh1 thanks a lot for your post.

From my perspective your configuration looks really fine except for the type: container on the filebeat config.

The container type means that filebeat expect to find logs that are written by the CRI ( mostly docker daemon ) on /var/logs/containers/* on a specific log format ( mostly json )

Regarding your log/log.go, filebeat is written in go so its probably its own log formated in a json format.

I have not tried reproducing the issue but could you try this and get back to us ? :

• exec into the sidecar container and check if the logs are correctly written there and are readable.
• if that’s okay, change to type: log

Apart from that what you want to achieve seems a valid use case, and should work without issues.

Best regards,

[Sumit_Singh1 (Sumit Singh)]

Hi @jeremy.cotineau

Thanks for your time to look into my issue.

Yes, I changed type: filestream and it start sending the logs.

We can mark this post as done.

Best Regards
Sumit