[Tech Story] Takeaways from building a SAST product, and why OWASP benchmark is not enough

Sonar Updates
4

Hello,

Please note now a days in 2023, as part of AppDefenceAlliance CASA Audit, auditors require an official OWASP Benchmark to be submitted beside SAST results. Orientações para o teste de outras ferramentas  |  App Defense Alliance

Using OWASP Benchmark on Fresh version of 9.9LTS Enterprise. The Official OWASP Script seems to be totally broken with JQ errors argument list too long. If we clone and run test only few Results will be picked up (~200 issue i.e. total result json file size is few kbs only). This is evidently wrong and super small comparing to the actual ~18000 vulnerabilities and ~ 1200 Security hotspots recorded on SonarQube dashboard. If using that to generate score card the actual score will be a red Fail score close to 0-3% points.

Got mannaged to modify this script from Official OWASP REPO https://github.com/OWASP-Benchmark/BenchmarkJava/blob/master/scripts/runSonarQube.sh

Could work out collecting All Vulnerabilities and SecurityHotspots and create some what good result json file ~ 16Mb size, which ended up in the following Score:

TPR 69.64% with FPR 23.52% = Overal score of 46.26%

Not sure if Benchmark is broken or SonarQube is not reporting what Benchmark expects ? some wiered numbers on Score (Marked in Red):

SQL Injection: TPR 100% and FPR 100%
Trust Bundary: TPR 0.% and FPR 0%

When executing ./createScorecards.sh Multiple lines of Following repeating Errors happen:

  • SonarQubeReader: Unknown squid number: S5883 has no CWE mapping
  • WARN: Found new SonarQue HotSpot rule not seen before. Category: command-injection with message: “Make sure that this user-controlled command argument doesn’t lead to unwanted behaviour”
  • WARN: Failed to translate SonarQube security category: ‘command-injection’ with message: ‘Make sure that this user-controlled command argument doesn’t lead to unwanted behavior’
  • WARN: Failed to translate SonarQube security category: ‘others’ with message ‘Make sure creating this cookie without the “HTTPOnly” flag is safe.’

I think would be great if SonarSource team could contribute and provide an Official up to date OWASP Based Benchmark that is working for Version 9.9 LTS with a script/guide to run and get reports out.