False negative rule S6418

Rules and Languages Report False-positive / False-negative...
, , ,
1

Dear all, I have detected what I think is a false negative in rule S6418 “Secrets should not be hard-coded” and/or S2068 “Credentials should not be hard coded”.

Problem: Hardcoded password SuperSecretABCD1234!? not detected.

Context: I have run the scanner sonar-scanner-cli 8.1 against SonarQube Cloud Enterprise. The scanner logs show that the file with the password is indexed both in ansible and yaml sensors, the project is using SonarWay profiles in all languages, in which bot rules are activated. The file is analyzed for sure, because other issues of ansible have arisen.

Evidences:

Starting point: The file does not detect a hardcoded password. No issue is arisen nor credential nor secret.

vars:
app_db_password: "SuperSecretABCD1234!?"
tasks:

First change: Change the password and use some random string without the word “secret”. I get then an issue of the rule S2068 “password detected here” of the yaml language. This shows that yaml profile is being correctly applied.

vars:
app_db_password: "fawñeohkap93023Añgae`w3QR"
tasks:

Second change: Add the word “secret” as suffix of the random string. I get nothing again.

vars:
app_db_password: "secret_fawñeohkap93023Añgae`w3QR"
tasks:

I can understand that if the password has the word “secret” or “token” it is likely a test or demo password that maybe it is not a real secret. I do not know if this false negative is intentional (not a real false negative) because of this.

However, I think some issue should tell the developer that this password is hardcoded and cannot be written there. I do not really know which of those rules should arise (I will be inclined by S2068, since it is not a secret) but any of them would be necessary, whouldn`t be?

Best regards and thank you.

4

Hello Maria,

Thank you very much for your report!

To give you some context, we intentionally don’t raise issues on the two negative examples you provided. We’ve built filters to prevent raising alerts on fake secrets in code, which helps save developers from annoying false positives. You can see a part of this filtering logic here: SecretClassifier.java

For rules implemented in our Secrets analyzer, we provide analysis options to disable these filters if you prefer:

  • sonar.secrets.disableEntropyFilter
  • sonar.secrets.disableKnownFakeSecretFilter

You can read more about these in our documentation: Analyzing Source Code / Secrets

That said, your post did uncover a gap: In your specific case, the applicable rule to raise is the S2068 version implemented in our YAML analyzer. Because this rule is implemented differently than the core Secrets rules, it currently does not respect the override flags mentioned above.

I have created a ticket to align this behavior in the future so that S2068 for YAML and all the other languages will respect these parameters.

Thanks again for taking the time to report this and helping us improve!

Best regards,
Daniel

5

Dear Daniel,

Thanks a lot for the response and the explanation. I am convinced about no raising issues that are false positives.

However, what do you think I should tell a developer that is comparing sonar with another tool (specifically checkov) and says that Sonar detects less :slight_smile: specifically this hardcoded credential?

Best regards.

6

Hi Maria,

I am glad you see the value in this approach! This is actually a very common question we get from users when they are benchmarking our product against others.

When explaining this to others, I would suggest highlighting that filtering out obviously fake values is actually a mark of quality as it prevents alert fatigue and saves developers from spending time on false positives.

If the goal is to test the contexts in which we detect secrets, we suggest to use realistic or randomly generated secrets rather than generic placeholders like password123 that would never be used as a real password.
Additionally, rather than just comparing the raw number of raised issues, look into whether the raised issues actually make sense and if they would realistically appear in production code or be seen as a false positive by a developer.

We are actually planning to introduce a specific flag that disables all fake value checks and automatic test-file exclusions. This will be helpful for users who want to run benchmarks across all files to see absolutely every potential flag. While this feature isn’t implemented just yet, it should make these kinds of tool comparisons much easier in the future.

Best,

Daniel