Help with `Fail to resolve 159 file(s)`

[stdedos]

Must-share information (formatted with Markdown):

• which versions are you using (SonarQube, Scanner, Plugin, and any relevant extension)
• how is SonarQube deployed: zip, Docker, Helm
• what are you trying to achieve
• what have you tried so far to achieve this

Do not share screenshots of logs – share the text itself (bonus points for being well-formatted)!

9.9 LTS: I am trying to add rubocop issues to SonarQube.

I have added the

        properties["sonar.ruby.rubocop.reportPaths"] = file(project.rootDir).resolve("rubocop-report.json")

line, and during pushing those files, the report file exists - also the sources:

e[32;1m$ ls -lahe[0;m
total 732K
drwxrwxrwx 18 root root 4.0K Jun 28 16:28 .
drwxrwxrwx  4 root root 4.0K Jun 28 16:26 ..
-rw-rw-rw-  1 root root    4 Jun 28 16:27 .dockerignore
drwxrwxrwx  7 root root 4.0K Jun 28 16:27 .git
...
drwxrwxrwx 27 root root 4.0K Jun 28 16:28 backend
...
-rw-r--r--  1 root root 350K Jun 28 16:06 rubocop-report.json

(I hope I don’t need to do ls -lahR to prove my point).

I have checked the rubocop-report.json, and the files are reported relative to the above pwd; also the sonar task is running at the same pwd.

{"metadata":{"rubocop_version":"1.64.1","ruby_engine":"jruby","ruby_version":"3.1.4","ruby_patchlevel":"0","ruby_platform":"java"},"files":[{"path":"backend/.irbrc","offenses":[{"severity":"warning","message":...

However, still I get:

Fail to resolve 159 file(s). No RuboCop issues will be imported on the following file(s): backend/.irbrc;.....

What am I missing here?

Could it be that ./gradlew sonar -Dsonar.exclusions="backend/**/*" is affecting that resolution?

If not, what is it?

I assume that the “juice” is slang/sonar-ruby-plugin/src/main/java/org/sonarsource/ruby/externalreport/rubocop/RuboCopSensor.java at b3bb437b7c8d713e82fe23c26d99c0af8519fc48 · SonarSource/slang · GitHub, but debugging CI (and a foreign program) is always hard.

[Colin]

Hey there.

@stdedos

Could it be that ./gradlew sonar -Dsonar.exclusions="backend/**/*" is affecting that resolution?

Well, if all your backend/**/* files are excluded, and all your rubocop issues are on files in that directory… sure sounds like a conflict to me. Why are you trying to exclude all those files?

[stdedos]

@Colin

Why are you trying to exclude all those files?

Pipeline optimization. It’s not happening always of course, but it is a “possible” code path.

However

@stdedos

Fail to resolve 159 file(s)

That could sound different to my ears. Does Fail to resolve mean “files not found”, or does it mean “After processing your exclusions, I cannot match ‘these’ files to the files you asked me to
‘lint’?”

[Colin]

If files are excluded from analysis, they will not be able to be matched to issues reported by external issue reports.

[stdedos]

I specifically asked you “Is this exactly what this message means?”. You are giving me the “Community Manager” response

I have verified that, even after removing those excludes, that thing still happens.

Can we please now get “some more documentation” on how this feature is supposed to work? Or “get someone on the line” familiar with the feature(s) involved?

[Colin]

Believe it or not, I’m pretty familiar with the features involved!

When I recreate the exact same scenario in my environment, I get the “fail to resolve” message when I exclude the files included in the report, and the message goes away when I remove the exclusion.

@stdedos

Fail to resolve 159 file(s). No RuboCop issues will be imported on the following file(s): backend/.irbrc;.....

One thing to note here is that hidden files are never indexed by the scanner. That being said, I’d be surprised if all 159 files in your report are hidden files. Unless that’s the case, has the number shifted at all when removing the exclusion?

- Colin
Community Manager

[Colin]

And just to make sure you get your question answered:

@stdedos

Does Fail to resolve mean “files not found”, or does it mean “After processing your exclusions, I cannot match ‘these’ files to the files you asked me to
‘lint’?”

It could mean either.

[stdedos]

@Colin

That being said, I’d be surprised if all 159 files in your report are hidden files.

They are not. I truncated it for brevity and/or anonymization reasons.

@Colin

has the number shifted at all when removing the exclusion?

Actually, the number has increased It’s not a small repo (nor a small pipeline), so I cannot “quickly” ascertain “why” that happened - but I treat it as “expected” (159 then === 164 now)

[Colin]

Let me double-check if the Rubocop sensor is run once per module or once per project… I have a sinking feeling that using the Gradle scanner might be getting in the way here. That would be one element of your setup I didn’t recreate.

Are there proper JVM based languages in this project (Java, Kotlin)? Just want to make sure there’s a valid reason for using the SonarScanner for Gradle as opposed to the SonarScanner CLI.

[Colin]

Ah I think I know what’s going on.

If you’re using the Scanner for Gradle, exclusions or not, the backend folder probably isn’t being indexed properly. You need to add it as a proper sourceset, at which point the files will get indexed, and you can attach external issues. If this seems a bit convoluted, we know, and our first stop was the Maven scanner to improve the default behavior.

You can confirm right now if the backend folder is being indexed by just checking the Code tab of your project (following an analysis where you’ve removed the backend folder from your exclusions).

All of this mostly unnecessary if you aren’t actually using any JVM-based languages, and can switch to the SonarScanner CLI.

@Colin

Are there proper JVM based languages in this project (Java, Kotlin)? Just want to make sure there’s a valid reason for using the SonarScanner for Gradle as opposed to the SonarScanner CLI.

[stdedos]

My Gradle-fu is non-existent, but it seems that this is done?

and just to be on the safe side:

Fail to resolve 164 file(s). No RuboCop issues will be imported on the following file(s): backend/.irbrc;backend/Gemfile;backend/app/x/y.rb;backend/app/z/a.rb;...

Are there proper JVM based languages in this project (Java, Kotlin)?

Java, Kotlin, JRuby, TS, Scala … it’s a huge monorepo (orchestrated by Gradle)

[Colin]

Maybe it’s better to check on this side:

@Colin

You can confirm right now if the backend folder is being indexed by just checking the Code tab of your project (following an analysis where you’ve removed the backend folder from your exclusions).

[stdedos]

The pipeline is currently red, so I cannot do it

I’m keeping it in mind

[stdedos]

@Colin

You can confirm right now if the backend folder is being indexed by just checking the Code tab of your project (following an analysis where you’ve removed the backend folder from your exclusions).

It seems that you are right. The files are not indexed. I guess that would need follow up from gradle people, then

Is there any way do dump the “final” configuration of the sonar task? All of those properties “may” be nice - but I don’t trust where ends up what

---

Also, I’m receiving a bunch of

File '/a/b/repo/backend/spec/a.rb' is present in coverage report but cannot be found in filesystem

errors - but they could not be further from the truth.

(ls -lah proves them to be there. This MAY or MAY NOT be related with the issue at hand.)

[Colin]

@stdedos

Is there any way do dump the “final” configuration of the sonar task? All of those properties “may” be nice - but I don’t trust where ends up what

I think you’re looking for sonar.scanner.dumpToFile!

@stdedos

Also, I’m receiving a bunch of

File '/a/b/repo/backend/spec/a.rb' is present in coverage report but cannot be found in filesystem

errors - but they could not be further from the truth.

(ls -lah proves them to be there. This MAY or MAY NOT be related with the issue at hand.)

I think ultimately the fix is the same – making sure that the files get indexed.

[stdedos]

Project(s) look “weird”, but well-defined to me. Anything that includes /backend/:

Project server settings:
  - sonar.exclusions=backend/spec/**/*,...
  - sonar.javascript.file.suffixes=.js,.jsx,.mjs,.vue
Project scanner properties:
  - sonar.coverage.exclusions=...,backend/spec/**/*,...,backend/app/renderers/**/*,...
  - sonar.modules=:...,:backend,:...,:...
  - sonar.projectBaseDir=/b/d/pc
  - sonar.projectDescription=pc
  - sonar.projectName=pc
  - sonar.ruby.rubocop.reportPaths=/b/d/pc/rubocop-report.json
  - sonar.scanner.app=ScannerGradle
  - sonar.scanner.appVersion=5.0.0.4638/Gradle 8.7
  - sonar.sourceEncoding=UTF-8
  - sonar.sources=/b/d/pc/dip/src/main/kotlin,/b/d/pc/buildSrc/src/main/kotlin
  - sonar.tests=/b/d/pc/dip/src/test/kotlin,/b/d/pc/buildSrc/src/test/kotlin
  - sonar.working.directory=/b/d/pc/build/sonar
Scanner properties of module: ...:backend
  - sonar.java.libraries=/b/d/pc/.gradle/caches/...
  - sonar.java.test.libraries=/b/d/pc/.gradle/caches/...
  - sonar.libraries=/b/d/pc/.gradle/caches/...
  - sonar.moduleKey=...:backend
  - sonar.projectBaseDir=/b/d/pc/backend
  - sonar.projectKey=...:backend
  - sonar.projectName=backend
  - sonar.projectVersion=10.4
  - sonar.ruby.coverage.reportPaths=/b/d/pc/backend/coverage_rspec-tests 3/13/coverage.json,/b/d/pc/backend/coverage_rspec-tests 12/13/coverage.json,...
  - sonar.sources=app,spec
...
Close

(I’d be nice if one was able to actually free-text filter /project/background_tasks?, instead of having to play around with the date - or find the “settings” of that scan from within /dashboard :/)

[Colin]

It looks like you still have all the backend/spec files excluded.

@stdedos

  - sonar.exclusions=backend/spec/**/*,...

So that would at least explain some of the coverage-related logs. Is that intentional? Any other exclusions? (I see the ...)

[stdedos]

@Colin

It looks like you still have all the backend/spec files excluded.

“Intentional”. (idk if really or not, but idc at this point about /spec/s)

@Colin

Any other exclusions? (I see the ...)

Non-pertinent to our discussion. “Anything that includes /backend/” is there already. The rest is shortened / sanitized.

[stdedos]

Here’s my build engineering’s response:

The way I understand you are complaining why the whole codebase is not being scanned within MRs and it’s answer lies within Pull request analysis & SonarQube which is intentional and by design to speed up the scan.

So I believe you should go with Branch analysis & SonarQube e.g. master to handle your analysis rather than MR.

… I don’t understand much more beyond “face value”, so - any ideas?

---

Idk how pertinent it is to the above, but let me re-iterate my targets:

1. Rubocop has a .rubocop_todo.yml file created - i.e., as of now, the repository is clean
2. Changes usually get merged via MR
3. It is imperative that:
   a. Rubocop generating issues means that this will be reflected to the SonarQube Code Analysis\n## Quality Gate passed ... message
   b. When manipulating .rubocop_todo.yml “wrongly” - which means that issues are generated, triggers the above condition (3a) - regardless if the files-in-question are modified in said MR or not

[stdedos]

@Colin? Any ideas?