SonarScanner fails with duplicate key exception in the BlameGenerator

ChristoWolf · September 27, 2023, 10:40am

Hi!

I am experiencing some confusing issue with a SonarQube instance that I am using.
The Azure DevOps Server error log can be expanded at the bottom.

The issue seems to be some kind of dangling reference during code analysis where the SonarScanner (in particular its BlameGenerator) wants to merge blocks for a file which hasn’t existed since 2 years now.
Also, the error mentions another, completely unrelated file.
This makes the task fail without any (to me) obvious reason/solution.

Further, I am not sure what the BlameGenerator means by source path and original path, as those files in my case are unrelated.

Versions:
- SonarQube 10.2.0.77647
- SonarScanner 4.8.0.2856 (SonarScanner for MSBuild 5.13)
- Oracle JDK 17.0.8 (64-bit)
- SonarQube extension for Azure DevOps 5.15.0
How is SonarQube deployed: unsure, on K8s
I have tried various things to work around this:
- Excluded both files from analysis. Works, but this is not an option though, as this also excludes the unrelated, existing file.
- Excluded the non-existent file only. The issue remained the same.
- Added the non-existent file again and modified the existing one. Also did not work.
- I checked all retained, analyzed branches and PRs to see if any of them contains the non-existent files. None of them does, the file has been (git) removed about 2 years ago.

Would appreciate any help!
Thanks

Expand for build error log...

2023-09-26T13:47:37.3704165Z INFO: EXECUTION FAILURE
2023-09-26T13:47:37.3707079Z INFO: ------------------------------------------------------------------------
2023-09-26T13:47:37.3708572Z INFO: Total time: 4:13.192s
2023-09-26T13:47:37.5508566Z INFO: Final Memory: 63M/214M
2023-09-26T13:47:37.5511157Z INFO: ------------------------------------------------------------------------
2023-09-26T13:47:37.5515605Z ##[error]ERROR: Error during SonarScanner execution
2023-09-26T13:47:37.5518448Z ERROR: Error during SonarScanner execution
2023-09-26T13:47:37.5579817Z ##[error]java.lang.IllegalStateException: Duplicate key org.sonar.scm.git.blame.BlameGenerator$PathAndOriginalPath@d4acdd17 (attempted merging values Candidate[source path: <non-existent file>.cs, original path: <completely unrelated file>.cs, regions:0-5] and Candidate[source path: <non-existent file>.cs, original path: <completely unrelated file>.cs, regions:8-9])
2023-09-26T13:47:37.5584460Z java.lang.IllegalStateException: Duplicate key org.sonar.scm.git.blame.BlameGenerator$PathAndOriginalPath@d4acdd17 (attempted merging values Candidate[source path: <non-existent file>.cs, original path: <completely unrelated file>.cs, regions:0-5] and Candidate[source path: <non-existent file>.cs, original path: <completely unrelated file>.cs, regions:8-9])
2023-09-26T13:47:37.5824352Z ##[error]at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
2023-09-26T13:47:37.5830172Z 	at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135)
2023-09-26T13:47:37.5832743Z ##[error]at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
2023-09-26T13:47:37.5839691Z 	at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:182)
2023-09-26T13:47:37.5866828Z ##[error]at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
2023-09-26T13:47:37.5868519Z 	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
2023-09-26T13:47:37.5870920Z ##[error]at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
2023-09-26T13:47:37.5872645Z 	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
2023-09-26T13:47:37.5874359Z ##[error]at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
2023-09-26T13:47:37.5875885Z 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
2023-09-26T13:47:37.5877547Z ##[error]at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
2023-09-26T13:47:37.5879105Z 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
2023-09-26T13:47:37.5880766Z ##[error]at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
2023-09-26T13:47:37.5882316Z 	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
2023-09-26T13:47:37.5883938Z ##[error]at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2023-09-26T13:47:37.5885440Z 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
2023-09-26T13:47:37.5887042Z ##[error]at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
2023-09-26T13:47:37.5888570Z 	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
2023-09-26T13:47:37.5890448Z ##[error]at org.sonar.scm.git.blame.BlameGenerator.push(BlameGenerator.java:102)
2023-09-26T13:47:37.5892101Z 	at org.sonar.scm.git.blame.BlameGenerator.push(BlameGenerator.java:102)
2023-09-26T13:47:37.5893659Z ##[error]at org.sonar.scm.git.blame.BlameGenerator.process(BlameGenerator.java:154)
2023-09-26T13:47:37.5895133Z 	at org.sonar.scm.git.blame.BlameGenerator.process(BlameGenerator.java:154)
2023-09-26T13:47:37.5896720Z ##[error]at org.sonar.scm.git.blame.BlameGenerator.generateBlame(BlameGenerator.java:127)
2023-09-26T13:47:37.5898231Z 	at org.sonar.scm.git.blame.BlameGenerator.generateBlame(BlameGenerator.java:127)
2023-09-26T13:47:37.5899864Z ##[error]at org.sonar.scm.git.blame.RepositoryBlameCommand.call(RepositoryBlameCommand.java:123)
2023-09-26T13:47:37.5901425Z 	at org.sonar.scm.git.blame.RepositoryBlameCommand.call(RepositoryBlameCommand.java:123)
2023-09-26T13:47:37.5903206Z ##[error]at org.sonar.scm.git.CompositeBlameCommand.blameWithFilesGitCommand(CompositeBlameCommand.java:199)
2023-09-26T13:47:37.5904855Z 	at org.sonar.scm.git.CompositeBlameCommand.blameWithFilesGitCommand(CompositeBlameCommand.java:199)
2023-09-26T13:47:37.5906526Z ##[error]at org.sonar.scm.git.CompositeBlameCommand.blame(CompositeBlameCommand.java:97)
2023-09-26T13:47:37.5908028Z 	at org.sonar.scm.git.CompositeBlameCommand.blame(CompositeBlameCommand.java:97)
2023-09-26T13:47:37.5909526Z ##[error]at org.sonar.scanner.scm.ScmPublisher.publish(ScmPublisher.java:87)
2023-09-26T13:47:37.5910944Z 	at org.sonar.scanner.scm.ScmPublisher.publish(ScmPublisher.java:87)
2023-09-26T13:47:37.5912659Z ##[error]at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:368)
2023-09-26T13:47:37.5914316Z 	at org.sonar.scanner.scan.SpringProjectScanContainer.doAfterStart(SpringProjectScanContainer.java:368)
2023-09-26T13:47:37.5916530Z ##[error]at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
2023-09-26T13:47:37.5918591Z 	at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
2023-09-26T13:47:37.5920552Z ##[error]at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
2023-09-26T13:47:37.5922197Z 	at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
2023-09-26T13:47:37.5924262Z ##[error]at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:139)
2023-09-26T13:47:37.5925918Z 	at org.sonar.scanner.bootstrap.SpringGlobalContainer.doAfterStart(SpringGlobalContainer.java:139)
2023-09-26T13:47:37.5927726Z ##[error]at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
2023-09-26T13:47:37.5929400Z 	at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
2023-09-26T13:47:37.5938919Z ##[error]at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
	at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:71)
	at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:65)
	at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
	at jdk.proxy1/jdk.proxy1.$Proxy0.execute(Unknown Source)
	at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
	at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
	at org.sonarsource.scanner.cli.Main.execute(Main.java:126)
	at org.sonarsource.scanner.cli.Main.execute(Main.java:81)
	at org.sonarsource.scanner.cli.Main.main(Main.java:62)
ERROR:
2023-09-26T13:47:37.5944451Z 	at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
2023-09-26T13:47:37.5945360Z 	at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:71)
2023-09-26T13:47:37.5946119Z 	at org.sonar.batch.bootstrapper.Batch.execute(Batch.java:65)
2023-09-26T13:47:37.5947057Z 	at org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
2023-09-26T13:47:37.5948019Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-09-26T13:47:37.5949177Z 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
2023-09-26T13:47:37.5950252Z 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-09-26T13:47:37.5951186Z 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
2023-09-26T13:47:37.5952083Z 	at org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
2023-09-26T13:47:37.5952931Z 	at jdk.proxy1/jdk.proxy1.$Proxy0.execute(Unknown Source)
2023-09-26T13:47:37.5953704Z 	at org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
2023-09-26T13:47:37.5954623Z 	at org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
2023-09-26T13:47:37.5955439Z 	at org.sonarsource.scanner.cli.Main.execute(Main.java:126)
2023-09-26T13:47:37.5956155Z 	at org.sonarsource.scanner.cli.Main.execute(Main.java:81)
2023-09-26T13:47:37.5956875Z 	at org.sonarsource.scanner.cli.Main.main(Main.java:62)
2023-09-26T13:47:37.5957405Z ERROR: 
2023-09-26T13:47:38.3822708Z ##[error]The SonarScanner did not complete successfully
2023-09-26T13:47:38.3836302Z The SonarScanner did not complete successfully
2023-09-26T13:47:38.3876734Z ##[error]15:47:38.368  Post-processing failed. Exit code: 1
2023-09-26T13:47:38.3880946Z 15:47:38.368  Post-processing failed. Exit code: 1

Wouter_Admiraal · October 6, 2023, 1:54pm

Hi @ChristoWolf, and welcome to the community!

Thanks for the detailed explanation.

To help us better understand the problem, could there be any “relationship” between those files in the history? Could the existent file (let’s call it “FileA.cs”) have a name that has existed in the past, but for a completely different reason? Say, 3 years ago, you had “FileA.cs”, which was renamed to “FileB.cs”. And then “FileB.cs” was deleted 2 years ago. And now, recently, you introduced a new file, which happens to be called “FileA.cs” as well (even if it has no relationship to the old “FileA.cs”). Am I making sense? I have a hard time putting this into words, to be honest .

ChristoWolf · October 9, 2023, 5:04am

Hi @Wouter_Admiraal!

Thanks for the response.

I think I understand what you are getting at, but unfortunately no, the files do have completely different names/paths.
Thing is that I even tried to fiddle around with git to check if there is anything weird about the files’ histories, and removing/re-adding the affected files (all such combinations) did not help either.

Wouter_Admiraal · November 2, 2023, 9:48am

Hi @ChristoWolf ,

Sorry for the late reply. We’re still scratching our heads over this. We tried multiple things but could not reproduce the error.

Do you still have the problem? If not, how did you solve it? If yes, do you have a repo we could look at?

ChristoWolf · November 3, 2023, 6:24am

Hi @Wouter_Admiraal!

No worries, I’m happy about any help!

The issue is still there, but I worked around it by excluding both those files (even the non-existent one) from SonarQube analysis.

Unfortunately, I cannot share any such information due to policies, sorry about that
I know, this makes it a very tough nut to crack.
If I remember correctly, there were actually some weird things in git regarding the non-existent file (it was moved at some point and later on deleted, which is fine, but one of our repo browsers had some issue visualizing that).
So I guess there might be something wrong with the repo which I could not fix.

In any case, I think it would still be useful to take a look at the BlameGenerator why it would react to (possibly) dangling references like that, and why it would try to target to completely different files during the same step.
Also, just for my understanding, what does it mean by source path and original path?

Thanks!

Attila_Heidrich · April 15, 2024, 11:20am

It looks like a rare issue, but it occurs to us too
We looked for any anomaly in the git history, but without success.

Caused by: java.lang.IllegalStateException: Duplicate key org.sonar.scm.git.blame.BlameGenerator$PathAndOriginalPath@ca7e0ff2 (attempted merging values Candidate[source path: smartvision/src/main/java/com/sagemcom/smartvillage/smartvision/controller/VendingController.java, original path: smartvision/src/main/java/com/sagemcom/smartvillage/smartvision/controller/VendingSystemController.java, regions:9-11,54-55,226-227] and Candidate[source path: smartvision/src/main/java/com/sagemcom/smartvillage/smartvision/controller/VendingController.java, original path: smartvision/src/main/java/com/sagemcom/smartvillage/smartvision/controller/VendingSystemController.java, regions:69-70,89-90,91-92,110-112])

sonarqube version is 10.4, deployed locally with docker
gradle is used for the build, the sonarqube plugin version is 5.0.0.4638

ChristoWolf · April 16, 2024, 5:02am

Hi @Attila_Heidrich!

When you exclude those affected files from analysis, does it work again then?
It’s just a work-around of course, but might help for now.
Good to know however that I am not the only one experiencing this issue.

Lucas_Paulger · April 26, 2024, 9:31am

Hello @Attila_Heidrich and @ChristoWolf,
Thanks for raising this issue. We’ll take your findings into consideration and will try to reproduce the issue on our end to investigate what could be the root cause.

Please follow this thread for any updates!

Thank you,
Lucas