How to find the duplicated code using SonarQube API

naveenaggarwal1 · May 20, 2020, 4:21am

Template for a good bug report, formatted with Markdown:

versions used (SonarQube Enterprise Edition
Version 8.2 (build 32929))
There are api available to find number of code duplications , number of files have duplication and number of duplicate lines using metrices but there is no way to find what;s that duplicate code and name of files with line number
steps to reproduce - https://url of sonarqube/api/measures/component?component=projectkey-20200319&metricKeys=duplicated_files
potential workaround - Not Known

#Api #codeduplicate

P.S.: use the #bug:fault sub-category if you’re hitting a specific crash/error , or the #bug:fp sub-category for rules-related behaviour

bugbouncer · May 20, 2020, 11:30am

Hi @naveenaggarwal1,
It is a bit tricky to get the invdividual figures from SonarQube.
The overall duplication rate is comparatively easy, when you query the following URL (it should show number of bugs and duplicated lines ratio:
https://<sonar-server-url>/api/measures/component?component=<your-id>&metricKeys=bugs%2Cduplicated_lines_density

For the other measures your should:

Logon to SonarQube with your browser
Open the developer tools (hit <F12> on your keyboard)
Browse to the page in SonarQube and monitor the request in the Developer Tools.

In the picture below, you can see a request, that reveals individual files and their duplication ratio

RSchaeferHIG · May 20, 2020, 11:58am

The issues/search API gives you the capability to search for issues against specific rules. You could search based on the duplicate code rules. The results include a lot of information about the issue including component and line. That might get you right to the duplication.

naveenaggarwal1 · May 20, 2020, 5:11pm

Thanks @bugbouncer for your response. I am more looking from automation perspective and don’t want to login SonarQube UI. I am looking for API support which give all details for code duplication including number of lines, names of files and line numbers which has code duplication with duplicate code block as well. I intend to generate an automated report with this all details.

naveenaggarwal1 · May 20, 2020, 5:12pm

Thanks @RSchaeferHIG for your response. I will try this and see if i can get code duplication file name with duplicated code blocks using search API.

naveenaggarwal1 · May 20, 2020, 8:50pm

Hi @RSchaeferHIG i can’t find the rule for duplicate code rule for issue search API ? Do you mind sharing the API call for issue/search API and all parameters for finding duplicate code files with details. Thanks in advance,

RSchaeferHIG · May 21, 2020, 11:07am

I was mistaken. I thought duplications had a rule associated with them. They do not. I’ll poke at the API to see if I can find a way to get duplication info.

naveenaggarwal1 · May 21, 2020, 10:33pm

Thanks @RSchaeferHIG

bugbouncer · May 22, 2020, 6:41am

Hello @naveenaggarwal1,
I got your point.
My hint was actually: Look how the SonarQube-UI uses the REST-API to retrieve information from the backend.
The SonarQube-UI uses the same REST-API as you intend to use.
So if you see a data item in the UI, you can use my method to find the respective REST call and use it for your automation purposes.

naveenaggarwal1 · May 23, 2020, 7:47pm

Hi @bugbouncer i understand the solution you are suggesting. But i am finding below issues in that solution.

In SonarQube UI i can view the code duplicate metrics as below

For finding the details of duplicate lines i have to click on duplicate lines which will browse the project structure on right hand side for seeing the actual file name with duplicate code i have to click on each and every folder and for each API call for click the URl becomes so complex. replicating same scenario in API automation is very complex. What I was thinking as this information is already there in SonarQube database there should be some direct way of retrieving same using API. I hope you understand how complex it could be.

There are APi calls which give you metrices for code duplication as below code . I am looking similar API for getting details of duplication which has file name and line numbers of duplicate files.

$metricname = “duplicated_files”
$url = $SonarQubeQualityGateURL + “/measures/component?component=$projectkey&metricKeys=” + $metricname
Write-Host $url
$response= Invoke-RestMethod -Uri $url -Headers $headers
Write-Host “Number of Duplicate files=” $response.component.measures | Format-Table

$metricname = “duplicated_lines_density”
$url = $SonarQubeQualityGateURL + “/measures/component?component=$projectkey&metricKeys=” + $metricname
Write-Host $url
$response= Invoke-RestMethod -Uri $url -Headers $headers
Write-Host “Duplicate files Density=” $response.component.measures -ForegroundColor Red

$metricname = “duplicated_lines”
$url = $SonarQubeQualityGateURL + “/measures/component?component=$projectkey&metricKeys=” + $metricname
Write-Host $url
$response= Invoke-RestMethod -Uri $url -Headers $headers
Write-Host “Number of duplicate lines=” $response.component.measures -ForegroundColor Red

Colin · May 25, 2020, 5:57pm

You are probably looking for GET api/duplications/show

naveenaggarwal1 · May 26, 2020, 7:08pm

Thanks @Colin yes dats correct I am looking for same api (SonarQube) but I am not getting any output when i am hitting this api but there are duplication in my project which I am able to see using other metrices api. Am i missing anything ?

https://mycompanyinstance/api/duplications/show?key=Projectkey

Output is always blank as below

Number of duplicate blocks= @{duplications=System.Object[]; files=}

duplication object and number of files is always blank.

Colin · May 26, 2020, 8:06pm

You’ll need to provide a specific file.

For example:

// 20200526220438
// http://localhost:9000/api/duplications/show?key=org.sonarsource.php%3Aphp%3Aphp-frontend%2Fsrc%2Fmain%2Fjava%2Forg%2Fsonar%2Fphp%2Ftree%2Fimpl%2Fexpression%2FLiteralTreeImpl.java

{
  "duplications": [
    {
      "blocks": [
        {
          "from": 39,
          "size": 25,
          "_ref": "1"
        },
        {
          "from": 38,
          "size": 25,
          "_ref": "2"
        }
      ]
    }
  ],
  "files": {
    "1": {
      "key": "org.sonarsource.php:php:php-frontend/src/main/java/org/sonar/php/tree/impl/expression/LiteralTreeImpl.java",
      "name": "php-frontend/src/main/java/org/sonar/php/tree/impl/expression/LiteralTreeImpl.java",
      "uuid": "AXJSV6bHmYM8o0DG8TZc",
      "project": "org.sonarsource.php:php",
      "projectUuid": "AXJSV5XtWfZtOXuuH8nd",
      "projectName": "SonarPHP"
    },
    "2": {
      "key": "org.sonarsource.php:php:php-frontend/src/main/java/org/sonar/php/tree/impl/expression/ExpandableStringCharactersTreeImpl.java",
      "name": "php-frontend/src/main/java/org/sonar/php/tree/impl/expression/ExpandableStringCharactersTreeImpl.java",
      "uuid": "AXJSV6bHmYM8o0DG8TZI",
      "project": "org.sonarsource.php:php",
      "projectUuid": "AXJSV5XtWfZtOXuuH8nd",
      "projectName": "SonarPHP"
    }
  }
}

It might also be useful to know the Internal Web API GET api/sources/lines provides duplication data per line as well.

naveenaggarwal1 · May 26, 2020, 8:30pm

Thanks @Colin my scenario is different I need to find all files for a given project which has duplicate code . I want to create a report using automation in Powershell. I don’t have name of files to pass as parameter to this api. I want to get all information using api automation without any manual step.