New project creation + v2 APIs

[JackWhelpton]

Hi all,

Is there any available documentation on the current best practice for automating SonarQube Cloud project creation? I’ve inherited some code that’s doing this for us today, but it has some problems.

Specifically, we are using GitLab as an ALM, which for some time now has created a default branch called main. We’re invoking the api/projects/create endpoint:

https://sonarcloud.io/web_api/api/projects?query=create&deprecated=false

This by default creates a branch called master, which becomes the MAIN BRANCH in Sonar; when main is published, this is then seen as a short-lived branch. I don’t see any way to specify the branch name in this call. It also lacks the ALM integration that would enable PR decoration etc.

In investigating alternatives, I was drawn to the ALM endpoints, although they seem to be a) SonarQube specific(?) and b) deprecated:

https://next.sonarqube.com/sonarqube/web_api/api/alm_integrations/import_gitlab_project?deprecated=true

and this in turn took me to what appears to be the replacement, and may well be common across SonarQube and SonarCloud:

https://sonarcloud.io/web_api/new_api/dop-translation?deprecated=false

Is this a fruitful avenue of exploration? Is there any documentation around how to authenticate with and use these new APIs?

UPDATE: Found some documentation at SonarQube Web API, and one of my first problems seems to be a result of using Basic auth, where the new APIs rely on a Bearer token.

[Colin]

Hey @JackWhelpton

There are lots of differences today between SonarQube Cloud and SonarQube Server, including but not limited to how projects are bound to the DevOps Platforms, how branches behave… in a dream world (that we’re working towards, but it takes time), we’ll achieve some better consistency between the two implementations.

Today, While the dop-translation domain of web services exists in both SonarQube Server and SonarQube Cloud, you’ll notice that the Web APIs underneath those domains are completely different (POST /api/v2/dop-translation/bound-projects for SonarQube Server and POST dop-translation/project-bindings for SonarQube Cloud)

If your team is working with SonarQube Cloud (I believe you are?)

@JackWhelpton

Is there any available documentation on the current best practice for automating SonarQube Cloud project creation?

I encourage you to work specifically with the SonarQube Cloud APIs and not pay too much attention to the SonarQube Server Web API documentation. The inverse is also true if your team is working with SonarQube Server.

[JackWhelpton]

Thanks @Colin. It seems like I might need to mix and match the SonarQube Cloud APIs for now: some v1 calls using sonarcloud.io/api, and some v2 ones using api.sonarcloud.io.

I’m working on a pipeline intended to configure a new Sonar project with the correct default branch and repo binding, using the Gradle plugin - this is what I have so far, which seems overly complex, so was hoping I could get a quick sanity check on whether this is the recommended path.

I then have a follow-up question specifically on the Gradle plugin, as to whether it can automatically create a Sonar project: this seems to be happening when I run in a docker container locally, but the GitLab CI pipeline is behaving differently/failing: happy to take that to a new thread once we’ve got over the first piece, if that’s preferable.

./gradlew clean build test sonar \
    -Dsonar.organization=example_org \
    -Dsonar.qualitygate.wait=true \
    -Dsonar.working.directory=.sonarwork

set -o allexport && . .sonarwork/report-task.txt && set +o allexport

sonar_project_id=$(curl -sSH "Authorization: Bearer $SONAR_TOKEN" \
    "https://sonarcloud.io/api/navigation/component?component=${projectKey}&organization=${organization}" |
    jq -r '.id')

curl -sSH "Authorization: Bearer $SONAR_TOKEN" \
    'https://api.sonarcloud.io/dop-translation/project-bindings' \
    -d "{\"projectId\":\"${sonar_project_id}\",\"repositoryId\":\"${CI_PROJECT_ID}\"}"

curl -sSH "Authorization: Bearer $SONAR_TOKEN" \
    'https://sonarcloud.io/api/project_branches/rename' \
    -d "project=${projectKey}&name=${CI_DEFAULT_BRANCH}"

Does this look sane? Is there a simpler way to achieve the same result?

[JackWhelpton]

… and here’s a bit more debugging info for the SonarQube Cloud project auto-creation from Gradle problem; can move this into another thread if we do split the two issues. When I execute in a GitLab pipeline, which isn’t working (not automatically creating a project on the Sonar side), I see:

[INFO] [org.sonarsource.scanner.lib.internal.ScannerEngineLauncher] Found an active CI vendor: 'Gitlab CI'

as opposed to, locally:

[DEBUG] [org.sonarsource.scanner.lib.internal.ScannerEngineLauncher] Could not detect any CI vendor

The next deviation is a result of the pipeline executing in an MR context:

[INFO] [org.sonarsource.scanner.lib.internal.ScannerEngineLauncher] Load branch configuration
[INFO] [org.sonarsource.scanner.lib.internal.ScannerEngineLauncher] Auto-configuring pull request 3

Whereas locally I see:

[INFO] [org.sonarsource.scanner.lib.internal.ScannerEngineLauncher] Load branch configuration
[INFO] [org.sonarsource.scanner.lib.internal.ScannerEngineLauncher] Load branch configuration (done) | time=1ms

I’m not sure where exactly the project creation occurs, but the local execution downloads rules/plugins etc. and uploads results:

https://sonarcloud.io/api/ce/submit?organization=example_org&projectKey=org.example%3Aapp&projectName=app (999ms, 44-byte body)
[INFO] [org.sonarsource.scanner.lib.internal.ScannerEngineLauncher] Analysis report uploaded in 1005ms

whereas the GitLab version exits pretty soon after attempting to configure the PR and does not create the project on the SonarCloud side.

This isn’t unique to MR pipelines: it seems to be a common result of the Gitlab CI vendor being detected. For branch pipelines we see:

[INFO] [org.sonarsource.scanner.lib.internal.ScannerEngineLauncher] Load branch configuration
[INFO] [org.sonarsource.scanner.lib.internal.ScannerEngineLauncher] Auto-configuring branch hotfix/debug

… and as before, the process exits soon after this without creating anything on the Sonar side.

One last observation, which ties these two questions pretty closely together: if the scanner can detect GitLab CI as a vendor, could it not automatically create the ALM binding, thereby drastically simplifying the script I shared in my previous message?

[Colin]

@JackWhelpton

Thanks @Colin. It seems like I might need to mix and match the SonarQube Cloud APIs for now: some v1 calls using sonarcloud.io/api, and some v2 ones using api.sonarcloud.io.

That’s fine, and what we expect. No harm here.

@JackWhelpton

Does this look sane? Is there a simpler way to achieve the same result?

Yes, this looks sane! It’s more or less what we would recommend to somebody wanting to automatically provision their projects outside of the SonarCloud UI. However, let’s zoom in on the actual project creation/provisioning.

A new project can be provisioned (created) via analysis (Gradle) assuming the following conditions are met:

• The user the token was generated for has Create Projects permissions in the SonarQube Cloud organization
• The first analysis does not include sonar.branch.name or any sonar.pullrequest.* analysis parameters, including those automatically configured when running in Gitlab CI (or other supported CI tools). When the first analysis does include branch/PR configuration, the analysis will fail and the project will not be created.
  • This behavior was fixed in SonarQube Server years ago to provision a project even when branch/PR configuration is provided, but the ticket is still open for SonarQube Cloud. I know the ticket number by heart.

So instead of letting Gradle provision the project and having to deal with whether or not the first analysis is of the main branch, I would suggest you provision the project with POST api/projects/create prior to running analysis. You can make your other setup API calls as well prior to running Gradle.

@JackWhelpton

One last observation, which ties these two questions pretty closely together: if the scanner can detect GitLab CI as a vendor, could it not automatically create the ALM binding, thereby drastically simplifying the script I shared in my previous message?

Yes, this makes a lot of sense. It’s something we’re considering, because as you’ve noted, this is all a bit convoluted. We have an open roadmap item here.

[JackWhelpton]

Thanks, good to know I’m vaguely on the right track.

Whilst I like the idea of invoking the api/projects/create endpoint, the problem I then face is generating a projectKey; I can’t find any way to know in advance what key the Gradle plugin would compose.

My current thought is to replicate instead the logic the UI uses to generate these keys, which seems to be along the lines of CI_PROJECT_ROOT_NAMESPACE-CI_PROJECT_NAME; I need to investigate how it handles clashes, e.g. importing gitlab.com/example_org/group_1/my_app and gitlab.com/example_org/group_2/my_app.

Then when running the analysis I would need to override the key with this value:

      ./gradlew sonar \
          -Dsonar.organization=example_org \
          -Dsonar.projectKey=${generated_project_key} \
          -Dsonar.qualitygate.wait=true \
          -Dsonar.working.directory=.sonarwork

Is that what you had in mind?

The alternative approaches would be either a) trying to fool the CI into operating in a “disconnected” fashion by gaming the CI vendor detection (which would potentially result in MRs being analyzed as branches or vice versa), or b) somehow extracting the projectKey out of the Gradle config in advance of an execution (which seems challenging, without hacking the Gradle scripts themselves).

[Colin]

@JackWhelpton

Is that what you had in mind?

Yes. Personally, I think relying on what Gradle computes as the project key on its own carries some risk.

• No guarantee that there won’t be a clash with another SonarCloud project key. SonarCloud project keys are scoped to all of SonarCloud, not just your org (maybe someday!)
• If you change the name of your Gradle project, you have to remember to update the SonarCloud project key or else a new project will be generated. Impossible to merge the two after.
  • And this is why we reccomend storing the project key in a your build.gradle file instead of dynamically generating it on each analysis

[JackWhelpton]

Agreed! The struggle I’m facing is attempting to create a pipeline that is foolproof enough that the creation and binding are independent of whether the user has defined Sonar properties themselves or not: in an ideal world the Sonar piece would be completely invisible to users of that pipeline, beyond the presence of MR comments and blocked merges.

In my approach b) above, what I really wanted to do was to extract the exact key that Gradle would have used, with explicit definition of sonar.projectKey within the build scripts taking precedence (not just the logic it uses if the key is missing).

For the option we seem to be pushing more towards, there’s one piece that’s still unclear in my head. I can generate an initial key, based on knowing that a) the project is currently unbound and b) generating a unique key, handling collisions by regenerating (e.g. a counter or somesuch).

If I’m not to commit any changes/Gradle hackery to their repo, which undermines the invisibility tenet above, I need a way to identify the ALM binding based on the repo ID, rather than from the Sonar project ID (which I know I can do). Then my flow is straightforward:

Is this GitLab CI_PROJECT_ID bound to an existing Sonar project?
YES: use that binding (so I don’t need a true/false here, I want the projectId or ideally key)
NO: OK, create a new project and bind it; then the same lookup will return that Sonar project for subsequent runs.

I need to go back to snooping on the API, as there must be a way to at least return true/false, as when you select a new repo for binding it clearly greys it out if it’s already bound. Thinking about it, though, that may be a property on a search endpoint, meaning I would have to paginate through all of our repos, and that is pretty damn painful in the UI today (it times out for me more times than it loads). It appears to include archived repositories, which may be one reason why its performance hasn’t improved as we clean house.

UPDATE: This connects right back to my confusion on APIs between SonarQube and SonarCloud, as the endpoint the UI calls for that search screen is https://sonarcloud.io/api/alm_integration/list_repositories?organization=example_org. The alm_integration endpoints aren’t part of the SonarCloud API docs, but are in the SonarQube Server ones. That said, the specific list_repositories endpoint isn’t in either. Still very confused.

[Colin]

@JackWhelpton

UPDATE: This connects right back to my confusion on APIs between SonarQube and SonarCloud, as the endpoint the UI calls for that search screen is https://sonarcloud.io/api/alm_integration/list_repositories?organization=example_org. The alm_integration endpoints aren’t part of the SonarCloud API docs, but are in the SonarQube Server ones. That said, the specific list_repositories endpoint isn’t in either. Still very confused.

There are some API endpoints that, when developed, were classified as internal and not publicly documented. You are probably running into some of that. I’m not a fan of this. At least on SonarQube Server, a fearless user can tick a box to see those internal APIs in the docs. Not so for SonarQube Cloud today.

It might be interesting to understand that for several years, the teams working on SonarQube Server and SonarQube Cloud (nee SonarCloud) were largely separate teams at Sonar. That is no longer the case, but there’s 5 years of somewhat diverging products (and different implementations of features) to reckon with. Ideally we will eventually land on more consistent behavior and a common(ish) v2 API between the products. That will take some time.

In any case, you can always snoop those internal APIs as you’d expect.

How to reverse engineer the Sonar Web API

Every action that is taken in SonarQube is driven by the Web API. There is nothing you can do in SonarQube that you cannot do using the Web API. We put a great deal of effort into the UI and UX of our products and sincerely think that most actions should be completed there. Especially when it comes to viewing analysis results, the UI is the best place to view up-to-date information with the right context. We think it is much better that you give access to others to view analysis data …

[JackWhelpton]

Thanks, that does make a lot of sense (and believe me, I’m working at a place that seems to have created this level of confusion over API versions without requiring multiple teams to do it, so no judgement here).

The question stands, though: what is the recommended way is to determine the Sonar project binding for a known CI repository? The only approach I can see so far is listing all the repositories and paginating; even there the only correlation to the repo seems to be label, which can change frequently and is definitely non-unique. And that API is internal and sloooooooow.

Ideally, in the same way that I can do:

GET https://api.sonarcloud.io/dop-translation/project-bindings?projectId=${sonar_project_id}

I would be able to do:

GET https://api.sonarcloud.io/dop-translation/project-bindings?repositoryId=${CI_PROJECT_ID}

Assuming that’s a feature request and there’s no timeline for when such an API would be available, how can this be solved today?

[Colin]

Unforuntately, I think that today GET api/alm_integration/list_repositories is the only option here today.

@JackWhelpton

The only approach I can see so far is listing all the repositories and paginating; even there the only correlation to the repo seems to be label, which can change frequently and is definitely non-unique.

The API also returns installationKey, which should match CI_PROJECT_ID on the Gitlab side.

{
  "repositories": [
    {
      "label": "newtest",
      "installationKey": "65612230",
      "linkedProjects": [
        {
          "key": "colin-mueller_esfadsfsf",
          "name": "esfadsfsf"
        }
      ],
      "private": false
    },

It is an interesting point that a lot of automation woes would be solved if there was a single endpoint to query whether a repo has a SonarCloud project already (without having to list all repos). I’ll make sure to pass along that feedback.

[JackWhelpton]

Hope you had a good Christmas! Just to close this thread out, given the absence of suitable lookup APIs, what I settled on in the end is a flow like this:

• Execute ./gradlew sonar
  • Parse stderr for the text not find a default branch for project with key '{}'
  • Extract projectKey from this text if it appears
  • If such text was found, POST https://sonarcloud.io/api/projects/create
  • Otherwise, project exists, so source report-task.txt for metadata
• Obtain projectId using GET https://sonarcloud.io/api/navigation/component
• Set ALM binding using POST https://api.sonarcloud.io/dop-translation/project-bindings
  • If the project exists and is correctly bound, this is a no-op
  • If the project exists and is bound elsewhere, this will be an error (desirable)
  • Otherwise, we now have a correctly bound project
• Rename the main branch to CI_DEFAULT_BRANCH using POST https://sonarcloud.io/api/project_branches/rename
  • Parse the error messages in the response for already exists
  • If found, delete CI_DEFAULT_BRANCH using POST https://sonarcloud.io/api/project_branches/delete and retry
• Finally, if we created the project or deleted the branch, re-execute the Gradle scan

It feels a bit convoluted, but it’s optimized for the happy path and seems to be suitably “no touch” for users of the pipeline. Things that might make it easier, in rough priority order:

1. Ability to look up project key based on installation key / repo ID (we discussed this one)
2. Non-destructive way to change default branch (I think SonarQube Server has this now)
3. Ability to specify default branch on creation
4. Return projectId on creation, to save an additional lookup
   • Better still, enable projectKey to be used rather than projectId when binding a project

#1 is still important to me, for handling non-JVM languages and the scenario you already touched, on, where the project name is changed in a way that affects its implicit projectKey.

Of course, in my dream scenario, the scanner could handle a lot of this itself.

One last note on the internal API for browsing existing bindings that you mentioned above, from Fiddler:

ServerGotRequest:		10:23:47.488
ServerBeginResponse:	10:24:05.905

This causes the SonarQube Cloud UI itself to time out repeatedly for us. I think part of the reason may be the inclusion of archived repos in the list.

[Eric_MORAND]

@Colin

It might be interesting to understand that for several years, the teams working on SonarQube Server and SonarQube Cloud (nee SonarCloud) were largely separate teams at Sonar. That is no longer the case, but there’s 5 years of somewhat diverging products (and different implementations of features) to reckon with. Ideally we will eventually land on more consistent behavior and a common(ish) v2 API between the products. That will take some time.

@Colin , so much for the “we don’t need technical direction” dogma, right?

[JackWhelpton]

Revisiting this thread with a few months of operational experience: we’ve had to move away from the list_repositories API for performance reasons, mentioned above but apparently deepening: it’s deteriorated from slow responses to regularly and repeatedly returning 500 errors.

Our newer approach is here:

Intermittent 500 errors from sonarcloud.io/api/alm_integration/list_repositories

Hi, Using an undocumented endpoint means that it’s explicitly marked internal and thus there are no promised about functionality (or performance!). Nonetheless, I’ll flag this for the team. But I wouldn’t hold my breath IIWY.   Ann

which is pretty much a continuation of this thread. It has its own drawbacks, but rather than reiterating them here I’ll let the link serve to tie the two conversations together.

@Eric_MORAND, to your point: I am slightly miffed by the response from Sonar over there re. undocumented API usage, as our solution was what @Colin recommended above, not something we dreamed up. Recommending a way to solve our use cases and then refusing to provide any support or further recommendations when it breaks down isn’t a great developer experience, and “I wouldn’t hold my breath” isn’t a solution.

[Eric_MORAND]

@JackWhelpton, well, Colin is not really responsible for the current state of the API. It is very clear, looking at the network calls done by the SonarQube Web Application, that the public API is the one that is used by SonarQube UI to render its content.

Which is in therory a very good thing, and a great design decision: it puts the SonarQube Web Application on the same level than any other applications in terms of what can be done, which says a lot about the original intention to not consider the official Web Application as a first-class citizen.

I really love that. On paper.

So, why is it so flawed in Sonar’s case? Why did someone, at one point, decided that there would be public internal APIs?

Well, I’m quite sure this is coming from the start-up culture that they were not able to move from, and that prevents the company to have some horizontal vision. But I’m just guessing here.