Azure DevOps service connection: User token is changed on re-save

jvilimek · January 25, 2022, 10:09am

versions used: Azure DevOps SonaCloud extension 1.27.1 (Latest) Last updated Nov 30, 2021 at 12:13 PM GMT+1
error observed:

our pipelines started to failing on very first API calls to SonarCloud (i.e. https://sonarcloud.io/api/settings/values?component=xxxxx). Our tokens to SonarCloud were probably revoked by Sonarsource guys?
once the token fixed first calls were succesfull but then projects/search api still have http 401 error…
I have noticed there is “Requires ‘System Administrator’ permission” …which was not before…

… again, lot’s of our azure devops pipelines stopped working, lot’s of man/hours already spent investigating what the heck is broken again

I can only guess, that:

someone/something has revoked our tokens
API authorization has changed

Is my guess right? If yes, how can we update our pipeline to be “compliant”?

Is there any public information about such changes and why those are being introduced?

We are reallly, really, really, really (did I mention how much?) some public information about current version of SonarCloud, API, release notes, history of changes…

PS: why we are calling projects/search? There is simply no GET project api and we want to crate a project if it does not exist during our pipeline…

FYI @Tom_Howlett ?

jvilimek · January 25, 2022, 10:11am

How we check/create projects in the pipeline:

 $projects = Invoke-RestMethod -Uri "${schost}api/projects/search?organization=$scorganization&projects=$scprojectKeyEncoded" -Method Get -Headers $headers #authorization headers
    if ($null -eq $projects.paging.total)
    {
      throw "wrong response: $projects";
    }

    if (1 -eq $projects.paging.total)
    {
      Write-Output "SonarCloud project $scprojectKey already created.";
    }
    elseif (0 -eq $projects.paging.total)
    {
      if ("default" -eq $env:sonarCloudBranch)
      {
        throw "You are trying to initialize project without sonarCloudBranch parameter changed. Please change to [master] or [main]."
      }
      Write-Output "Creating SonarCloud project $scprojectKey...";
      $project = Invoke-RestMethod -Uri "${schost}api/projects/create?organization=$scorganization&name=$scprojectName&project=$scprojectKeyEncoded&visibility=private" -Method Post -Headers $headers
      Write-Output "Created OK";          
    }
    else
    {
      throw ("Unknown response: [{0}]" -f $projects.paging.total)
    }

Colin · January 25, 2022, 11:17am

Hey @jvilimek

SonarCloud tokens do not end up revoked without user action (SonarSource isn’t revoking any tokens on behalf of users), and there have been no deployments related to API authorization or these API endpoints.

GET api/projects/search requires Administer Organization permissions on the organization. This has always been the case. If the permissions are assigned to specific groups instead of individual users, it may be worth checking that group membership has not changed.

jvilimek · January 25, 2022, 11:31am

Thanks for the quick answer.

I am the organization owner and the token, that I have created and assigned to the pipelines, is mine (and it was all working before…like for months).

I would like to highlight, that I see Requires ‘System Administrator’ permission in the web api documentation:

Colin · January 25, 2022, 11:37am

Remnants of SonarQube past… I’ll flag this internally for review so that we change most instances of “System” to “Organization”. Thanks.

Are you still facing a 401 issue on GET api/projects/search, even with a new token?

You should be able to simply browse to https://sonarcloud.io/api/projects/search?organization= to make sure you have the right permissions.
A Postman/curl call using the token you’re using in your pipeline

If you’re happy to share your full Powershell script (including how you construct the headers) I can also test with my own inputs.

jvilimek · January 25, 2022, 11:59am

Oh, make sense. I have had worries this was changed somehow.

Yep. I am able to navigate (with correct organization parameter) and get results from that API call via browser.

Yep, sure, thanks. Since this worked fine for several moths (in fact since 2021/03) and just today it starts failing even the connection was OK/fixed, I kind of suspected some server side change…

First error (fixed once we fixed the token):

After connection fix:

As fo the script source:

function test-variable($v, $m)
    {
      Write-Output "Checking for $m..."
      if ([string]::IsNullOrEmpty($v))
      {
        throw "$m not found. Did SonarCloudPrepare step finished OK?"
      }
    }

    test-variable -v $env:sonarCloudBranch -m "sonarCloudBranch environment variable";

    test-variable -v $env:SONARQUBE_SCANNER_PARAMS -m "SONARQUBE_SCANNER_PARAMS environment variable";
    $sqparams = $env:SONARQUBE_SCANNER_PARAMS | ConvertFrom-Json

    $schost = $sqparams.{sonar.host.url};
    test-variable -v $schost -m "sonar.host.url configuration";

    $scorganization = $sqparams.{sonar.organization};
    test-variable -v $scorganization -m "sonar.organization configuration";
    $scorganization = [System.Web.HttpUtility]::UrlEncode($scorganization)

    $scprojectKey = $sqparams.{sonar.projectKey};
    test-variable -v $scprojectKey -m "sonar.projectKey configuration";
    $scprojectKeyEncoded = [System.Web.HttpUtility]::UrlEncode($scprojectKey)

    $scprojectName = $sqparams.{sonar.projectName};
    test-variable -v $scprojectName -m "sonar.projectName configuration";
    $scprojectName = [System.Web.HttpUtility]::UrlEncode($scprojectName)

    $sclogin = $sqparams.{sonar.login};
    test-variable -v $sclogin -m "sonar.login configuration";
    $base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("${sclogin}:"));
    $headers = @{
      Authorization = "Basic $base64AuthInfo";
    }

    $projects = Invoke-RestMethod -Uri "${schost}api/projects/search?organization=$scorganization&projects=$scprojectKeyEncoded" -Method Get -Headers $headers
    if ($null -eq $projects.paging.total)
    {
      throw "wrong response: $projects";
    }

    if (1 -eq $projects.paging.total)
    {
      Write-Output "SonarCloud project $scprojectKey already created.";
    }
    elseif (0 -eq $projects.paging.total)
    {
      if ("default" -eq $env:sonarCloudBranch)
      {
        throw "You are trying to initialize project without sonarCloudBranch parameter changed. Please change to [master] or [main]."
      }
      Write-Output "Creating SonarCloud project $scprojectKey...";
      $project = Invoke-RestMethod -Uri "${schost}api/projects/create?organization=$scorganization&name=$scprojectName&project=$scprojectKeyEncoded&visibility=private" -Method Post -Headers $headers
      Write-Output "Created OK";          
    }
    else
    {
      throw ("Unknown response: [{0}]" -f $projects.paging.total)
    }

    Write-Output "Updating PR integration to use Azure DevOps provider...";
    $POSTParams = @{
      key="sonar.pullrequest.provider";
      component=$scprojectKey;
      value="Azure DevOps Services";
    }
    Invoke-WebRequest -Uri "${schost}api/settings/set" -Headers $headers -Method POST -Body $POSTParams -UseBasicParsing | Out-Null;
    Write-Output "Updated OK.";

    Write-Output "Updating PR integration to use agent PAT token...";
    $POSTParams = @{
      key="sonar.pullrequest.vsts.token.secured";
      component=$scprojectKey;
      value="$(System.AccessToken)";
    }
    Invoke-WebRequest -Uri "${schost}api/settings/set" -Headers $headers -Method POST -Body $POSTParams -UseBasicParsing | Out-Null;
    Write-Output "Updated OK.";

Colin · January 25, 2022, 2:39pm

@jvilimek Using your script as-is, it works on my side.

I think the best debugging steps you could do would be printing the full URL that is being queried (to make sure it’s expected) as well as the value used for sonar.login (to make sure it’s the one you expect – the new token).

jvilimek · January 25, 2022, 5:02pm

Thanks Colin to pointing me to a right direction. Indeed the tocken was a different one… after digging through tons of audit logs (btw. for azure devops the link is Azure DevOps Services | Sign In & it can be exported to Azure Monitor and more…) I have found out the connection was modified by one of my colleague in parallel… that’s what has confused me tbh.

So sorry for filling up this bug & wasting your time while the issue was on our side. And really thanks a lot for helping me.

Anyway one question is still valid: is there any public information about a version (and release date/when it was deployed, release notes…) so we can eliminate server side changes/updates in the future?

Colin · January 25, 2022, 5:43pm

Hey @jvilimek

No worries.

It’s a valid question and also valid feedback which I’m happy to pass along. Here’s my honest take on it right now:

We don’t deploy “versions” of SonarCloud — we continuously merge and deploy changes to SonarCloud. When there are big changes (new features, deprecations, requirements changes, etc.) we announce them. Our general issue tracker (bugs/improvements/deployments) is internal.

This works pretty well as is. Can you honestly tell me that if such detailed notes were available, and something went wrong without a good explanation, the question wouldn’t become “what’s missing from the release notes”?

Personally (taking my SonarSourcer hat off), I wish we had a way of helping users know what changed on SonarCloud since yesterday, even if it’s the small stuff (as a user of other cloud services, I know I care about the small stuff sometimes!)

jvilimek · January 26, 2022, 11:29am

To be honest I believe it would at least point us in the right direction. Pipelines stops working: first thing we try to figure out is “what the hell was changed”…

jvilimek · January 26, 2022, 11:35am

We have finished investigation what had broken the service connection in the Azure DevOps:

It was re-saving the connection without changing the token but only the name:

Not sure, if this is something “built-in” in Azure DevOps or if this is something you as SonarSource as author of the service connection type can fix maybe?

Context: SonarCloud is shared service connection. Unfortunatelly it can not be shared “for all projects”, you need to assign it project per project. So yesterday colleague of mine was sharing the connection to some newly created project. Nother unfortunate thing is that the name of the connection is changed to “SonarCloud - NewProject” …while my colleague was trying to fix this (actually only changing the name of the connection) and saving it in the scope of the new project, he fails to “verify it”…so he saved it without verification… this process changed the token from real token to ‘****’ which was ofc not valid…

system · February 2, 2022, 11:35am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.