Updating your Azure DevOps ARM Service Connections to use the recommended Workload Identity Federation

This post is showcased on Microsoft Premier Developer Blog.

With the recent arrival of the Public preview of Workload identity federation for Azure Pipelines, you may be wondering how can I efficiently migrate my dozens or even hundreds of ARM Service Connections to take advantage of these main benefits:

Simplified management: You do not need to generate, copy, and store secrets from service principals in Azure Entra ID to Azure DevOps anymore. Secrets that are used in other authentication schemes of Azure service connections (e.g., service principal) expire after a certain period (2 years currently). When they expire, pipelines fail. You have to generate a new secret and update the service connection. Switching to workload identity federation eliminates the need to manage these secrets and improves the overall experience of creating and managing service connections.
Improved security: With workload identity federation, the federation subject sc://<org>/<project>/<service connection name> uniquely identifies what the identity can be used for, which provides a better constraint than a (shared) secret. There is no persistent secret involved in the communication between Azure Pipelines and Azure. As a result, tasks running in pipeline jobs cannot leak or exfiltrate secrets that have access to your production environments. This has often been a concern for our customers.

Figure 1
Workload Identity Federation (automatic) is Recommended

Why Bother?

Not convinced yet? Then run the following simple yaml pipeline against a service principal ARM connection:

trigger:
- none

pool:
  vmImage: windows-latest

steps:
- task: AzureCLI@2
  displayName: save service connection secret
  inputs:
    azureSubscription: 'doNotUseSP' # use workload identity federation instead (with branch control)
    scriptType: 'pscore'
    scriptLocation: 'inlineScript'
    inlineScript: |
      Write-Host "##vso[task.setvariable variable=SpId;]$env:servicePrincipalId"
      Write-Host "##vso[task.setvariable variable=SpKey;]$env:servicePrincipalKey"
      Write-Host "##vso[task.setvariable variable=TenantId;]$env:tenantId"
      Write-Host "##vso[task.setvariable variable=TestVar;]ShouldAlwaysSeeMe"
    addSpnToEnvironment: true
- task: PowerShell@2
  displayName: exfiltrate production credentials
  inputs:
    targetType: 'inline'
    script: |
      Write-Host "Exfiltrating secrets..."
      Write-Host $env:SpId
      Write-Host $env:SpKey
      Write-Host $env:TenantId
      Write-Host $env:TestVar

      echo $env:SpKey > spkey.txt
      echo $env:SpId > spid.txt
      echo $env:TenantId > tenant.txt
      cat spkey.txt
      cat spid.txt
      cat tenant.txt
- task: CopyFiles@2
  displayName: copy secrets in plain text
  inputs:
    SourceFolder: '$(System.DefaultWorkingDirectory)'
    Contents: '**/*.txt'
    TargetFolder: '$(build.artifactstagingdirectory)'
- task: PublishPipelineArtifact@1
  displayName: publish secrets
  inputs:
    targetPath: '$(build.artifactstagingdirectory)'
    artifact: 'dropSecretsExfiltrated'
    publishLocation: 'pipeline'

You have just exfiltrated production secrets (service principal id and key along with tenant id):

Figure 1b
Avoid Service Principal Credential Exfiltration

Before the conversion

First, you will need an inventory of all your Azure DevOps ARM Service Connections which you can obtain from the az devops CLI.

Converting Authentication Scheme from Service Principal to Workload Identity Federation

The bulk conversion is done via the script Convert-ServicePrincipals.ps1.

It relies on a (yet) undocumented PUT call:

PUT https://dev.azure.com/${organizationName}/_apis/serviceendpoint/endpoints/${endpointId}?operation=ConvertAuthenticationScheme&api-version=${apiVersion}

with payload

{
    "id": "${endpointId}",
    "type": "azurerm",
    "authorization": {
        "scheme": "WorkloadIdentityFederation"
    },
    "serviceEndpointProjectReferences": [
        {
            "description": "",
            "name": "${serviceConnectionName}",
            "projectReference": {
                "id": "${projectId}",
                "name": "${projectName}"
            }
        }
    ]
}

Whether you convert the service connection in the Azure DevOps UI or programmatically via the PUT call above, you should see a screen similar to:

conversion in ado — Figure 2
Conversion in Progress

Reverting a conversion

You have 7 days to revert back to a service principal. The same PUT call will work after simply changing the payload’s authorization.scheme from WorkloadIdentityFederation to ServicePrincipal. Reverting looks like this in the Azure DevOps portal:

reverting a converted sc — Figure 3a
Reverting a conversion within 7 days

A sample production run to revert all is:

./Convert-ServicePrincipals.ps1 -isProductionRun $true `
     -refreshServiceConnectionsIfTheyExist $true `
     -revertAll $true

generates a summary such as:

Handling Manual Conversions

When handling manual conversions through the API, you may see an error such as:

{
 "$id": "1",
 "innerException": null,
 "message": "The authorization scheme could not be upgraded to WorkloadIdentityFederation because the service principal could not be configured automatically, and no valid configuration exists.",
 "typeName": "System.ArgumentException, mscorlib",
 "typeKey": "ArgumentException",
 "errorCode": 0,
 "eventId": 0
}

Or in the Azure DevOps Portal, you may see the following message:

Automatic authentication conversion failed. Your service connection was not modified. To continue the conversion manually, create a Federated Credential for the underlying Service
Principal using the Federation Subject Identifier below and try again.

as in this screenshot:

Figure 4
Automatic authentication conversion failure

To create a federated credential, first create a file called credential.json which follows this template:

{
 "name": "__ENDPOINT_ID__",
 "issuer": "https://vstoken.dev.azure.com/__ORGANIZATION_ID__",
 "subject": "sc://__ORGANIZATION_NAME__/__PROJECT_NAME__/__SERVICE_CONNECTION_NAME__",
 "description": "Federation for Service Connection __SERVICE_CONNECTION_NAME__ in https://dev.azure.com/__ORGANIZATION_NAME__/__PROJECT_NAME__/_settings/adminservices?resourceId=__ENDPOINT_ID__",
 "audiences": [
     "api://AzureADTokenExchange"
 ]
}

Then use the az cli with the application registration id (aka Client Id) $appObjectId as such:

az ad app federated-credential create --id $appObjectId --parameters credential.json

Bear in mind that the script Convert-ServicePrincipals.ps1 automatically handles this case and will pre-create the necessary federated credentials prior to attempting a conversion for a manual Service Principal.

Verify and Save

It’s important to “Verify and Save” the newly converted service connections, especially for the manual service principals that got converted!

If you see an error such as the one above:

Failed to query service connection API: 'https://management.azure.com/subscriptions/********-****-****-****-************?api-version=2016-06-01'. Status Code: 'Forbidden', Response from server: '{"error":{"code":"AuthorizationFailed","message":"The client 'dd5*****-****-****-****-************' with object id 'dd5*****-****-****-****-************' does not have authorization to perform action 'Microsoft.Resources/subscriptions/read' over scope '/subscriptions/********-****-****-****-************' or the scope is invalid. If access was recently granted, please refresh your credentials."}}'

You will need to update the RBAC permissions of the corresponding app registration (service principal). This can be done in the Azure Portal or through the command line.

Finally, it’s always a good idea to test a pipeline that uses the service connection post conversion to ensure everything is in working order.

Conversion of manual service principals referenced by multiple service connections

It is not recommended to have a single app registration referenced by multiple service connections. However, the script will convert the multiple service connections leveraging multiple federated credentials such as:

multiple fed creds — Figure 7
Service Principal referenced by Multiple Service Connections

Common errors while attempting to convert

While running the conversion script, you may see the following error:

{
 "$id": "1",
 "innerException": null,
 "message": "Converting endpoint type azurerm scheme from WorkloadIdentityFederation to WorkloadIdentityFederation is neither an upgrade or a downgrade and is not supported.",
 "typeName": "System.ArgumentException, mscorlib",
 "typeKey": "ArgumentException",
 "errorCode": 0,
 "eventId": 0
}

This indicates that you are trying to convert to the same authorization scheme, in the above case: WorkloadIdentityFederation.

If you are using Azure Stack, you may encounter the following Azure Stack related error:

{
 "$id": "1",
 "innerException": null,
 "message": "Unable to connect to the Azure Stack environment. Ignore the failure if the source is Azure DevOps.",
 "typeName": "Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi.ServiceEndpointException, Microsoft.VisualStudio.Services.ServiceEndpoints.WebApi",
 "typeKey": "ServiceEndpointException",
 "errorCode": 0,
 "eventId": 3000
}

You may ignore this as Azure Stack is not supported (see Create an Azure Resource Manager service connection using workload identity federation ).

After Conversion

A sample production run

./Convert-ServicePrincipals.ps1 -isProductionRun $true `
  -refreshServiceConnectionsIfTheyExist $true

generates a summary such as:

You may need to iterate a few times. Don’t worry, the script is eventually consistent (then idempotent). This is great as we have managed to convert a large majority of our service connections automatically thus saving us lots of time!

Gotchas

After conversion, ensure pipelines using the newly converted Workload Identity Federation ARM Service Connection still work. In particular, they must use a supported task. For more information, refer to Public preview of Workload identity federation for Azure Pipelines – Azure DevOps Blog (microsoft.com). For convenience, we list the supported built-in tasks in the following paragraph.

Built-in Pipeline tasks

Azure Pipelines built-in tasks that target Azure have been updated to take advantage of workload identity federation. This is the complete list:

AzureAppServiceManage
AzureAppServiceSettings
AzureCLI
AzureCloudPowerShellDeployment
AzureContainerApps
AzureFunctionAppContainer
AzureFunctionApp
AzureKeyVault
AzureMonitor
AzureMysqlDeployment
AzurePolicy
AzurePowerShell
AzureResourceGroupDeployment
AzureResourceManagerTemplateDeployment
AzureRmWebAppDeployment
AzureSpringCloud
AzureVmssDeployment
AzureWebAppContainer
AzureWebApp
DockerCompose
Docker
HelmDeploy
InvokeRestApi
JavaToolInstaller
JenkinsDownloadArtifacts
Kubernetes

Azure DevOps Workload Identity Federation