Azure Workload Identity: Remove Secrets From Your AKS Pods in 45 Minutes

If you are still mounting a Kubernetes Secret containing an AZURE_CLIENT_SECRET into your pods, you are one kubectl get secret -o yaml away from an incident report. And if you are using AAD Pod Identity, you are running a component that reached end of support in September 2023.

Azure Workload Identity replaces both. It uses the OIDC issuer built into your AKS cluster to let a Kubernetes ServiceAccount exchange its projected token for an Azure AD (Microsoft Entra ID) access token. No secret, no rotation job, no NodeManagedIdentity DaemonSet intercepting IMDS traffic.

This walkthrough takes about 45 minutes end to end, assuming you already have an AKS cluster and the Azure CLI installed.

The problem: static credentials are a liability

A typical "pre-Workload-Identity" deployment looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: app-azure-credentials
  namespace: payments
type: Opaque
stringData:
  AZURE_TENANT_ID: 00000000-0000-0000-0000-000000000000
  AZURE_CLIENT_ID: 11111111-1111-1111-1111-111111111111
  AZURE_CLIENT_SECRET: "<never do this>"

Three structural problems:

1. Base64 is not encryption. Anyone with get secrets RBAC in that namespace — or read access to your etcd backups, or to the Git repo where the manifest lives — owns that identity.
2. Rotation is manual. Client secrets expire (6, 12, 24 months). Your rotation is a calendar reminder and a pipeline run, so it fails at 2 a.m. on a Sunday.
3. No blast radius control. The same secret works from a laptop, a CI runner, or a compromised pod in another namespace. Nothing ties it to a workload.

AAD Pod Identity tried to fix this by intercepting calls to the Instance Metadata Service (169.254.169.254) and injecting a managed identity token. It worked, but it was node-scoped, raced at pod startup (AzureIdentityBinding not reconciled yet → 404 from IMDS), and required privileged DaemonSets with iptables rules. It is deprecated and removed from new clusters.

Workload Identity takes a different path: federation. The cluster becomes an OIDC identity provider. Entra ID trusts tokens it issues, for one specific subject. No network interception, no DaemonSet, no secret.

How the token exchange actually works

Understanding the flow makes debugging trivial later:

1. The kubelet projects a short-lived (default 1 hour), audience-scoped JWT for the pod's ServiceAccount at /var/run/secrets/azure/tokens/azure-identity-token. The audience is api://AzureADTokenExchange.
2. The Azure SDK (or DefaultAzureCredential) in your app reads that file, plus the AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_AUTHORITY_HOST environment variables injected by the mutating webhook.
3. The SDK calls Entra ID with a client_assertion = the projected token.
4. Entra ID validates the signature against the cluster's public JWKS (fetched from the OIDC issuer URL), checks iss, sub and aud against the federated identity credential you registered, and returns an Azure access token.
5. Your app calls Key Vault / Storage / Cosmos DB with that token.

The only thing you configure is step 4: which issuer, which subject, which audience. Everything else is plumbing.

Step 1 — Enable the OIDC issuer and the webhook

Set your variables first:

export RG=rg-platform-prod
export CLUSTER=aks-platform-prod
export LOCATION=westeurope
export SUBSCRIPTION=$(az account show --query id -o tsv)

Both features are flags on the cluster. On an existing cluster:

az aks update \
  --resource-group "$RG" \
  --name "$CLUSTER" \
  --enable-oidc-issuer \
  --enable-workload-identity

This takes 5 to 10 minutes and performs a control-plane reconcile. It is non-disruptive for running workloads, but the mutating webhook only applies to newly created pods — existing pods will not get the projected token until they are recreated.

For a new cluster:

az aks create \
  --resource-group "$RG" \
  --name "$CLUSTER" \
  --location "$LOCATION" \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --node-count 2 \
  --generate-ssh-keys

Retrieve the issuer URL — you will need it twice:

export AKS_OIDC_ISSUER=$(az aks show \
  --resource-group "$RG" \
  --name "$CLUSTER" \
  --query "oidcIssuerProfile.issuerUrl" -o tsv)

echo "$AKS_OIDC_ISSUER"
# https://westeurope.oic.prod-aks.azure.com/<tenant-guid>/<cluster-guid>/

Sanity check that it is publicly reachable (Entra ID must be able to fetch the JWKS):

curl -s "${AKS_OIDC_ISSUER}.well-known/openid-configuration" | jq .jwks_uri

Confirm the webhook is running:

kubectl get pods -n kube-system -l azure-workload-identity.io/system=true
# azure-wi-webhook-controller-manager-xxxxx   1/1   Running

Step 2 — Create the managed identity

Use a user-assigned managed identity (UAMI). You can also federate an App Registration, but a UAMI has no secret at all and lives in your resource group, which keeps RBAC and lifecycle in one place.

export UAMI=id-payments-api

az identity create \
  --name "$UAMI" \
  --resource-group "$RG" \
  --location "$LOCATION"

export USER_ASSIGNED_CLIENT_ID=$(az identity show \
  --name "$UAMI" --resource-group "$RG" \
  --query 'clientId' -o tsv)

export USER_ASSIGNED_PRINCIPAL_ID=$(az identity show \
  --name "$UAMI" --resource-group "$RG" \
  --query 'principalId' -o tsv)

Grant it something concrete — read-only access to one Key Vault:

export KV_NAME=kv-payments-prod
export KV_ID=$(az keyvault show --name "$KV_NAME" --query id -o tsv)

az role assignment create \
  --assignee-object-id "$USER_ASSIGNED_PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "$KV_ID"

Note: Key Vault Secrets User only works if your vault uses Azure RBAC for data plane authorization (--enable-rbac-authorization true). If it still uses access policies, use az keyvault set-policy --object-id "$USER_ASSIGNED_PRINCIPAL_ID" --secret-permissions get list instead.

Step 3 — Create the ServiceAccount

The ServiceAccount is the Kubernetes-side identity. The annotation tells the webhook which client ID to inject.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-api
  namespace: payments
  annotations:
    azure.workload.identity/client-id: "11111111-1111-1111-1111-111111111111"

Apply it with the real value substituted:

kubectl create namespace payments --dry-run=client -o yaml | kubectl apply -f -

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-api
  namespace: payments
  annotations:
    azure.workload.identity/client-id: "${USER_ASSIGNED_CLIENT_ID}"
EOF

Optional annotations worth knowing:

• azure.workload.identity/tenant-id — override the tenant (multi-tenant scenarios).
• azure.workload.identity/service-account-token-expiration — token lifetime in seconds, between 3600 and 86400. Leave it at the default unless you have a specific reason.

Step 4 — Create the federated identity credential

This is the trust anchor. The subject must match exactly the format system:serviceaccount:<namespace>:<name>.

az identity federated-credential create \
  --name fc-payments-api \
  --identity-name "$UAMI" \
  --resource-group "$RG" \
  --issuer "$AKS_OIDC_ISSUER" \
  --subject "system:serviceaccount:payments:payments-api" \
  --audience "api://AzureADTokenExchange"

A few rules that bite people:

• One federated credential = one (issuer, subject) pair. A ServiceAccount in staging needs its own credential, even for the same identity. You can attach up to 20 per identity.
• The subject is case-sensitive and has no wildcards (except for the newer flexible federated credentials with * support, which are not GA everywhere — assume exact match).
• Propagation takes a few seconds to a couple of minutes. If your first test fails, wait and retry before debugging.

Verify:

az identity federated-credential list \
  --identity-name "$UAMI" --resource-group "$RG" -o table

Step 5 — Label the pod and deploy

The webhook only mutates pods carrying the label azure.workload.identity/use: "true" on the pod template, not on the Deployment metadata. This is the single most common mistake.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
        azure.workload.identity/use: "true"
    spec:
      serviceAccountName: payments-api
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: ghcr.io/example/payments-api:1.4.2
          env:
            - name: KEYVAULT_URL
              value: "https://kv-payments-prod.vault.azure.net/"
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi

automountServiceAccountToken: false disables the default Kubernetes API token. The Workload Identity projected volume is added independently by the webhook, so Azure auth still works while your pod loses needless API server access.

Verify the mutation happened:

kubectl -n payments get pod -l app=payments-api \
  -o jsonpath='{.items[0].spec.containers[0].env}' | jq .

You should see AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_FEDERATED_TOKEN_FILE and AZURE_AUTHORITY_HOST.

Testing from inside a pod

A quick smoke test without touching your application code:

kubectl -n payments run wi-test \
  --image=mcr.microsoft.com/azure-cli:latest \
  --labels="azure.workload.identity/use=true" \
  --overrides='{"spec":{"serviceAccountName":"payments-api"}}' \
  --restart=Never -it --rm -- bash

Inside the pod:

az login --federated-token "$(cat $AZURE_FEDERATED_TOKEN_FILE)" \
  --service-principal \
  -u "$AZURE_CLIENT_ID" \
  -t "$AZURE_TENANT_ID"

az keyvault secret show --vault-name kv-payments-prod --name db-password

If that works, DefaultAzureCredential in your Python, .NET, Java, Go or Node app will work with zero code changes — the SDKs have supported WorkloadIdentityCredential since 2023.

Debugging token errors

AADSTS70021: No matching federated identity record found

The most frequent error. The subject or issuer does not match. Decode the token the pod actually received:

kubectl -n payments exec deploy/payments-api -- \
  cat /var/run/secrets/azure/tokens/azure-identity-token \
  | cut -d. -f2 | base64 -d 2>/dev/null | jq '{iss, sub, aud, exp}'

Compare iss and sub character by character with the federated credential. Typical causes: namespace mismatch, trailing slash difference on the issuer, or the app was copied from another environment.

AZURE_FEDERATED_TOKEN_FILE is not set

The webhook did not mutate the pod. Check, in order:

# 1. Is the label on the POD, not just the Deployment?
kubectl -n payments get pod -l app=payments-api --show-labels

# 2. Is the annotation on the ServiceAccount?
kubectl -n payments get sa payments-api -o yaml

# 3. Is the webhook alive?
kubectl -n kube-system logs -l azure-workload-identity.io/system=true --tail=50

Remember: the mutation happens at pod admission. Changing a ServiceAccount annotation requires a kubectl rollout restart deployment/payments-api.

AADSTS700016: Application not found in the directory

The client-id annotation points to an identity that does not exist, or exists in a different tenant. Re-read it from the source: az identity show --name "$UAMI" -g "$RG" --query clientId -o tsv.

403 Forbidden from Key Vault / Storage

Authentication succeeded, authorization failed. The federation is fine — the role assignment is missing or scoped wrong. Check:

az role assignment list \
  --assignee "$USER_ASSIGNED_PRINCIPAL_ID" \
  --all -o table

Also check Key Vault firewall rules: a correct token still gets blocked if your AKS egress IP is not allowed.

Security best practices

One identity per workload, not per cluster. The temptation is a single id-aks-prod with Contributor at subscription scope. That rebuilds the exact blast radius problem you were trying to remove. One UAMI per application, scoped to the specific vault, storage account or database.

Keep subjects narrow. Avoid wildcard federated credentials if your Azure tenant supports them. system:serviceaccount:payments:payments-api is auditable; system:serviceaccount:payments:* means any new ServiceAccount in that namespace silently inherits production access.

Lock down ServiceAccount creation. Anyone who can create a ServiceAccount named payments-api in namespace payments plus a pod using it can assume the identity. Enforce it with RBAC and an admission policy. Example with a Kyverno-style guard or a plain ValidatingAdmissionPolicy:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: restrict-wi-annotation
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["serviceaccounts"]
  validations:
    - expression: >
        !has(object.metadata.annotations) ||
        !('azure.workload.identity/client-id' in object.metadata.annotations) ||
        object.metadata.namespace in ['payments', 'billing']
      message: "Workload Identity annotations are restricted to approved namespaces"

Disable the default API token. automountServiceAccountToken: false as shown above. The projected Azure token is independent.

Never disable issuer validation. If a tool offers to skip JWKS verification, that tool is not production-grade.

Rotate the OIDC signing keys periodically. AKS supports it; it invalidates all outstanding projected tokens, which is exactly what you want after a suspected compromise:

az aks oidc-issuer rotate-signing-keys --resource-group "$RG" --name "$CLUSTER"

Expect a short window where existing tokens fail until pods refresh them.

Audit federated credentials like you audit SSH keys. They are long-lived trust statements. Export them regularly and diff against your IaC:

for id in $(az identity list -g "$RG" --query "[].name" -o tsv); do
  echo "== $id"
  az identity federated-credential list --identity-name "$id" -g "$RG" \
    --query "[].{name:name, subject:subject, issuer:issuer}" -o table
done

Codify everything. The four moving parts (UAMI, role assignment, federated credential, ServiceAccount annotation) belong in Terraform or Bicep plus GitOps. Manual az commands are fine for the first 45 minutes; they are not fine for the 50th application.

Conclusion

The migration path is short and mechanical:

1. az aks update --enable-oidc-issuer --enable-workload-identity — once per cluster.
2. Create a user-assigned managed identity per workload and give it a narrowly scoped Azure role.
3. Annotate the ServiceAccount with azure.workload.identity/client-id.
4. Register a federated identity credential matching system:serviceaccount:<ns>:<sa> with audience api://AzureADTokenExchange.
5. Add the azure.workload.identity/use: "true" label to the pod template and restart.

Three things to remember when it breaks: the label goes on the pod template, the subject string must match exactly, and the webhook only acts at pod creation time — so restart after any annotation change.

What you gain is not just convenience. You remove an entire class of credentials from Git, from etcd, from CI logs and from your rotation backlog. The token your pod holds lives one hour, is bound to one ServiceAccount in one namespace in one cluster, and is useless anywhere else. That is a meaningfully smaller attack surface for less than an hour of work.