How To Mount Azure Key Vault Secret to Pods in Azure Kubernetes Service

Scenario: You require a pod to mount a secret stored in an Azure Key Vault. So that an application running in the pod can access the secret as a file and environment variable. Also manage the access security between the AKS cluster to the key vault using a user assigned managed identity.

Background:

To integrate Azure Key Vault to AKS, this requires an add-on called azure-keyvault-secrets-provider

There are two pieces of this add on. One is the Secrets Store CSI Driver for Kubernetes secrets – Integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume.

The Secrets Store CSI Driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.

The second is the Azure Key Vault Provider for Secrets Store CSI Driver which allows for the integration of an Azure key vault with an Azure Kubernetes Service (AKS) cluster.

The following steps are mainly taken from the following articles, but I will walk through in implementing the scenario outlined above.

Assumptions

An existing AKS Cluster. My demo Kubernetes version is 1.23.12
An existing Azure Key Vault (in a different resource group and subscription than the AKS cluster)

Implementation Steps

1) Bash script variable initialization:

rgName='aks-solution'
aksName='rkaksdev'

keyVaultName='rkimKeyVault'
keyVaultRG='Enterprise'
keyVaultSubName='Enterprise'
# Kubernetes
# key vault demo namespace
keyVaultDemoNamespace=keyvault-demo

2) Enable or check if azure-keyvault-secrets-provider addon is installed

az aks enable-addons --addons azure-keyvault-secrets-provider --name $aksName --resource-group $rgName

3) Create a secret into an existing Key Vault

az keyvault secret set --vault-name $keyVaultName -n ExampleSecret --value s3cr3tV@lue

4) Create a user assigned managed identity to assign to VMSS and set permissions to Azure Key Vault secrets

aks2kvUserassignedidentityname='aks2kv-uami'
echo $aks2kvUserassignedidentityname
az identity create -g $rgName -n $aks2kvUserassignedidentityname

export identityResourceId=$(az identity show -g $rgName -n $aks2kvUserassignedidentityname --query id -o tsv)
echo $identityResourceId
export identityClientId=$(az identity show -g $rgName -n $aks2kvUserassignedidentityname --query clientId --output tsv)
echo $identityClientId
export identityPrincipalId=$(az identity show -g $rgName -n $aks2kvUserassignedidentityname --query principalId -o tsv)
echo $identityPrincipalId

5) Set agent pool VMSS name by going to the AKS infrastructure resource group to find the VMSS resource

# set policy to access secrets in your key vault or set in Azure Portal in Key Vault > Access Policies
az keyvault set-policy -g $keyVaultRG -n $keyVaultName  --subscription $keyVaultSubName --secret-permissions get --spn $identityClientId

6) Set policy to access secrets in your key vault or set in Azure Portal in Key Vault > Access Policies

# set policy to access secrets in your key vault or set in Azure Portal in Key Vault > Access Policies
az keyvault set-policy -g $keyVaultRG -n $keyVaultName  --subscription $keyVaultSubName --secret-permissions get --spn $identityClientId

7) Create the SecretProvider Class in your Kubernetes namespace. This is to interface with the given Azure Key Vault and the secret to mount.

# Create namespace for key vault demo
kubectl create namespace $keyVaultDemoNamespace
# Find key vault tenant ID
export keyvaultTenantId=$(az keyvault show  -g $keyVaultRG -n $keyVaultName --subscription $keyVaultSubName -o tsv --query properties.tenantId)
echo $keyvaultTenantId

cat <<EOF | kubectl apply -f -
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-rkimkv-secret-provider
  namespace: $keyVaultDemoNamespace
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "true"    # Set to true for using managed identity
    userAssignedIdentityID: $identityClientId      # If empty, then defaults to use the system assigned identity on the VM
    keyvaultName: $keyVaultName
    cloudName: ""                   # [OPTIONAL for Azure] if not provided, the Azure environment defaults to AzurePublicCloud
    objects:  |
      array:
        - |
          objectName: ExampleSecret
          objectType: secret        # object types: secret, key, or cert
          objectVersion: ""         # [OPTIONAL] object versions, default to latest if empty
    tenantId: $keyvaultTenantId           # The tenant ID of the key vault
  secretObjects:                              # [OPTIONAL] SecretObjects defines the desired state of synced Kubernetes secret objects
  - data:
    - key: examplesecretkey                           # data field to populate
      objectName: ExampleSecret                        # name of the mounted content to sync; this could be the object name or the object alias
    secretName: example-secret                     # name of the Kubernetes secret object
    type: Opaque       
EOF

8) Create a Pod that mounts the secret from Azure Key Vault as a file and environment variable. When an application such as a .NET Core app is deployed into this pod, it has access.

cat << EOF | kubectl apply -f -
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline-uami
  namespace: $keyVaultDemoNamespace
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
      env:
      - name: EXAMPLE_SECRET
        valueFrom:
          secretKeyRef:
            name: example-secret
            key: examplesecretkey
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-rkimkv-secret-provider"
EOF

8) Display the secret in the pod.

## show secrets held in secrets-store
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- ls /mnt/secrets-store/ 
## print a test secret 'ExampleSecret' held in secrets-store
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- cat /mnt/secrets-store/ExampleSecret
## Display the environment variables that includes the secret
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- printenv

Here you can see the output the secrets from a file and environment variable:

Understand that when updating the secret in Azure Key Vault, it is not automatically updated in the pod with this setup. To update, you have to delete and recreate the pod in this situation. To support polling for changes, you can read https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver#enable-and-disable-autorotation

Here you can see the relevant manifest yaml to support the integration and configuration.