How to reload an Env variable sync’d to a Pod mounted Azure Key Vault Secret

Scenario: The objective is to have the environment variable in the Pod updated when the secret is updated in Azure Key Vault. You already have a secret in an Azure Key Vault mounted to your pod and sync’d as an environment variable; however, when you update the secret value in Azure Key Vault, the environment variable is not updated.

This blog post is an extension to the scenario setup in my previous blog post How To Mount Azure Key Vault Secret to Pods in Azure Kubernetes Service.

This blog post will go over the setup of an open source tool called Reloader to watch for changes on the synced Kubernetes Secret and perform rolling upgrades on pods. And to go over a test scenario to see the resulting behaviour.

References:

https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver#enable-and-disable-autorotation

https://github.com/stakater/Reloader

0) The scenario starts with the setup as defined in this my previous blog post How To Mount Azure Key Vault Secret to Pods in Azure Kubernetes Service

And to add the following K8s deployment

# Create a Pod that mounts the secret from Azure Key Vault as a file and environment variable
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-deployment-secrets-store-inline-uami
  namespace: $keyVaultDemoNamespace
  labels:
    app: busybox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
        - name: busybox
          image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
          command:
            - "/bin/sleep"
            - "20000"
          volumeMounts:
          - name: secrets-store01-inline
            mountPath: "/mnt/secrets-store"
            readOnly: true
          env:
          - name: EXAMPLE_SECRET
            valueFrom:
              secretKeyRef:
                name: example-secret
                key: examplesecretkey
      volumes:
        - name: secrets-store01-inline
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "azure-rkimkv-secret-provider"
EOF

1) Enable auto rotation and set the polling interval as desired. The Azure Key Vault Provider for Secrets Store CSI Driver polls for changes and updates the pod mount and the Kubernetes secret.

az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation --rotation-poll-interval 1m

This will not update the environment variables sync’d to the pod mounted secret. To do so, you need to install Reloader.

2) I prefer to install with the helm chart

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install stakater/reloader --generate-name

3) View the current secret value mounted to the pod, Kubernetes secret and environment variable.

## print a test secret 'ExampleSecret' held in secrets-store
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- cat /mnt/secrets-store/ExampleSecret; echo
## Display the environment variables sync'd with the mounted secret
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- printenv EXAMPLE_SECRET

4) Test the update of the key vault secret and run the same commands to see what has been updated.

az keyvault secret set --name ExampleSecret --vault-name $keyVaultName  --value Secret10

# wait a minute to view so that the autoration polling can recognize the update
## print a test secret 'ExampleSecret' held in secrets-store
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- cat /mnt/secrets-store/ExampleSecret; echo
## Display the environment variables sync'd with the mounted secret
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- printenv EXAMPLE_SECRET

As you can see, the pod mount secret is updated to the newer value, but the environment variable is no updated.

5) The pod yaml has to be updated to include the annotation that will enable Reloader can work with the k8s deployment. This will eseentially restart the pods.

# Create a Pod that mounts the secret from Azure Key Vault as a file and environment variable
cat << EOF | kubectl apply -f -
kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline-uami
  namespace: $keyVaultDemoNamespace
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  containers:
    - name: busybox
      image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
      command:
        - "/bin/sleep"
        - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
      env:
      - name: EXAMPLE_SECRET
        valueFrom:
          secretKeyRef:
            name: example-secret
            key: examplesecretkey
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-rkimkv-secret-provider"
EOF

6) Add new secret value so that Reloader will notice a change

az keyvault secret set --name ExampleSecret --vault-name $keyVaultName  --value Secret10

You can notice the pods in the deployment doing a rollout restart

And the view the pod mount volume and the environment variable

As you see in the last command I am able to print out the updated environment variable.

To recap and to make clear:

  1. The enabling of the secret rotation will update the pod volume mount to the secret. This is handled by the Azure Key Vault Provider for Secret Store SCI Driver. The deployment doesn’t need any restart.
  2. The Reloader is required to update the environment variable which essentially restarts the pods in the k8s deployment.

Hope this insightful and made some points more clearer in the Microsoft documentation I have referenced.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s