Scenario: The objective is to have the environment variable in the Pod updated when the secret is updated in Azure Key Vault. You already have a secret in an Azure Key Vault mounted to your pod and sync’d as an environment variable; however, when you update the secret value in Azure Key Vault, the environment variable is not updated.
This blog post is an extension to the scenario setup in my previous blog post How To Mount Azure Key Vault Secret to Pods in Azure Kubernetes Service.
This blog post will go over the setup of an open source tool called Reloader to watch for changes on the synced Kubernetes Secret and perform rolling upgrades on pods. And to go over a test scenario to see the resulting behaviour.
References:
https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver#enable-and-disable-autorotation
https://github.com/stakater/Reloader
0) The scenario starts with the setup as defined in this my previous blog post How To Mount Azure Key Vault Secret to Pods in Azure Kubernetes Service
And to add the following K8s deployment
# Create a Pod that mounts the secret from Azure Key Vault as a file and environment variable
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: busybox-deployment-secrets-store-inline-uami
namespace: $keyVaultDemoNamespace
labels:
app: busybox
spec:
replicas: 1
selector:
matchLabels:
app: busybox
template:
metadata:
labels:
app: busybox
spec:
containers:
- name: busybox
image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
command:
- "/bin/sleep"
- "20000"
volumeMounts:
- name: secrets-store01-inline
mountPath: "/mnt/secrets-store"
readOnly: true
env:
- name: EXAMPLE_SECRET
valueFrom:
secretKeyRef:
name: example-secret
key: examplesecretkey
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-rkimkv-secret-provider"
EOF
1) Enable auto rotation and set the polling interval as desired. The Azure Key Vault Provider for Secrets Store CSI Driver polls for changes and updates the pod mount and the Kubernetes secret.
az aks addon update -g myResourceGroup -n myAKSCluster2 -a azure-keyvault-secrets-provider --enable-secret-rotation --rotation-poll-interval 1m
This will not update the environment variables sync’d to the pod mounted secret. To do so, you need to install Reloader.
2) I prefer to install with the helm chart
helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install stakater/reloader --generate-name
3) View the current secret value mounted to the pod, Kubernetes secret and environment variable.
## print a test secret 'ExampleSecret' held in secrets-store
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- cat /mnt/secrets-store/ExampleSecret; echo
## Display the environment variables sync'd with the mounted secret
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- printenv EXAMPLE_SECRET
4) Test the update of the key vault secret and run the same commands to see what has been updated.
az keyvault secret set --name ExampleSecret --vault-name $keyVaultName --value Secret10
# wait a minute to view so that the autoration polling can recognize the update
## print a test secret 'ExampleSecret' held in secrets-store
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- cat /mnt/secrets-store/ExampleSecret; echo
## Display the environment variables sync'd with the mounted secret
kubectl exec busybox-secrets-store-inline-uami -n $keyVaultDemoNamespace -- printenv EXAMPLE_SECRET
As you can see, the pod mount secret is updated to the newer value, but the environment variable is no updated.
5) The pod yaml has to be updated to include the annotation that will enable Reloader can work with the k8s deployment. This will eseentially restart the pods.
# Create a Pod that mounts the secret from Azure Key Vault as a file and environment variable
cat << EOF | kubectl apply -f -
kind: Pod
apiVersion: v1
metadata:
name: busybox-secrets-store-inline-uami
namespace: $keyVaultDemoNamespace
annotations:
reloader.stakater.com/auto: "true"
spec:
containers:
- name: busybox
image: k8s.gcr.io/e2e-test-images/busybox:1.29-1
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store01-inline
mountPath: "/mnt/secrets-store"
readOnly: true
env:
- name: EXAMPLE_SECRET
valueFrom:
secretKeyRef:
name: example-secret
key: examplesecretkey
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-rkimkv-secret-provider"
EOF
6) Add new secret value so that Reloader will notice a change
az keyvault secret set --name ExampleSecret --vault-name $keyVaultName --value Secret10
You can notice the pods in the deployment doing a rollout restart
And the view the pod mount volume and the environment variable
As you see in the last command I am able to print out the updated environment variable.
To recap and to make clear:
- The enabling of the secret rotation will update the pod volume mount to the secret. This is handled by the Azure Key Vault Provider for Secret Store SCI Driver. The deployment doesn’t need any restart.
- The Reloader is required to update the environment variable which essentially restarts the pods in the k8s deployment.
Hope this insightful and made some points more clearer in the Microsoft documentation I have referenced.
Roy Kim on Azure and Microsoft 365 