Azure Kubernetes Service Horizontal Pod Autoscaling Issue Resolution

Issue: In my Azure Kubernetes Service cluster, the horizontal pod autoscaler is not working due to resource cpu on pods is unknown.

Horizontal Pod Autoscaling increases (and decreases) the number of replicas (or pods) on deployments based on cpu usage. I have come across this issue a number of times when setting up HPA.

HPA not working – Events:

  Type     Reason               Age                     From                       Message
  —-     ——               —-                    —-                       ——-
Warning  FailedGetPodsMetric  3m46s (x3719 over 15h)  horizontal-pod-autoscaler  unable to get metric packets-per-second: unable to fetch metrics from custom metrics API: no custom metrics API (custom.metrics.k8s.io) registered

Ignore the get metrics packets-per second metric as that is not relevant for basic HPA.

kubectl get deployments

NAME             READY   UP-TO-DATE   AVAILABLE   AGE

details-v1       1/1     1            1           455d
productpage-v1   5/5     5            5           455d
ratings-v1       1/1     1            1           455d
reviews-v1       1/1     1            1           455d
reviews-v2       1/1     1            1           455d
reviews-v3       1/1     1            1           455d

Check that the metrics server is running. It is installed by default with an AKS cluster deployment.

kubectl get deploy,svc -n kube-system | egrep metrics-server

kubectl get –raw “/apis/metrics.k8s.io/v1beta1/nodes” 

Data is returned from all nodes, so metrics server is working 

One observation is that the k8s deployment does not have any resources defined under containers. Resources 

Resolution 

Ensure to include 

spec: 
      volumes: 
        - name: tmp 
          emptyDir: {} 
      containers: 
        - name: productpage 
          image: 'docker.io/istio/examples-bookinfo-productpage-v1:1.15.0' 
          ports: 
            - containerPort: 9080 
              protocol: TCP 
          resources: 
            limits: 
              cpu: 100m 
              memory: 128Mi 
            requests: 
              cpu: 50m 
              memory: 64Mi

Verify: 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s