A Confusing Aspect of the Kubernetes API Explained

To explain, we end up learning about Kubernetes API Groups, API Versions, and Resource Types.

This article was written using Kubernetes 1.20.X.

$ kubectl version                               
Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.8-dispatcher", GitCommit:"283881f025da4f5b3cefb6cd4c35f2ee4c2a79b8", GitTreeState:"clean", BuildDate:"2021-09-14T05:14:54Z", GoVersion:"go1.15.13", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.9-gke.1001", GitCommit:"1fe18c314ed577f6047d2712a9d1c8e498e22381", GitTreeState:"clean", BuildDate:"2021-08-23T23:06:28Z", GoVersion:"go1.15.13b5", Compiler:"gc", Platform:"linux/amd64"}

The Confusing Aspect

The scenario is that we are looking to take advantage of the behavior feature of the Kubernetes Horizontal Pod Autoscaler.

Starting from v1.18 the v2beta2 API allows scaling behavior to be configured through the HPA behavior field. Behaviors are specified separately for scaling up and down in scaleUp or scaleDown section under the behavior field.

— Kubernetes — Horizontal Pod Autoscaler

Specifically, we create a horizontalpodautoscaler (hpa) Resource using the following manifest.

We then turn around and get this hpa Resource.

$ kubectl get hpa debug -o yaml
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
...
name: debug
namespace: default
...
spec:
maxReplicas: 3
minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: debug
targetCPUUtilizationPercentage: 80
status:
currentCPUUtilizationPercentage: 1
currentReplicas: 2
desiredReplicas: 2

Things to observe:

  • While the apiVersion in the manifest is autoscaling/v2beta2, the output shows the Resource as being apiVersion autoscaling/v1
  • The output does not show the behavior feature that we specified

API Groups, API Versions, and Resource Types

To explain what happened in and properly handle this scenario, we need to understand Kubernetes API Groups, API Versions, and Resource Types.

To make it easier to evolve and to extend its API, Kubernetes implements API groups that can be enabled or disabled.

— Kubernetes — API Overview

The authoritative mechanism to list the API Groups supported by a Kubernetes Cluster is to directly query the apis endpoint. Here we see that one of them is autoscaling.

$ kubectl get --raw /apis | jq
{
"kind": "APIGroupList",
"apiVersion": "v1",
"groups": [
...
{
"name": "autoscaling",
"versions": [
{
"groupVersion": "autoscaling/v1",
"version": "v1"
},
{
"groupVersion": "autoscaling/v2beta1",
"version": "v2beta1"
},
{
"groupVersion": "autoscaling/v2beta2",
"version": "v2beta2"
}
],
"preferredVersion": {
"groupVersion": "autoscaling/v1",
"version": "v1"
}
},
...

The output also indicates the three API Versions supported by this API Group.

To make it easier to eliminate fields or restructure resource representations, Kubernetes supports multiple API versions, each at a different API path, such as /api/v1 or /apis/rbac.authorization.k8s.io/v1alpha1.

Versioning is done at the API level rather than at the resource or field level to ensure that the API presents a clear, consistent view of system resources and behavior, and to enable controlling access to end-of-life and/or experimental APIs.

— Kubernetes — The Kubernetes API

The API Group and API Version supports particular Resource Types.

A resource type is the name used in the URL (pods, namespaces, services)

— Kubernetes — Kubernetes API Concepts

Here we can query the apis/autoscaling/v1 endpoint to observe that it supports a horizontalpodautoscalers Resource Type.

$ kubectl get --raw /apis/autoscaling/v1 | jq       
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "autoscaling/v1",
"resources": [
{
"name": "horizontalpodautoscalers",
"singularName": "",
"namespaced": true,
"kind": "HorizontalPodAutoscaler",
"verbs": [
"create",
"delete",
"deletecollection",
"get",
"list",
"patch",
"update",
"watch"
],
"shortNames": [
"hpa"
],
"categories": [
"all"
],
"storageVersionHash": "oQlkt7f5j/A="
},
{
"name": "horizontalpodautoscalers/status",
"singularName": "",
"namespaced": true,
"kind": "HorizontalPodAutoscaler",
"verbs": [
"get",
"patch",
"update"
]
}
]
}

Things to observe:

  • By querying the v2beta1 and v2beta2 endpoints, we see that they also support horizontalpodautoscalers Resource Types

Given the differences in the APIs, e.g., only the v2beta2 API Version supports the behavior feature, one would think that these three horizontalpodautoscalers are different Resource Types. Or are they?

It turns out that there is only one horizontalpodautoscalers Resource Type; not one per API Version.

API resources are distinguished by their API group, resource type, namespace (for namespaced resources), and name. The API server handles the conversion between API versions transparently: all the different versions are actually representations of the same persisted data. The API server may serve the same underlying data through multiple API versions.

For example, suppose there are two API versions, v1 and v1beta1, for the same resource. If you originally created an object using the v1beta1 version of its API, you can later read, update, or delete that object using either the v1beta1 or the v1 API version.

— Kubernetes — The Kubernetes API

Knowing all this, we can now get the hpa Resource using the appropriate API Version using the more complete format: [Resource Type].[API Version].[API Group].

$ kubectl get hpa.v2beta2.autoscaling debug -o yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
...
name: debug
namespace: default
...
spec:
behavior:
scaleDown:
policies:
- periodSeconds: 60
type: Pods
value: 4
- periodSeconds: 60
type: Percent
value: 10
selectPolicy: Max
scaleUp:
policies:
- periodSeconds: 15
type: Pods
value: 4
- periodSeconds: 15
type: Percent
value: 100
selectPolicy: Max
stabilizationWindowSeconds: 0
maxReplicas: 3
metrics:
- resource:
name: cpu
target:
averageUtilization: 80
type: Utilization
type: Resource
minReplicas: 2
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: debug
...

Interestingly enough, kubectl describe, appropriately identifies the appropriate API Version.

$ kubectl describe hpa debug
Name: debug
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Sat, 09 Oct 2021 15:45:36 -0400
Reference: Deployment/debug
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 1% (1m) / 80%
Min replicas: 2
Max replicas: 3
Behavior:
Scale Up:
Stabilization Window: 0 seconds
Select Policy: Max
Policies:
- Type: Pods Value: 4 Period: 15 seconds
- Type: Percent Value: 100 Period: 15 seconds
Scale Down:
Select Policy: Max
Policies:
- Type: Pods Value: 4 Period: 60 seconds
- Type: Percent Value: 10 Period: 60 seconds
Deployment pods: 2 current / 2 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica count
Events: <none>

Wrap Up

This aspect has confused me for awhile; just never dug into it until now. Hope you found it useful.

Broad infrastructure, development, and soft-skill background