Vcluster

Kubernetes

Virtual Cluster

Tutorial

vCluster by Example (Part 3)

John Tucker

5 min readMay 16, 2025

Exploring reliability configurations.

This is a continuation of this series starting with the article vCluster by Example (Part 1).

External etcd Backing Store

From Backing Store.

By default, vCluster uses an embedded SQLite database as the backing store. This option is great for smaller sandbox virtual clusters, but for production, it’s recommended to use a different backing store option.

Looking at the backing store options, the only one that is provided with the Core distribution is the external etcd option.

note: The documentation, Embedded SQLite, would suggest that one is required to explicitly disable SQLite in order to use other backing store options. It turns out that this is inaccurate, i.e., simply specifying a separate backing store is sufficient.

Building on our earlier vcluster.yaml file that we used to create a vCluster instance, we now have.

controlPlane:
  distro:
    k8s:
      version: v1.31.8
  backingStore:
    etcd:
      deploy:
        enabled: true

We create a my-etc-vcluster using the following command.

$ helm install my-etc-vcluster loft/vcluster \
--version=0.24.1 \
--values vcluster.yaml \
--namespace=team-x \
--create-namespace

Lets look under the hood of this installation using the following command (only showing the most relevant output).

$ helm status my-etc-vcluster --namespace=team-x --show-resources
...
==> v1/Deployment
NAME              READY   UP-TO-DATE   AVAILABLE   AGE
my-etc-vcluster   1/1     1            1           112s
...
==> v1/StatefulSet
NAME                   READY   AGE
my-etc-vcluster-etcd   1/1     112s

Previously, the active resources of the vCluster instance consisted of a single pod for the my-vcluster statefulset. Here the active resources are a single pod for the my-etc-vcluster deployment and a single pod for the my-etc-vcluster-etcd statefulset.

note: There are other interesting customizations that we might want to consider, e.g., setting the container resources.

High Availability

From Deploy in High Availability.

By default, vCluster runs one instance of each of its components. This deployment method is recommended for any uses cases that are very ephemeral (e.g. dev environments, CI/CD, etc.), but for most production use cases, you will want to run vCluster with more redundancy. We recommend deploying vCluster in High Availability (HA) mode to run multiple copies of the vCluster components so that the virtual cluster is more resistant to partial failures.

Building on our earlier vcluster.yaml file that we used to create a vCluster instance, we now have.

controlPlane:
  distro:
    k8s:
      version: v1.31.8
  backingStore:
    etcd:
      deploy:
        enabled: true
        statefulSet:
          highAvailability:
            replicas: 3
  statefulSet:
    highAvailability:
      replicas: 3

note: It is a bit strange here to continue refer to the vCluster instance’s pod for the Kubernetes API server, etc. as a statefulset when it changes to a deployment when using the external etcd backing store.

We create a my-ha-etc-vcluster using the following command.

$ helm install my-ha-etc-vcluster loft/vcluster \
--version=0.24.1 \
--values vcluster.yaml \
--namespace=team-x \
--create-namespace

So, the deployed resources are the same; just the replica count goes from one to three.

$ kubectl get all --namespace=team-x
...
NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-ha-etc-vcluster   3/3     3            3           2m9s
...
NAME                                       READY   AGE
statefulset.apps/my-ha-etc-vcluster-etcd   3/3     2m9s

Failure Domains

Now that we have set up high availability with three replicas, we want to ensure that the pods for both the deployment (Kubernetes API server, etc) and statefulset (etcd):

Do not land on the same node
Are spread across the host cluster node’s failure domains

To illustrate this, we will need to change out the host cluster to have three nodes; each in a different failure domain. For a kind cluster, the following cluster.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: host
nodes:
- role: control-plane
  image: kindest/node:v1.30.10@sha256:4de75d0e82481ea846c0ed1de86328d821c1e6a6a91ac37bf804e5313670e507
- role: worker
  image: kindest/node:v1.30.10@sha256:4de75d0e82481ea846c0ed1de86328d821c1e6a6a91ac37bf804e5313670e507
  labels:
    topology.kubernetes.io/zone: a
- role: worker
  image: kindest/node:v1.30.10@sha256:4de75d0e82481ea846c0ed1de86328d821c1e6a6a91ac37bf804e5313670e507
  labels:
    topology.kubernetes.io/zone: b
- role: worker
  image: kindest/node:v1.30.10@sha256:4de75d0e82481ea846c0ed1de86328d821c1e6a6a91ac37bf804e5313670e507
  labels:
    topology.kubernetes.io/zone: c

and created with this command does the trick.

$ kind create cluster --config cluster.yaml

After a bit of trial and error and reading the document Pod Topology Spread Constraints, we build on our earlier vcluster.yaml file that we used to create a vCluster instance, we now have.

controlPlane:
  distro:
    k8s:
      version: v1.31.8
  backingStore:
    etcd:
      deploy:
        enabled: true
        statefulSet:
          highAvailability:
            replicas: 3
          scheduling:
            topologySpreadConstraints:
              - labelSelector:
                  matchLabels:
                    app: vcluster-etcd
                maxSkew: 1
                topologyKey: topology.kubernetes.io/zone
                whenUnsatisfiable: ScheduleAnyway
  statefulSet:
    highAvailability:
      replicas: 3
    scheduling:
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              app: vcluster
          maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway

We create a my-fd-ha-etc-vcluster using the following command.

$ helm install my-fd-ha-etc-vcluster loft/vcluster \
--version=0.24.1 \
--values vcluster.yaml \
--namespace=team-x \
--create-namespace

Listing the pods and their nodes we indeed see that they are spread across the three nodes where each node is in a different failure domain (topology.kubernetes.io/zone).

% kubectl get pods --namespace=team-x -o wide
NAME                                     READY   STATUS    RESTARTS   AGE   IP            NODE           NOMINATED NODE   READINESS GATES
my-fd-ha-etc-vcluster-64b98f7dc9-8nct5   1/1     Running   0          75s   10.244.1.10   host-worker2   <none>           <none>
my-fd-ha-etc-vcluster-64b98f7dc9-rbg6l   1/1     Running   0          75s   10.244.2.10   host-worker3   <none>           <none>
my-fd-ha-etc-vcluster-64b98f7dc9-xtgvn   1/1     Running   0          75s   10.244.3.9    host-worker    <none>           <none>
my-fd-ha-etc-vcluster-etcd-0             1/1     Running   0          75s   10.244.3.10   host-worker    <none>           <none>
my-fd-ha-etc-vcluster-etcd-1             1/1     Running   0          75s   10.244.1.11   host-worker2   <none>           <none>
my-fd-ha-etc-vcluster-etcd-2             1/1     Running   0          74s   10.244.2.11   host-worker3   <none>           <none>

Resources

Up until this point we have used the default pod resources for the pods for both the deployment (Kubernetes API server, etc) and statefulset (etcd); we now will explicitly set them.

Looking at the document vcluster.yaml configuration reference, we see the default resources for the deployment (Kubernetes API server, etc) is:

and for the statefulset (etcd).

Here we arbitrarily double the resources to illustrate that we can.

controlPlane:
  distro:
    k8s:
      version: v1.31.8
  backingStore:
    etcd:
      deploy:
        enabled: true
        statefulSet:
          highAvailability:
            replicas: 3
          resources:
            limits:
              cpu: 40m
              memory: 300Mi
            requests:
              cpu: 40m
              memory: 300Mi
          scheduling:
            topologySpreadConstraints:
              - labelSelector:
                  matchLabels:
                    app: vcluster-etcd
                maxSkew: 1
                topologyKey: topology.kubernetes.io/zone
                whenUnsatisfiable: ScheduleAnyway
  statefulSet:
    highAvailability:
      replicas: 3
    resources:
      limits:
        cpu: 400m
        ephemeral-storage: 16Gi
        memory: 4Gi
      requests:
        cpu: 400m
        ephemeral-storage: 16Gi
        memory: 4Gi
    scheduling:
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              app: vcluster
          maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway

We create a my-rsrc-fd-ha-etc-vcluster using the following command.

$ helm install my-rsrc-fd-ha-etc-vcluster loft/vcluster \
--version=0.24.1 \
--values vcluster.yaml \
--namespace=team-x \
--create-namespace

Did not show it here, but we can check the resources of the deployment and statefulset and observe that these resources were applied.

All Done

Whew!