Cluster Bootstrapping in a Multi-Cluster Argo CD Environment (Part 1)

Using Git repository tags to declaratively define the mappings between workloads and managed clusters.

The article, Argo CD for Cluster Administration by Example (Part 2), provides a solution in a multi-cluster Argo CD environment to deploy workloads to clusters. This solution has some nice features:

• Workloads are defined declaratively in Git
• Provides an authoritative list of clusters and their features; i.e., represented by their labels
• Workload can be deployed to particular clusters using cluster labels, i.e., provides a mapping between workloads and clusters
• The clusters, the workloads, and their mappings can be visualized through a web UI

It, on the other hand, has a problem; the mapping between workloads and clusters (via a manually applied applicationset per workload) is not declaratively defined in Git.

The Argo CD documentation, Cluster Bootstrapping, provides a solution; the apps of apps pattern.

This guide is for operators who have already installed Argo CD, and have a new cluster and are looking to install many apps in that cluster.

There’s no one particular pattern to solve this problem, e.g. you could write a script to create your apps, or you could even manually create them. However, users of Argo CD tend to use the app of apps pattern.

The trouble with this documentation is that it illustrates the pattern for a single-cluster Argo CD environment. Here we explore the app of apps pattern in a multi-cluster Argo CD environment.

We will use a hypothetical setup diagramed below where we simulate four managed clusters, a, b, c, and d, distributed across two regions and two environments. They are further differentiated as having GPU node pools or not.

The solution will consist of a Git repository tag that declaratively defines the mapping between workloads and managed clusters. The sample workloads are:

• common: Deployed identically to all clusters
• environment: Deployed to all clusters where the workloads have variations based on environment
• gpu-environment: Only deployed to clusters with GPU node pools in the testing environment and region region-1 where the workloads have variations based on environment and gpu type

Prerequisites

If you wish to follow along, you will need the multi-cluster Argo CD setup described in the article Argo CD for Cluster Administration by Example (Part 2).

Here we can see all the clusters; including the hub cluster represented by the name in-cluster.

common

Let us first consider the common workload consisting of the:

• common namespace
• hello configmap in common namespace

note: For the purposes of this article, workloads will be represented as workload-specific namespaces and configmaps.

It is declaratively defined in the larkintuckerllc/common-app Git repository with plain Kubernetes manifests stored in the manifests folder. Being plain Kubernetes manifests it is necessarily deployed identically.

It is also versioned using Git tags; here 0.1.1 is the latest version.

app-of-apps

Let us first consider the app-of-apps resource; it is not so much a workload but rather a representation of the mapping between workloads and managed clusters.

It is declaratively defined in the larkintuckerllc/app-of-apps Git repository with a Helm chart stored in the manifests folder.

It is versioned in the same way as the common workload is; i.e., using Git tags; at this point in the article the tag is 0.1.3 (browse files).

note: For completeness, the Helm chart versions in Chart.yaml is maintained in to be aligned with the Git tag (but it the Git tag that actually matters here).

The manifests/template/required.yaml file uses a nifty Helm trick to enforce required name value; as we will see shortly this is the name of the cluster to deploy the workloads to.

{{- $_ := required "name is a required value" .Values.name -}}

The manifests/templates/common-application.yaml file declares that the common workload version 0.1.1 is to be deployed to a cluster.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  name: "{{ .Values.name }}-common"
  namespace: argocd
spec:
  destination:
    name: "{{ .Values.name }}"
    namespace: common
  project: default
  source:
    path: manifests
    repoURL: https://github.com/larkintuckerllc/common-app
    targetRevision: 0.1.1
  syncPolicy:
    automated:
      allowEmpty: true
      prune: true
      selfHeal: true

Some observations:

• The name value is used in both the name of the application as well as the destination (cluster)
• The finalizer ensures that if we remove this file, the common workload is removed from all the clusters
• This automated sync policy allows for the most aggressive alignment of the resources in the managed cluster to what is declared in the workload’s Git repository

apps ApplicationSet

Finally, to deploy the versioned app-of-apps resource, which provides for the mapping between workloads and clusters. We deploy the following applicationset (apps-applicationset.yaml) to the hub cluster.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
  - clusters:
      selector:
        matchLabels:
          argocd.argoproj.io/secret-type: cluster
  template:
    metadata:
      name: "{{.name}}-apps"
      namespace: argocd
    spec:
      destination:
        namespace: argocd
        name: in-cluster
      project: default
      source:
        helm:
          valuesObject:
            name: "{{.name}}"
        path: manifests
        repoURL: https://github.com/larkintuckerllc/app-of-apps
        targetRevision: 0.1.3
      syncPolicy:
          automated:
            allowEmpty: true
            prune: true
            selfHeal: true 

Some observations:

• The cluster generator generates one one application, named [CLUSTER]-apps, per cluster; the label match ensures that the hub cluster (in-cluster) is excluded
• Because the destination of these applications, e.g., a-apps, is the hub (in-cluster) cluster, the application resources defined in app-of-apps are also created on the hub cluster, e.g, the a-common application
• Through the Helm name value, assigned the name template parameter, the destination of these secondary applications, e.g., a-common, is the corresponding managed cluster
• This automated sync policy allows for the most aggressive alignment of the secondary applications in the cluster to what is declared in the app-of-apps’ Git repository
• When using applicationsets to generate applications, finalizers are automatically added (so no need to include them in the application template). Also, removing an applicationset always removes its generated applications (no need for a finalizer)

Deploying the applicationset.

$ kubectl apply -f apps-applicationset.yaml --context=kind-hub

We see that for each cluster there is a pair of applications.

$ kubectl get applications -n argocd --context=kind-hub
NAME       SYNC STATUS   HEALTH STATUS
a-apps     Synced        Healthy
a-common   Synced        Healthy
b-apps     Synced        Healthy
b-common   Synced        Healthy
c-apps     Synced        Healthy
c-common   Synced        Healthy
d-apps     Synced        Healthy
d-common   Synced        Healthy

In summary (visualized below for clusters a and b) the flow is:

1. We deploy the apps applicationset referencing the apps-of-apps repository v0.1.3 tag
2. The apps applicationset generates the per-managed cluster [CLUSTER]-apps applications referencing the apps-of-apps repository v0.1.3 tag; each with a destination of the hub cluster
3. Each of the [CLUSTER]-apps applications, in turn, deploy the per-cluster [CLUSTER]-common applications referencing the common-app repository v0.1.1 tag; each with a destination of the appropriate managed cluster
4. Each of the [CLUSTER]-common applications, finally, deploy the common workload to the appropriate managed cluster; each based on the common-app repository v0.1.1 tag

Indeed, we can see the final result of the common application (represented by the hello configmap) on the a cluster.

$ kubectl get configmap -n common --context=kind-a
NAME               DATA   AGE
hello              1      9m31s
kube-root-ca.crt   1      9m32s

From the web UI, we can filter by cluster, here a, to which workloads are deployed to it.

Interestingly, there is currently no mechanism to see which clusters a particular, i.e., common, workload is deployed to.

Next Steps

In the Part 2, we will start by improving on the solution by allowing us to see which clusters a particulare workload is deployed to. We will then describe and deploy the environment and gpu-environment workloads.