Flux Multi-Cluster Multi-Tenant by Example (Continued)

A continuation of a walk-through of using Flux to deliver applications in a multi-cluster multi-tenant Kubernetes environment.

This article is a continuation of the article Flux Multi-Cluster Multi-Tenant by Example.

Bootstrap Multi-Cluster Single-Tenant

Here we refactor the Terraform configuration to support bootstrapping multiple clusters but sticking with the single-tenant case. In order to follow along, one will need two Kubernetes Clusters and another empty GitHub Repository.

We download the larkintuckerllc /hello-flux-multi-single-bootstrap Repository.

We create a terraform.tfvar file in the root of the downloaded Repository with the following key / value pairs; we update to match our environment.

Please note: Because Terraform providers cannot be dynamically created, we unfortunately have to duplicate code in the Terraform configuration.

From the root of the downloaded Repository, we initialize Terraform:

$ terraform init

We then create the resources.

$ terraform apply

As with the details for the single-cluster single-tenant case, the output of the terraform apply command has key / value pairs for us to create three files per cluster in the newly created Git Repository. Likewise, we then continue to grant the Flux components in each Cluster read-only access to the newly created GitHub Repository

Deploy a Customized per Cluster Sample Application

Now that we have Flux bootstrapped into both Clusters we can use it to deploy a sample application into each; we however customize the application per Cluster. This application consists of a single Namespace (sample) and Pod in it (sample). We create the following files in a new apps/sample folder in the newly created Git Repository.

Things to observe:

• Here we use the DownwardAPIVolumeFile to expose the Pod information to the Container; tried to do the same with environment variables with no success

The last step is to configure the Flux components in each of the Clusters to synchronize this application; this time customizing it per cluster. We do this by creating a file in the clusters/cluster-1 folder.

We then create a similar file in the clusters/cluster-2 folder; just changing the cluster-name annotation to cluster-2.

After a minute or so, the application (one per Cluster) is deployed and we can indeed see that they are customized.

$ kubectl exec -it -n sample sample -- /bin/cat /etc/podinfo/annotations
cluster-name="cluster-1"
[OMITTED]

Deploy a Sample Application from Another Repository

In the previous examples, we have deployed an application by adding the manifests to the newly created Git Repository. Here, instead, we create another application Git Repository; e.g., hello-flux-sample2.

We then create the appropriate Flux configurations to synchronize this application by creating files in the clusters/cluster-1 (and cluster-2) folders. First we create a GitRepository resource:

and then the Kustomization resource.

Enable Multi-Cluster Multi-Tenant

In the previous examples we deployed applications by creating resources in the newly created Git Repository; going forward we will refer to it as the Fleet Repository. The two example were:

• Creating application and Kustomization resources in the Fleet Repository
• Creating GitRepository (referencing to what we will now call an App Repository) and Kustomization resources in the Fleet Repository

We can build off the bootstrapped multi-cluster single-tenant example to enable the final multi-cluster multi-tenant example. To do this we introduce a third category of Git Repository; a Tenant Repository.

— Flux — flux2-multi-tenancy

Things to observe:

• The Fleet Clusters block represents the Clusters, e.g., cluster-1 and cluster-2
• Again, the Fleet Repository (managed by ops team) is what we previously referred to as the newly created Git Repository
• The App Repositories (managed by dev teams), e.g., similar to hello-flux-sample2, contain the application resources
• The newly introduced Tenant Repository (managed by dev teams) will be used to create the GitRepository and Kustomization resources to deploy applications

Following the flux2-multi-tenancy example, we will restrict the dev team to only being able to deploy applications to the apps Namespace.

Please note: The flux2-multi-tenancy example uses the Flux CLI to create the necessary manifests; making it super confusing to follow. We instead will create the necessary resources ourselves.

The good news is that the solution only involves, other than standard Kubernetes resources, the use of the GitRepository and Kustomize resources that we have already seen. The bad news is that there are a number of them.

To help visualize the work, we will be working with the following Repositories with arrows representing the Flux Kustomization resources linking things together.

We start with an example App Repository; hello-flux-app1 (only deploys a Pod in the apps Namespace). One difference between this and the earlier hello-flux-sample2 Repository is that we do not create Namespace resources in App Repositories.

We next create a new private Tenant Repository, hello-flux-tenant1, and populate it with the following files.

app1-git-repository.yaml

app1-kustomization.yaml

kustomization.yaml

Things to observe:

• For each additional application, we would create a pair of GitRepository and Kustomization resource manifests and then add references to them to the third file above
• It is unfortunate that the Flux Kustomization resource shares the same kind as the Kustomize resource (the third file above)

The next step is to add the appropriate resources to the Fleet Repository, hello-flux-repository, to use the Tenant Repository. First we create resources in the tenants/tenant1 folder; starting with:

apps-Namespace.yaml

tenant1-sa.yaml

tenant1-reconciler-rb.yaml

Things to observe:

• It is not entirely clear why the toolkit.fluxcd.io/tenant label is needed for each of the resources
• The ServiceAccount and and associated RoleBinding will be used later when configuring the Kustomization resource; notice that this configuration gives the ServiceAccount access to manage any resources in the apps Namespace
• It is not entirely clear what the gotk:apps:reconciler User is (or if it is even used)

We next create a GitRepository referencing the Tenant Repository; update the URL to match our Repository.

tenant1-git-repository.yaml

We create the associated Flux Kustomization.

tenant1-kustomization.yaml

Things to observe:

• Here we introduce the ServiceAccount which causes the Flux Components to impersonate it when managing resources; in this case limited to managing resources in the apps Namespace

Finally, we include all the resources in a Kustomize Kustomization.

kustomization.yaml

Next we create a Flux Kustomization for cluster-1 (could repeat for cluster-2 if we desired).

clusters/cluster-1/tenant1-kustomization.yaml

Because the Tenant Repository is private, we now need to grant the Flux components read-only access to it. Run the following command, updated with the appropriate GitHub owner and repository.

flux create secret git tenant1 --url=ssh://git@github.com/larkintuckerllc/hello-flux-tenant1 --namespace=apps --export > tenant1-secret.yaml

Add the value of the key identity.pub in the generated manifest as a read-only deploy key in the Tenant Repository (hello-flux-tenant1).

Create the Kubernetes Secret with the following command.

$ kubectl apply -f tenant1-secret.yaml

If all goes well, we can examine the ready Kustomizations in the flux-system Namespace:

$ flux get kustomizations
NAME        READY MESSAGE                                                         REVISION                                      SUSPENDED
flux-system True  Applied revision: main/6f6b9e80b8e9e37ad08c1b52aa255abc57414885 main/6f6b9e80b8e9e37ad08c1b52aa255abc57414885 False
sample      True  Applied revision: main/6f6b9e80b8e9e37ad08c1b52aa255abc57414885 main/6f6b9e80b8e9e37ad08c1b52aa255abc57414885 False
sample2     True  Applied revision: main/f7c86771db5ab5b1c7fe0e27ebeefb27c02dabcb main/f7c86771db5ab5b1c7fe0e27ebeefb27c02dabcb False
tenant1     True  Applied revision: main/6f6b9e80b8e9e37ad08c1b52aa255abc57414885 main/6f6b9e80b8e9e37ad08c1b52aa255abc57414885 False

Things to observe:

• The flux command defaults to showing resources in the flux-system Namespace
• The tenant1 Kustomization here links to the tenant1 directory in the Fleet Repository (hello-flux-repository)
• The other Kustomizations are from the initial bootstrap and earlier examples

We can examine the ready Kustomizations in the apps Namespace:

$ flux get kustomizations -n apps
NAME    READY MESSAGE                                                         REVISION                                      SUSPENDED
app1    True  Applied revision: main/24509ae948dd57bc58cd7514912cecc2dcb7cc43 main/24509ae948dd57bc58cd7514912cecc2dcb7cc43 False
tenant1 True  Applied revision: main/ebde2d7a7200ffd9b77a3fa52d07d7ecc90ac2ba main/ebde2d7a7200ffd9b77a3fa52d07d7ecc90ac2ba False

Things to observe:

• The tenant1 Kustomization here links to the Tenant Repository (hello-flux-tenant1)
• The app1 Kustomization (in the Tenant Repository) links to the App Repository (hello-flux-app1)

Wrap Up

With a bit of effort we have achieved our goal of configuring a multi-cluster multi-tenant Kubernetes environment. Looking back at the Flux documentation, we have touched on the core concepts but there are some extra topics that we might want to explore on our own, e.g, a better solution for managing the Secrets.