Flux Multi-Cluster Multi-Tenant by Example

A walk-through of using Flux to deliver applications in a multi-cluster multi-tenant Kubernetes environment.

Flux — the GitOps family of projects
Flux is a set of continuous and progressive delivery solutions for Kubernetes, and they are open and extensible.

— Flux — Flux

What is GitOps?

Before we dig into Flux, it is important to understand what GitOps is from the perspective of the Flux team.

Principles of GitOps
To start managing your cluster with GitOps workflows, the following must be in place:

#1. The entire system described declaratively.
#2. The canonical desired system state versioned in Git.
#3. Approved changes that can be automatically applied to the system.
#4. Software agents to ensure correctness and alert on divergence.

— Weaveworks — Guide To GitOps

From another broader perspective, this definition of GitOps is essentially the GitOps pull-based deployment strategy.

The Pull-based deployment strategy uses the same concepts as the push-based variant but differs in how the deployment pipeline works. Traditional CI/CD pipelines are triggered by an external event, for example when new code is pushed to an application repository. With the pull-based deployment approach, the operator is introduced. It takes over the role of the pipeline by continuously comparing the desired state in the environment repository with the actual state in the deployed infrastructure. Whenever differences are noticed, the operator updates the infrastructure to match the environment repository.

— GitOps — GitOps


If you wish to follow along, you will need:

  • Kubernetes Cluster 1.16 or newer (here I used Google Kubernetes Engine Standard 1.20.8-gke.900)
  • kubectl CLI; configured with the Cluster
  • Terraform CLI 1.0.5 or newer (here I used 1.0.5)
  • GitHub Account
  • Git CLI
  • The Flux CLI

Bootstrap Single-Cluster Single-Tenant

Here we start with the simplest case; a single-cluster with a single-tenant.

The first step is to create the GitHub Repository to hold the desired system state; here we simply use the GitHub UI to create a private Repository (in my case larkintuckerllc / hello-flux-repository).

Next, we download the larkintuckerllc /hello-flux-single-single-bootstrap Repository. While one can use the flux CLI to bootstrap flux into our Cluster, here we use Terraform for a more (less magic) declarative approach.

We create a terraform.tfvar file in the root of the downloaded Repository with the following key / value pairs; we update to match our environment.

Things to observe:

  • The keys prefixed with github are to be updated based on the newly created GitHub Repository; the branch is whatever Git branch we want to use (main is the new GitHub default)
  • The kubernetes_config_context (the context’s name) can be found in one’s kubeconfig file (assumes one has already configured the kubectl CLI with the Cluster)
  • The kubernetes_cluster_name is an arbitrary string for us to uniquely identify the cluster (will be more relevant as we move into the multi-cluster case)

Please note: If you are wondering what resources this Terraform configuration creates; they are Kubernetes resources in the flux-system Namespace consisting of Flux components (install resources) as well as a Flux bootstrap configuration (sync resources).

From the root of the downloaded Repository, we initialize Terraform:

$ terraform init

We then create the resources.

$ terraform apply

The output of the terraform apply command has the following key / values for us to manually create three files in the newly created Git Repository in the branch specified in the terraform.tfvars file; push the branch to GitHub.

  • install_path and install_content: The Flux components in the Cluster
  • sync_path and sync_content: The configuration resources for the Flux components to determine the desired state from the newly created GitHub Repository
  • kustomize_path and kustomize_content: A kustomization file declaring the aforementioned install and sync resource files

Please note: You can re-run the apply command to re-generate the output and direct it to a file; helpful as the install_content is rather long.

Please note: Other tutorials on bootstrapping Flux with Terraform automate the creation of these files. Here, for security reasons, it is not recommend as it requires creating long-lived GitHub personal access tokens.

At this point, the repository consists of three files.

We now need to grant the Flux components read-only access to the newly created GitHub Repository. Run the following command, updated with the appropriate GitHub owner and repository.

$ flux create secret git flux-system \
--url=ssh://git@github.com/larkintuckerllc/hello-flux-repository \
--export > flux-system-secret.yaml

Add the value of the key identity.pub in the generated manifest as a read-only deploy key in the newly created GitHub repository.

Create the Kubernetes Secret with the following command.

$ kubectl apply -f flux-system-secret.yaml

Please note: Other tutorials on bootstrapping Flux with Terraform automate the creation of this Secret. Here, for security reasons, it is not recommended as it stores GitHub deploy key in the Terraform state.

If all is working as expected, in a few minutes the following command should indicate that the flux-system kustomization is ready (True).

$ flux get kustomizationsNAME        READY MESSAGE                                                         REVISION                                      SUSPENDEDflux-system True  Applied revision: main/e0b4f7aac65cc69a8e9c25f37a866aa15f43801a main/e0b4f7aac65cc69a8e9c25f37a866aa15f43801a False

Custom Resources

Before deploying a sample application into the Cluster, it is instructive to examine the custom resources used to reflect the state of the resources bootstrapped in the flux-system Namespace.

Here we examine the first resource in the gotk-sync.yaml file we created in the GitHub Repository.

This is a GitRepository custom resource.

The GitRepository API defines a source for artifacts coming from Git. The resource exposes the latest synchronized state from Git as an artifact in a gzip compressed TAR archive.

— Flux — Git Repositories

We can infer a lot about GitRepositories simply by describing them.

$ kubectl describe gitrepository flux-system -n flux-system
Name: flux-system
Namespace: flux-system
Labels: kustomize.toolkit.fluxcd.io/name=flux-system
Annotations: kustomize.toolkit.fluxcd.io/checksum: 74af5c50a184a59640d3533cf577f07dd3e66bc5
API Version: source.toolkit.fluxcd.io/v1beta1
Kind: GitRepository
Git Implementation: go-git
Interval: 1m0s
Branch: main
Secret Ref:
Name: flux-system
Timeout: 20s
URL: ssh://git@github.com/larkintuckerllc/hello-flux-repository
Checksum: fe8007ddab6c5bf1329276f943eabb625ae92f4a
Last Update Time: 2021-08-24T10:14:32Z
Path: gitrepository/flux-system/flux-system/df906a5b44afc249bf7bf5f78becef7e89b71dfa.tar.gz
Revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
URL: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/df906a5b44afc249bf7bf5f78becef7e89b71dfa.tar.gz
Last Transition Time: 2021-08-24T10:14:32Z
Message: Fetched revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Reason: GitOperationSucceed
Status: True
Type: Ready
Observed Generation: 1
URL: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/latest.tar.gz
Type Reason Age From Message
---- ------ ---- ---- -------
Normal error 38m (x17 over 43m) source-controller auth secret error: Secret "flux-system" not found
Normal info 32m source-controller Fetched revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa

Things to observe:

  • Here we can infer that the source-controller Flux component is responsible for using GitRepository custom resources to regularly download states of a Git Repository and then expose their latest state on a URL

Next we examine the second resource in the gotk-sync.yaml file we created in the GitHub Repository.

This is a Kustomization custom resource.

The Kustomization API defines a pipeline for fetching, decrypting, building, validating and applying Kubernetes manifests.

— Flux — Kustomization

One interesting thing here is the interval specified here is fairly long, 10 minutes. This interval determines how often to reconcile the resources defined in the Kustomization.

While we previously used the flux get kustomizations command to examine a Kustomization, describing it provides more information.

$ kubectl describe kustomization flux-system -n flux-system
Name: flux-system
Namespace: flux-system
Labels: kustomize.toolkit.fluxcd.io/name=flux-system
Annotations: kustomize.toolkit.fluxcd.io/checksum: 74af5c50a184a59640d3533cf577f07dd3e66bc5
API Version: kustomize.toolkit.fluxcd.io/v1beta1
Kind: Kustomization
Force: false
Interval: 10m0s
Path: ./clusters/cluster-1
Prune: true
Source Ref:
Kind: GitRepository
Name: flux-system
Validation: client
Last Transition Time: 2021-08-24T11:03:45Z
Message: Applied revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Reason: ReconciliationSucceeded
Status: True
Type: Ready
Last Applied Revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Last Attempted Revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Observed Generation: 1
Type Reason Age From Message
---- ------ ---- ---- -------
Normal info 58m kustomize-controller clusterrole.rbac.authorization.k8s.io/crd-controller-flux-system configured
service/source-controller configured
gitrepository.source.toolkit.fluxcd.io/flux-system configured
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io configured
deployment.apps/notification-controller configured
kustomization.kustomize.toolkit.fluxcd.io/flux-system configured
deployment.apps/kustomize-controller configured
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io configured
networkpolicy.networking.k8s.io/allow-scraping configured
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io configured
namespace/flux-system configured
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io configured
networkpolicy.networking.k8s.io/allow-webhooks configured
serviceaccount/source-controller configured
deployment.apps/source-controller configured
networkpolicy.networking.k8s.io/allow-egress configured
deployment.apps/helm-controller configured
serviceaccount/helm-controller configured
serviceaccount/kustomize-controller configured
serviceaccount/notification-controller configured
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-flux-system configured
clusterrolebinding.rbac.authorization.k8s.io/crd-controller-flux-system configured
service/notification-controller configured
service/webhook-receiver configured
customresourcedefinition.apiextensions.k8s.io/buckets.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io configured
Normal info 9m50s (x6 over 58m) kustomize-controller Update completed

Things to observe:

  • Here we can see precisely which resources are being synchronized by this Kustomization custom resource

Deploy a Sample Application

Now that we have Flux bootstrapped into the Cluster, we can use it to deploy a sample application. This application consists of a single Namespace (sample) and Pod in it (sample). We create the following files in a new apps/sample folder in the newly created Git Repository.

Having committed and pushed the update, the GitHub Repository will have three new files.

The last step is to configure the Flux components in the Cluster to synchronize this application. We do this by creating a file in the clusters/cluster-1 folder.

Having committed and pushed this update, the GitHub Repository will have one new file.

After a minute or so, we can now see that the Flux components has created the application.

$ flux get kustomizations
flux-system True Applied revision: main/37a1751455599253a0086049aa4e0c855ca891af main/37a1751455599253a0086049aa4e0c855ca891af False
sample True Applied revision: main/37a1751455599253a0086049aa4e0c855ca891af main/37a1751455599253a0086049aa4e0c855ca891af False

Indeed, we can see the Pod running in the new Namespace.

$ kubectl get all -n sample
pod/sample 1/1 Running 0 17m

Next Steps

The next step is to explore the configuration of a multi-cluster single-tenant Kubernetes environment in the article Flux Multi-Cluster Multi-Tenant by Example (Continued).

Broad infrastructure, development, and soft-skill background