Flux Multi-Cluster Multi-Tenant by Example

What is GitOps?

Before we dig into Flux, it is important to understand what GitOps is from the perspective of the Flux team.


If you wish to follow along, you will need:

  • Kubernetes Cluster 1.16 or newer (here I used Google Kubernetes Engine Standard 1.20.8-gke.900)
  • kubectl CLI; configured with the Cluster
  • Terraform CLI 1.0.5 or newer (here I used 1.0.5)
  • GitHub Account
  • Git CLI
  • The Flux CLI

Bootstrap Single-Cluster Single-Tenant

Here we start with the simplest case; a single-cluster with a single-tenant.

  • The keys prefixed with github are to be updated based on the newly created GitHub Repository; the branch is whatever Git branch we want to use (main is the new GitHub default)
  • The kubernetes_config_context (the context’s name) can be found in one’s kubeconfig file (assumes one has already configured the kubectl CLI with the Cluster)
  • The kubernetes_cluster_name is an arbitrary string for us to uniquely identify the cluster (will be more relevant as we move into the multi-cluster case)
$ terraform init
$ terraform apply
  • install_path and install_content: The Flux components in the Cluster
  • sync_path and sync_content: The configuration resources for the Flux components to determine the desired state from the newly created GitHub Repository
  • kustomize_path and kustomize_content: A kustomization file declaring the aforementioned install and sync resource files
$ flux create secret git flux-system \
--url=ssh://git@github.com/larkintuckerllc/hello-flux-repository \
--export > flux-system-secret.yaml
$ kubectl apply -f flux-system-secret.yaml
$ flux get kustomizationsNAME        READY MESSAGE                                                         REVISION                                      SUSPENDEDflux-system True  Applied revision: main/e0b4f7aac65cc69a8e9c25f37a866aa15f43801a main/e0b4f7aac65cc69a8e9c25f37a866aa15f43801a False

Custom Resources

Before deploying a sample application into the Cluster, it is instructive to examine the custom resources used to reflect the state of the resources bootstrapped in the flux-system Namespace.

$ kubectl describe gitrepository flux-system -n flux-system
Name: flux-system
Namespace: flux-system
Labels: kustomize.toolkit.fluxcd.io/name=flux-system
Annotations: kustomize.toolkit.fluxcd.io/checksum: 74af5c50a184a59640d3533cf577f07dd3e66bc5
API Version: source.toolkit.fluxcd.io/v1beta1
Kind: GitRepository
Git Implementation: go-git
Interval: 1m0s
Branch: main
Secret Ref:
Name: flux-system
Timeout: 20s
URL: ssh://git@github.com/larkintuckerllc/hello-flux-repository
Checksum: fe8007ddab6c5bf1329276f943eabb625ae92f4a
Last Update Time: 2021-08-24T10:14:32Z
Path: gitrepository/flux-system/flux-system/df906a5b44afc249bf7bf5f78becef7e89b71dfa.tar.gz
Revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
URL: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/df906a5b44afc249bf7bf5f78becef7e89b71dfa.tar.gz
Last Transition Time: 2021-08-24T10:14:32Z
Message: Fetched revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Reason: GitOperationSucceed
Status: True
Type: Ready
Observed Generation: 1
URL: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/latest.tar.gz
Type Reason Age From Message
---- ------ ---- ---- -------
Normal error 38m (x17 over 43m) source-controller auth secret error: Secret "flux-system" not found
Normal info 32m source-controller Fetched revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
  • Here we can infer that the source-controller Flux component is responsible for using GitRepository custom resources to regularly download states of a Git Repository and then expose their latest state on a URL
$ kubectl describe kustomization flux-system -n flux-system
Name: flux-system
Namespace: flux-system
Labels: kustomize.toolkit.fluxcd.io/name=flux-system
Annotations: kustomize.toolkit.fluxcd.io/checksum: 74af5c50a184a59640d3533cf577f07dd3e66bc5
API Version: kustomize.toolkit.fluxcd.io/v1beta1
Kind: Kustomization
Force: false
Interval: 10m0s
Path: ./clusters/cluster-1
Prune: true
Source Ref:
Kind: GitRepository
Name: flux-system
Validation: client
Last Transition Time: 2021-08-24T11:03:45Z
Message: Applied revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Reason: ReconciliationSucceeded
Status: True
Type: Ready
Last Applied Revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Last Attempted Revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Observed Generation: 1
Type Reason Age From Message
---- ------ ---- ---- -------
Normal info 58m kustomize-controller clusterrole.rbac.authorization.k8s.io/crd-controller-flux-system configured
service/source-controller configured
gitrepository.source.toolkit.fluxcd.io/flux-system configured
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io configured
deployment.apps/notification-controller configured
kustomization.kustomize.toolkit.fluxcd.io/flux-system configured
deployment.apps/kustomize-controller configured
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io configured
networkpolicy.networking.k8s.io/allow-scraping configured
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io configured
namespace/flux-system configured
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io configured
networkpolicy.networking.k8s.io/allow-webhooks configured
serviceaccount/source-controller configured
deployment.apps/source-controller configured
networkpolicy.networking.k8s.io/allow-egress configured
deployment.apps/helm-controller configured
serviceaccount/helm-controller configured
serviceaccount/kustomize-controller configured
serviceaccount/notification-controller configured
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-flux-system configured
clusterrolebinding.rbac.authorization.k8s.io/crd-controller-flux-system configured
service/notification-controller configured
service/webhook-receiver configured
customresourcedefinition.apiextensions.k8s.io/buckets.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io configured
Normal info 9m50s (x6 over 58m) kustomize-controller Update completed
  • Here we can see precisely which resources are being synchronized by this Kustomization custom resource

Deploy a Sample Application

Now that we have Flux bootstrapped into the Cluster, we can use it to deploy a sample application. This application consists of a single Namespace (sample) and Pod in it (sample). We create the following files in a new apps/sample folder in the newly created Git Repository.

$ flux get kustomizations
flux-system True Applied revision: main/37a1751455599253a0086049aa4e0c855ca891af main/37a1751455599253a0086049aa4e0c855ca891af False
sample True Applied revision: main/37a1751455599253a0086049aa4e0c855ca891af main/37a1751455599253a0086049aa4e0c855ca891af False
$ kubectl get all -n sample
pod/sample 1/1 Running 0 17m

Next Steps

The next step is to explore the configuration of a multi-cluster single-tenant Kubernetes environment in the article Flux Multi-Cluster Multi-Tenant by Example (Continued).



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
John Tucker

John Tucker

Broad infrastructure, development, and soft-skill background