Open in app

Sign in

Write

Sign in

Flux Multi-Cluster Multi-Tenant by Example

John Tucker
7 min readAug 25, 2021

A walk-through of using Flux to deliver applications in a multi-cluster multi-tenant Kubernetes environment.

Flux — the GitOps family of projects
Flux is a set of continuous and progressive delivery solutions for Kubernetes, and they are open and extensible.

— Flux — Flux

What is GitOps?

Before we dig into Flux, it is important to understand what GitOps is from the perspective of the Flux team.

Principles of GitOps
To start managing your cluster with GitOps workflows, the following must be in place:

#1. The entire system described declaratively.
#2. The canonical desired system state versioned in Git.
#3. Approved changes that can be automatically applied to the system.
#4. Software agents to ensure correctness and alert on divergence.

— Weaveworks — Guide To GitOps

From another broader perspective, this definition of GitOps is essentially the GitOps pull-based deployment strategy.

The Pull-based deployment strategy uses the same concepts as the push-based variant but differs in how the deployment pipeline works. Traditional CI/CD pipelines are triggered by an external event, for example when new code is pushed to an application repository. With the pull-based deployment approach, the operator is introduced. It takes over the role of the pipeline by continuously comparing the desired state in the environment repository with the actual state in the deployed infrastructure. Whenever differences are noticed, the operator updates the infrastructure to match the environment repository.

— GitOps — GitOps

Prerequisites

If you wish to follow along, you will need:

  • Kubernetes Cluster 1.16 or newer (here I used Google Kubernetes Engine Standard 1.20.8-gke.900)
  • kubectl CLI; configured with the Cluster
  • Terraform CLI 1.0.5 or newer (here I used 1.0.5)
  • GitHub Account
  • Git CLI
  • The Flux CLI

Bootstrap Single-Cluster Single-Tenant

Here we start with the simplest case; a single-cluster with a single-tenant.

The first step is to create the GitHub Repository to hold the desired system state; here we simply use the GitHub UI to create a private Repository (in my case larkintuckerllc / hello-flux-repository).

Next, we download the larkintuckerllc /hello-flux-single-single-bootstrap Repository. While one can use the flux CLI to bootstrap flux into our Cluster, here we use Terraform for a more (less magic) declarative approach.

We create a terraform.tfvar file in the root of the downloaded Repository with the following key / value pairs; we update to match our environment.

Things to observe:

  • The keys prefixed with github are to be updated based on the newly created GitHub Repository; the branch is whatever Git branch we want to use (main is the new GitHub default)
  • The kubernetes_config_context (the context’s name) can be found in one’s kubeconfig file (assumes one has already configured the kubectl CLI with the Cluster)
  • The kubernetes_cluster_name is an arbitrary string for us to uniquely identify the cluster (will be more relevant as we move into the multi-cluster case)

Please note: If you are wondering what resources this Terraform configuration creates; they are Kubernetes resources in the flux-system Namespace consisting of Flux components (install resources) as well as a Flux bootstrap configuration (sync resources).

From the root of the downloaded Repository, we initialize Terraform:

$ terraform init

We then create the resources.

$ terraform apply

The output of the terraform apply command has the following key / values for us to manually create three files in the newly created Git Repository in the branch specified in the terraform.tfvars file; push the branch to GitHub.

  • install_path and install_content: The Flux components in the Cluster
  • sync_path and sync_content: The configuration resources for the Flux components to determine the desired state from the newly created GitHub Repository
  • kustomize_path and kustomize_content: A kustomization file declaring the aforementioned install and sync resource files

Please note: You can re-run the apply command to re-generate the output and direct it to a file; helpful as the install_content is rather long.

Please note: Other tutorials on bootstrapping Flux with Terraform automate the creation of these files. Here, for security reasons, it is not recommend as it requires creating long-lived GitHub personal access tokens.

At this point, the repository consists of three files.

We now need to grant the Flux components read-only access to the newly created GitHub Repository. Run the following command, updated with the appropriate GitHub owner and repository.

$ flux create secret git flux-system \
    --url=ssh://git@github.com/larkintuckerllc/hello-flux-repository \
    --export > flux-system-secret.yaml

Add the value of the key identity.pub in the generated manifest as a read-only deploy key in the newly created GitHub repository.

Create the Kubernetes Secret with the following command.

$ kubectl apply -f flux-system-secret.yaml

Please note: Other tutorials on bootstrapping Flux with Terraform automate the creation of this Secret. Here, for security reasons, it is not recommended as it stores GitHub deploy key in the Terraform state.

If all is working as expected, in a few minutes the following command should indicate that the flux-system kustomization is ready (True).

$ flux get kustomizationsNAME        READY MESSAGE                                                         REVISION                                      SUSPENDEDflux-system True  Applied revision: main/e0b4f7aac65cc69a8e9c25f37a866aa15f43801a main/e0b4f7aac65cc69a8e9c25f37a866aa15f43801a False

Custom Resources

Before deploying a sample application into the Cluster, it is instructive to examine the custom resources used to reflect the state of the resources bootstrapped in the flux-system Namespace.

Here we examine the first resource in the gotk-sync.yaml file we created in the GitHub Repository.

This is a GitRepository custom resource.

The GitRepository API defines a source for artifacts coming from Git. The resource exposes the latest synchronized state from Git as an artifact in a gzip compressed TAR archive.

— Flux — Git Repositories

We can infer a lot about GitRepositories simply by describing them.

$ kubectl describe gitrepository flux-system -n flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  kustomize.toolkit.fluxcd.io/checksum: 74af5c50a184a59640d3533cf577f07dd3e66bc5
API Version:  source.toolkit.fluxcd.io/v1beta1
Kind:         GitRepository
Metadata:
  [OMITTED]
Spec:
  Git Implementation:  go-git
  Interval:            1m0s
  Ref:
    Branch:  main
  Secret Ref:
    Name:   flux-system
  Timeout:  20s
  URL:      ssh://git@github.com/larkintuckerllc/hello-flux-repository
Status:
  Artifact:
    Checksum:          fe8007ddab6c5bf1329276f943eabb625ae92f4a
    Last Update Time:  2021-08-24T10:14:32Z
    Path:              gitrepository/flux-system/flux-system/df906a5b44afc249bf7bf5f78becef7e89b71dfa.tar.gz
    Revision:          main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
    URL:               http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/df906a5b44afc249bf7bf5f78becef7e89b71dfa.tar.gz
  Conditions:
    Last Transition Time:  2021-08-24T10:14:32Z
    Message:               Fetched revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
    Reason:                GitOperationSucceed
    Status:                True
    Type:                  Ready
  Observed Generation:     1
  URL:                     http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/latest.tar.gz
Events:
  Type    Reason  Age                 From               Message
  ----    ------  ----                ----               -------
  Normal  error   38m (x17 over 43m)  source-controller  auth secret error: Secret "flux-system" not found
  Normal  info    32m                 source-controller  Fetched revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa

Things to observe:

  • Here we can infer that the source-controller Flux component is responsible for using GitRepository custom resources to regularly download states of a Git Repository and then expose their latest state on a URL

Next we examine the second resource in the gotk-sync.yaml file we created in the GitHub Repository.

This is a Kustomization custom resource.

The Kustomization API defines a pipeline for fetching, decrypting, building, validating and applying Kubernetes manifests.

— Flux — Kustomization

One interesting thing here is the interval specified here is fairly long, 10 minutes. This interval determines how often to reconcile the resources defined in the Kustomization.

While we previously used the flux get kustomizations command to examine a Kustomization, describing it provides more information.

$ kubectl describe kustomization flux-system -n flux-system
Name:         flux-system
Namespace:    flux-system
Labels:       kustomize.toolkit.fluxcd.io/name=flux-system
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  kustomize.toolkit.fluxcd.io/checksum: 74af5c50a184a59640d3533cf577f07dd3e66bc5
API Version:  kustomize.toolkit.fluxcd.io/v1beta1
Kind:         Kustomization
Metadata:
  [OMITTED]
Spec:
  Force:     false
  Interval:  10m0s
  Path:      ./clusters/cluster-1
  Prune:     true
  Source Ref:
    Kind:      GitRepository
    Name:      flux-system
  Validation:  client
Status:
  Conditions:
    Last Transition Time:   2021-08-24T11:03:45Z
    Message:                Applied revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
    Reason:                 ReconciliationSucceeded
    Status:                 True
    Type:                   Ready
  Last Applied Revision:    main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
  Last Attempted Revision:  main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
  Observed Generation:      1
  Snapshot:
    [OMITTED]
Events:
  Type    Reason  Age   From                  Message
  ----    ------  ----  ----                  -------
  Normal  info    58m   kustomize-controller  clusterrole.rbac.authorization.k8s.io/crd-controller-flux-system configured
service/source-controller configured
gitrepository.source.toolkit.fluxcd.io/flux-system configured
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io configured
deployment.apps/notification-controller configured
kustomization.kustomize.toolkit.fluxcd.io/flux-system configured
deployment.apps/kustomize-controller configured
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io configured
networkpolicy.networking.k8s.io/allow-scraping configured
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io configured
namespace/flux-system configured
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io configured
networkpolicy.networking.k8s.io/allow-webhooks configured
serviceaccount/source-controller configured
deployment.apps/source-controller configured
networkpolicy.networking.k8s.io/allow-egress configured
deployment.apps/helm-controller configured
serviceaccount/helm-controller configured
serviceaccount/kustomize-controller configured
serviceaccount/notification-controller configured
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-flux-system configured
clusterrolebinding.rbac.authorization.k8s.io/crd-controller-flux-system configured
service/notification-controller configured
service/webhook-receiver configured
customresourcedefinition.apiextensions.k8s.io/buckets.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io configured
  Normal  info  9m50s (x6 over 58m)  kustomize-controller  Update completed

Things to observe:

  • Here we can see precisely which resources are being synchronized by this Kustomization custom resource

Deploy a Sample Application

Now that we have Flux bootstrapped into the Cluster, we can use it to deploy a sample application. This application consists of a single Namespace (sample) and Pod in it (sample). We create the following files in a new apps/sample folder in the newly created Git Repository.

Having committed and pushed the update, the GitHub Repository will have three new files.

The last step is to configure the Flux components in the Cluster to synchronize this application. We do this by creating a file in the clusters/cluster-1 folder.

Having committed and pushed this update, the GitHub Repository will have one new file.

After a minute or so, we can now see that the Flux components has created the application.

$ flux get kustomizations
NAME        READY MESSAGE                                                         REVISION                                      SUSPENDED 
flux-system True  Applied revision: main/37a1751455599253a0086049aa4e0c855ca891af main/37a1751455599253a0086049aa4e0c855ca891af False     
sample      True  Applied revision: main/37a1751455599253a0086049aa4e0c855ca891af main/37a1751455599253a0086049aa4e0c855ca891af False

Indeed, we can see the Pod running in the new Namespace.

$ kubectl get all -n sample
NAME         READY   STATUS    RESTARTS   AGE
pod/sample   1/1     Running   0          17m

Next Steps

The next step is to explore the configuration of a multi-cluster single-tenant Kubernetes environment in the article Flux Multi-Cluster Multi-Tenant by Example (Continued).

Kubernetes
Continuous Integration
Flux

--

--

John Tucker

Written by John Tucker

5.1K Followers

Broad infrastructure, development, and soft-skill background

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams