Flux Multi-Cluster Multi-Tenant by Example

What is GitOps?

Prerequisites

  • Kubernetes Cluster 1.16 or newer (here I used Google Kubernetes Engine Standard 1.20.8-gke.900)
  • kubectl CLI; configured with the Cluster
  • Terraform CLI 1.0.5 or newer (here I used 1.0.5)
  • GitHub Account
  • Git CLI
  • The Flux CLI

Bootstrap Single-Cluster Single-Tenant

  • The keys prefixed with github are to be updated based on the newly created GitHub Repository; the branch is whatever Git branch we want to use (main is the new GitHub default)
  • The kubernetes_config_context (the context’s name) can be found in one’s kubeconfig file (assumes one has already configured the kubectl CLI with the Cluster)
  • The kubernetes_cluster_name is an arbitrary string for us to uniquely identify the cluster (will be more relevant as we move into the multi-cluster case)
$ terraform init
$ terraform apply
  • install_path and install_content: The Flux components in the Cluster
  • sync_path and sync_content: The configuration resources for the Flux components to determine the desired state from the newly created GitHub Repository
  • kustomize_path and kustomize_content: A kustomization file declaring the aforementioned install and sync resource files
$ flux create secret git flux-system \
--url=ssh://git@github.com/larkintuckerllc/hello-flux-repository \
--export > flux-system-secret.yaml
$ kubectl apply -f flux-system-secret.yaml
$ flux get kustomizationsNAME        READY MESSAGE                                                         REVISION                                      SUSPENDEDflux-system True  Applied revision: main/e0b4f7aac65cc69a8e9c25f37a866aa15f43801a main/e0b4f7aac65cc69a8e9c25f37a866aa15f43801a False

Custom Resources

$ kubectl describe gitrepository flux-system -n flux-system
Name: flux-system
Namespace: flux-system
Labels: kustomize.toolkit.fluxcd.io/name=flux-system
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations: kustomize.toolkit.fluxcd.io/checksum: 74af5c50a184a59640d3533cf577f07dd3e66bc5
API Version: source.toolkit.fluxcd.io/v1beta1
Kind: GitRepository
Metadata:
[OMITTED]
Spec:
Git Implementation: go-git
Interval: 1m0s
Ref:
Branch: main
Secret Ref:
Name: flux-system
Timeout: 20s
URL: ssh://git@github.com/larkintuckerllc/hello-flux-repository
Status:
Artifact:
Checksum: fe8007ddab6c5bf1329276f943eabb625ae92f4a
Last Update Time: 2021-08-24T10:14:32Z
Path: gitrepository/flux-system/flux-system/df906a5b44afc249bf7bf5f78becef7e89b71dfa.tar.gz
Revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
URL: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/df906a5b44afc249bf7bf5f78becef7e89b71dfa.tar.gz
Conditions:
Last Transition Time: 2021-08-24T10:14:32Z
Message: Fetched revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Reason: GitOperationSucceed
Status: True
Type: Ready
Observed Generation: 1
URL: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/latest.tar.gz
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal error 38m (x17 over 43m) source-controller auth secret error: Secret "flux-system" not found
Normal info 32m source-controller Fetched revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
  • Here we can infer that the source-controller Flux component is responsible for using GitRepository custom resources to regularly download states of a Git Repository and then expose their latest state on a URL
$ kubectl describe kustomization flux-system -n flux-system
Name: flux-system
Namespace: flux-system
Labels: kustomize.toolkit.fluxcd.io/name=flux-system
kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations: kustomize.toolkit.fluxcd.io/checksum: 74af5c50a184a59640d3533cf577f07dd3e66bc5
API Version: kustomize.toolkit.fluxcd.io/v1beta1
Kind: Kustomization
Metadata:
[OMITTED]
Spec:
Force: false
Interval: 10m0s
Path: ./clusters/cluster-1
Prune: true
Source Ref:
Kind: GitRepository
Name: flux-system
Validation: client
Status:
Conditions:
Last Transition Time: 2021-08-24T11:03:45Z
Message: Applied revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Reason: ReconciliationSucceeded
Status: True
Type: Ready
Last Applied Revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Last Attempted Revision: main/df906a5b44afc249bf7bf5f78becef7e89b71dfa
Observed Generation: 1
Snapshot:
[OMITTED]
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal info 58m kustomize-controller clusterrole.rbac.authorization.k8s.io/crd-controller-flux-system configured
service/source-controller configured
gitrepository.source.toolkit.fluxcd.io/flux-system configured
customresourcedefinition.apiextensions.k8s.io/kustomizations.kustomize.toolkit.fluxcd.io configured
deployment.apps/notification-controller configured
kustomization.kustomize.toolkit.fluxcd.io/flux-system configured
deployment.apps/kustomize-controller configured
customresourcedefinition.apiextensions.k8s.io/helmreleases.helm.toolkit.fluxcd.io configured
networkpolicy.networking.k8s.io/allow-scraping configured
customresourcedefinition.apiextensions.k8s.io/providers.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/receivers.notification.toolkit.fluxcd.io configured
namespace/flux-system configured
customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmcharts.source.toolkit.fluxcd.io configured
networkpolicy.networking.k8s.io/allow-webhooks configured
serviceaccount/source-controller configured
deployment.apps/source-controller configured
networkpolicy.networking.k8s.io/allow-egress configured
deployment.apps/helm-controller configured
serviceaccount/helm-controller configured
serviceaccount/kustomize-controller configured
serviceaccount/notification-controller configured
clusterrolebinding.rbac.authorization.k8s.io/cluster-reconciler-flux-system configured
clusterrolebinding.rbac.authorization.k8s.io/crd-controller-flux-system configured
service/notification-controller configured
service/webhook-receiver configured
customresourcedefinition.apiextensions.k8s.io/buckets.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/gitrepositories.source.toolkit.fluxcd.io configured
customresourcedefinition.apiextensions.k8s.io/helmrepositories.source.toolkit.fluxcd.io configured
Normal info 9m50s (x6 over 58m) kustomize-controller Update completed
  • Here we can see precisely which resources are being synchronized by this Kustomization custom resource

Deploy a Sample Application

$ flux get kustomizations
NAME READY MESSAGE REVISION SUSPENDED
flux-system True Applied revision: main/37a1751455599253a0086049aa4e0c855ca891af main/37a1751455599253a0086049aa4e0c855ca891af False
sample True Applied revision: main/37a1751455599253a0086049aa4e0c855ca891af main/37a1751455599253a0086049aa4e0c855ca891af False
$ kubectl get all -n sample
NAME READY STATUS RESTARTS AGE
pod/sample 1/1 Running 0 17m

Next Steps

--

--

--

Broad infrastructure, development, and soft-skill background

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Development life cycle

What is Cloud Hosting?

Building Azure Functions in Azure Portal & Visual Studio 2017 using .NET Core

CS371P Spring 2022 Blog 9: Vedika Jain

From Flask, Gunicorn, Nginx to Docker and Security through HTTPS and Client Certificate

Kubernetes namespaces

The Design and Implementation of Pie Menus

Android fundamentals 03.1: The debugger

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
John Tucker

John Tucker

Broad infrastructure, development, and soft-skill background

More from Medium

Using OpenTelemetry auto-instrumentation/agents in Kubernetes

Graceful Shutdown in Kubernetes — Theory, Implementation, and Testing

Istio — How it solves EiT (Encryption-in-Transit) in a Microservices Architecture (MSA)

Monitoring multiple OKE clusters with Prometheus, Thanos and Grafana — Part 1