Backstage and Kubernetes by Example
In this article, we explore the Backstage Kubernetes core feature.
Kubernetes in Backstage is a tool that’s designed around the needs of service owners, not cluster admins. Now developers can easily check the health of their services no matter how or where those services are deployed — whether it’s on a local host for testing or in production on dozens of clusters around the world.
It will elevate the visibility of errors where identified, and provide drill down about the deployments, pods, and other objects for a service.
A video created by Spotify R&D provides us a demonstration of using it.
Installation
This article builds-off the series of articles starting with Backstage by Example (Part 1); in particular we will be building off the last example where we run the Backstage App in a container running on our workstation with Docker Compose.
We follow the Backstage Kubernetes core feature installation instructions; updating both the frontend and backend code as described. The instructions, however, suggest that we can start the Backstage App before we configure the Kubernetes plugin; we cannot (the backend will crash-loop until the Kubernetes plugin is configured).
minikube
We begin with configuring the Backstage App Kubernetes plugin for a minikube Kubernetes Cluster running on our workstation; we first go ahead and start minikube.
Unfortunately, the Backstage Configuring Kubernetes integration documentation is a bit confusing. Here we take a simplified approach.
We begin by adding the following to the end of our app-config.yaml.
In order for the Backstage App Kubernetes plugin to interact with the Kubernetes API it requires two things: the Kubernetes API URL and credentials (token) to authenticate / authorize to it.
The Kubernetes API URL can be obtained by running the following command.
$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:62052
CoreDNS is running at https://127.0.0.1:62052/api/v1/namespaces/kube-system/services/kube-dns:dns/proxyWe, however, need to replace the 127.0.0.1 in the URL with an appropriate hostname referring to the host workstation as the Backstage App runs in a container; for macOS that hostname is host.docker.internal.
As we are using a Kubernetes ServiceAccount for authentication / authorization, we first need to give it the appropriate access as documented in Configuring Kubernetes integration documentation. We, here, for simplicity will rather grant the ServiceAccount (in this case the default ServiceAccount in the default Namespace) full access to the API by applying the following ClusterRoleBinding to our Minikube Kubernetes Cluster.
We can use the following command to extract the ServiceAccount token.
$ kubectl -n default get secret $(kubectl -n default get sa default -o=json \
| jq -r '.secrets[0].name') -o=json \
| jq -r '.data["token"]' \
| base64 --decodeWe then update our docker-compose.yaml to pass both the K8S_MINIKUBE_URL and K8S_MINIKUBE_TOKEN environment variables into our container. Likewise we update our environment.sh with these appropriate values for these environment variables; the Kubernetes API URL and ServiceAccount token respectively.
With all this place, we can start (re-start) our Backstage App.
$ source environment.sh
$ docker-compose down
$ yarn clean
$ yarn install
$ yarn tsc
$ yarn build
$ yarn build-image
$ docker-compose up -dPlease note: In the interest in keeping this article focused on the key points, for the remainder of the article the above commands are required every we change the Backstage App code or configuration.
If all goes well, we can login to our Backstage App as before; nothing will be obviously different at this point.
Creating a Kubernetes Enabled Component
To illustrate the key concepts, will methodically work towards creating a Kubernetes enabled Component. We start by creating a new Github Repository in our organization, e.g., my-kubernetes-component, in our Organization with a single file, catalog-info.yaml, file in it.
Please note: It is important that the type be set to service; otherwise the Kubernetes core feature will not work (lost a couple of hours on this myself).
We then create a new Component by registering an existing component from the newly created Github Repository. When inspecting the Component, we can see that there is a KUBERNETES tab.
Selecting this tab give us an indication of what we need to do next; add an annotation.
We update our catalog-info.yaml in our GitHub Repository to:
While Backstage will eventually automatically update the Component based on changes in the GitHub Repository, we can expedite the process by requesting a refresh (the button with circular arrows on a Component’s screen). Selecting the KUBERNETES tab now shows us something different.
The next step is that we need to create some Kubernetes resources in our minikube Kubernetes Cluster that the Backstage Kubernetes core feature can associate to our new Component. The trick here is to label those Kubernetes resources with the annotation key-value pair we added to the Component.
Please note: It is important to label all the resources, i.e., both the Deployment and Pod template.
With the Kubernetes resources in the Cluster, we can now see that the Component’s Kubernetes screen shows the Deployment and its Pods.
Drilling down through the UI a bit more, we can get details on the Pods themselves.
Metrics Lookup
In the UI listing the Pods we can see that the CPU USAGE% and MEMORY USAGE % columns say unknown; let us see if we can correct this. We start by installing the metrics server into minikube.
$ minikube addons enable metrics-serverWe can confirm that the metrics-server is working by running the following command.
$ kubectl top pod
NAME CPU(cores) MEMORY(bytes)
my-kubernetes-component-dbf85dc87-kfhbq 0m 0Mi
my-kubernetes-component-dbf85dc87-twwkh 0m 0Mi
my-kubernetes-component-dbf85dc87-zrw5x 0m 0MiWe then update the key skipMetricsLookup to false in the file app-config.yaml. We re-start the Backstage App and re-register the Component (in our current process, re-starting the Backstage App also wipes out the PostgreSQL database).
Drilling down through the UI, we experience an error.
After thinking about this a bit, we never supplied the Container any limits / requests for cpu / memory and the unknown columns were percentages; we update our our Kubernetes Deployment to include them.
And now we get the expected result. Apparently, this is a subtle bug in the Kubernetes core feature.
Additional (Cloud Provider) Clusters
At this point, we have configured the Kubernetes core feature for a minikube Cluster; here wrap up by exploring what is required to add additional clusters and in particular Cloud Provider Clusters.
Instead of trying to use the minimally documented aws or googleServiceAccount values for the authProvider key in the app-config.yaml, we can continue to use the same approach of using the serviceAccount value as we did with minikube and add a second entry to the clusters key in app-config.yaml.
Please note: While I used a Google Kubernetes Engine (GKE) Cluster in this example, one could have as easily used a AWS Elastic Kubernetes Service (EKS) Cluster.
As before we need to next update both docker-compose.yml and environment.sh to support the environment variables for the second Cluster.
With these updates, and creating the same my-kubernetes-component Deployment on the cloud provider Cluster, we indeed see both Deployments (one on each Cluster) in the UI.
Finally, as the Cloud Provider Cluster Kubernetes API URL is likely public, we should (for security purposes) set the value for the skipTLSVerify key to false and supply (via another environment variable) a value for caData for the Cluster in app-config.yaml.
The value for caData can be extracted from one’s kubectl configuration file, e.g., ~/.kube/config, from the certificate-authority-data key for the cluster.
Wrap Up
That is about it; hope you found this useful.