Using OpenID Connect (OIDC) for Authentication to vCluster’s Kubernetes API

Or… A bit more about OpenID Connect and Okta than you likely want to know about.

This article assumes one is already familiar with vCluster; if not, please see the series of articles starting with vCluster by Example (Part 1).

Problem

The cloud providers’ Kubernetes offerings, e.g., Google Kubernetes Engine (GKE), Amazon Elastic Kubernetes Service (EKS), and Azure Kubernetes Service (AKS), have their their own proprietary solutions for authenticating to the Kubernetes API. These solutions, however, do not work with vanilla Kubernetes; vCluster is essentially vanilla Kubernetes.

Based on the vCluster documentation, Access and expose vCluster, vCluster supports two approaches to authenticating to the Kubernetes API.

By default, vCluster updates the current kubeconfig to access the vCluster that contains the default admin client certificate and client key to authenticate to the vCluster. This means that all kubeconfig files generated have cluster admin access within the vCluster.

Often this might not be desired. Instead of giving a user admin access to the virtual cluster, you can also use service account authentication to the virtual cluster. Let’s say we want to create a kubeconfig file that only has view access in the virtual cluster. Then you would create a new service account inside the vCluster and assign it the cluster role view via a cluster role binding. Then we would generate a service account token and use that instead of the client-cert and client-key inside the kubeconfig.

The problem with these two approaches is that they both are generally considered to be anti-patterns, e.g., mentioned (5 minute mark) in the presentation Everything Wrong with K8s Authentication and How We Worked Around It — Mo Khan & Margo Crawford. If you are not convinced, the interview, Kubernetes Security with Identity & OIDC, makes a stronger case (1:50 minute mark).

note: The focus of this article is the use of vCluster outside of the vCluster Platform management system which does support other authentication approaches such as supporting many Single-Sign-On (SSO) providers. The problem, however, with using vCluster Platform is that introduces an unnecessary amount of complexity if one is only interested in alternative authentication approaches.

A Solution

Assuming that one’s identity provider supports OpenID Connect (OIDC) authentication, it turns out that one can use it to authenticate to the vCluster’s Kubernetes API.

The key to this solution is using the extraArgs key in one’s vcluster.yaml file; this allows one to pass additional arguments to the Kubernetes API server binary. This is important as Kubernetes natively supports OIDC authentication if and only if one can pass these additional arguments.

note: As a side note, most (if not all) cloud providers’ Kubernetes offerings do not allow one to pass additional arguments to the Kubernetes API server binary; thus preventing this particular approach. Interestingly, however, Amazon EKS does allow one to configure OIDC authentication.

A diagram provided in the article How to Secure Your Kubernetes Cluster with OpenID Connect and RBAC provides a high-level overview of the components and flow in using OIDC authentication to the Kubernetes API; we will continually refer back to this diagram throughout this article.

This walk-through is based on Okta being the OIDC provider. If you are interested in following along, you can set up a free trial. While we are not using Amazon EKS, the article Secure Access to AWS EKS Clusters for Admins was critical to piecing all this together.

note: One can also use the open-source OIDC provider, dex, as a drop-in replacement for Okta. A reason, however, for using Okta is that many organizations already use Okta for other purposes.

Populate Directory

We start by populating a directory of people (aka. users) and groups in Okta.

In most organizations, populating this directory involves integrating Okta with another directory, e.g, Active Directory or LDAP directory. Here we will simply create the people and groups directly in Okta. As part of signing up for the Okta free trial, we automatically get a person, ourself. This screen can be found at Okta Admin Console > Directory > People.

We, however, will want to create groups so that we can authorize access to the Kubernetes API based on groups (and not individual people). This screen can be found at Okta Admin Console > Directory > Groups.

Here we create a group named, vcluster-admins, and populate it with a single person (ourself). Here we prefixed the group name with vcluster- to distinguish as a group that we will be using for vCluster authentication.

Create App Integration

From App integrations:

Okta app integrations are configured connections between Okta features and external applications that provide a service. App integrations are either prebuilt or can be created by admins or developers to provide users seamless access to their assigned apps.

In our context, the external application is our vCluster’s Kubernetes API server (we will create the vCluster in a bit).

note: One might have other external applications, e.g., Argo CD; each would have their own app integration.

We navigate to the Okta Admin Console > Applications > Applications and create an app integration. The key choices that we have to make are as follows:

• The sign-in method is OIDC — OpenID Connect ; an obvious choice
• The application type is Native Application. This choice uses the OAuth2.0 Authorization Code Grant Flow with PKCE; described in the interview What’s going on with the OAuth 2.0 Implicit flow?
• The only grant type that is expected is the default; i.e., only Authorization Code
• The Sign-in / Sign-out redirect URIs that are both to be set to http://localhost:8000 (this web server is temporarily run by the kubelogin application on our workstation; more on this later)
• Here we can choose the option to Allow everyone in your organization to access as we will control access to the Kubernetes API by using RBAC authorization (more on this later)

The result of our work is a new app integration; here called my-vcluster.

Create Authorization Server

From Authorization servers.

At its core, an authorization server is simply an engine for minting OpenID Connect (OIDC) or OAuth 2.0 tokens. An authorization server is also used to apply access policies. Each authorization server has a unique issuer URI and its own signing key for tokens to keep a proper boundary between security domains.

We navigate to the Okta Admin Console > Security > API and create (add) an authorization server. The initial choices that we will have to make are as follows:

• Audience: http://localhost:8000 ; The audience specifies the intended recipient of the response token (here it is kubelogin which presumably expects this audience)

Next, we edit the authorization server so that the Issuer is a fixed URL; this is the URL that both the vCluster Kubernetes API server and kubelogin will use to communicate to the authorization server.

In order to authorize users (later with RBAC) using group membership, we need to update the authorization server to return an additional claim; a claim is a statement about a user, like their name, email, or role, that’s included in the response token. Here we add a claim from the authorization server’s Claims tab with:

• named groups
• be returned in all id tokens and any scope
• return Groups that match those prefixed with vcluster- as this was the convention we established in the Populate Directory section above

We next need to allow the app integration we created in the previous section to use this authorization server by creating an access policy using the Access Policies tab to include the app integration. Unfortunately, this is insufficient as we need to also add a rule to the access policy to specifically allow:

• Only grant type of Authorization Code
• Any user user assigned to the app (which we already selected to be anyone in the organization)
• Any requested scopes

Create vCluster

In preparation of creating the vCluster, there are a number of values we need from the both the app integration and authorization server:

• The app integration’s Client ID
• The authorization server’s Issuer
• The claim containing the username; here the authorization server can automatically return email
• The claim that has the group memberships; here we had configured the authorization server to use groups

With these we can create the vCluster.

Here we assume that we already know how to create vClusters; the key, however, to enabling OIDC authentication to the vClusters Kubernetes API server is to provide the following extraArgs in the vcluster.yaml.

note: Be sure to update CLIENT-ID and ISSUER.

controlPlane:
  distro:
    k8s:
      apiServer:
        extraArgs:
        - "--oidc-client-id=CLIENT-ID"
        - "--oidc-issuer-url=ISSUER"
        - "--oidc-username-claim=email"  
        - "--oidc-groups-claim=groups"   
      image:
        tag: v1.32.6

The first observation is that we can still access the vCluster using the default admin client certificate and client key in the kubectl config file extracted from the secret.

$ kubectl auth whoami --kubeconfig=config
ATTRIBUTE                                           VALUE
Username                                            kubernetes-super-admin
Groups                                              [system:masters system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [X509SHA256=1cd87794993337380abbd4ac75648e477211b76a7324308fa16c8fe169afe47c]

RBAC

Based on the work so far, the expectation is that we will be able to authenticate the vCluster’s Kubernetes API using:

Username: EMAIL-ADDRESS; in my case john@larkintuckerllc.com
Groups: [vcluster-admins system:authenticated]

So, to ensure we have access we can create the following clusterrolebinding in the vCluster.

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: vcluster-admins-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: Group
  name: vcluster-admins

kubelogin

From int128 / kubelogin:

This is a kubectl plugin for Kubernetes OpenID Connect (OIDC) authentication, also known as kubectl oidc-login.

We need to install kubelogin onto our workstation by following these instructions.

Validate OIDC Login

Before we involve the vCluster’s Kubernetes API server, we can validate that everything else is working as expected.

We execute:

note: Be sure to update CLIENT-ID and ISSUER.

$ kubectl oidc-login setup \
--oidc-client-id=CLIENT-ID \
--oidc-issuer-url=ISSUER

The kubelogin plugin kicks in here and opens up a browser window asking us to authenticate.

Once authenticated, we can see that we get redirected back to http://localhost:8000as expected (again this is the web server that kubelogin runs temporarily).

The command output looks something like this:

Authentication in progress...
## Authenticated with the OpenID Connect Provider

You got the token with the following claims:

```
{
  "sub": "00ut2llzrqlC36ihP697",
  "ver": 1,
  "iss": "https://trial-9697824.okta.com/oauth2/aust3nrghzhcFMuux697",
  "aud": "0oat3kzjbl4RUMxnT697",
  "iat": 1752182326,
  "exp": 1752185926,
  "jti": "ID.tAkiVFneWpCuiNwzZQkEZ8zE3mThnof7Dfxg3A_6Sv8",
  "amr": [
    "mfa",
    "otp",
    "pwd",
    "okta_verify"
  ],
  "idp": "00ot2llznhWgWMAoO697",
  "nonce": "xFJHvz6qV4NrE7ICFVQ2rB9B9XdF2EbTmR02xJQnh74",
  "auth_time": 1752182325,
  "at_hash": "d5zr9yApH7icWn27Gvn-8g",
  "groups": [
    "vcluster-admins"
  ]
}
```

## Set up the kubeconfig

You can run the following command to set up the kubeconfig:

```
kubectl config set-credentials oidc \
  --exec-api-version=client.authentication.k8s.io/v1 \
  --exec-interactive-mode=Never \
  --exec-command=kubectl \
  --exec-arg=oidc-login \
  --exec-arg=get-token \
  --exec-arg="--oidc-issuer-url=https://trial-9697824.okta.com/oauth2/aust3nrghzhcFMuux697" \
  --exec-arg="--oidc-client-id=0oat3kzjbl4RUMxnT697"
```

note: If you are wondering, by the time you read this article, I will have deleted the Okta resources used in this article (so no harm in sharing this information).

OIDC Authentication to vCluster Kubernetes API

In this last step we complete the full OIDC authentication flow.

The output of the previous command is designed to create the oidc user in one’s kubectl configuration that uses OIDC authentication. The output, however, is unexpectedly missing the argument --exec-arg="--oidc-extra-scope=email" (the vCluster API server is configured to use the email claim as the username) and expectedly missing the --kubeconfig=config argument (here we are using the vCluster’s kubectl configuration).

We execute the updated command:

note: Be sure to update CLIENT-ID and ISSUER.

kubectl config set-credentials oidc \
  --exec-api-version=client.authentication.k8s.io/v1 \
  --exec-interactive-mode=Never \
  --exec-command=kubectl \
  --exec-arg=oidc-login \
  --exec-arg=get-token \
  --exec-arg="--oidc-issuer-url=ISSUER" \
  --exec-arg="--oidc-client-id=CLIENT-ID" \
  --exec-arg="--oidc-extra-scope=email" \
  --kubeconfig=config

We can then validate that we can indeed authenticate using OIDC using the oidc user that we just created.

$ kubectl auth whoami --user=oidc --kubeconfig=config
ATTRIBUTE                                           VALUE
Username                                            john@larkintuckerllc.com
Groups                                              [vcluster-admins system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [JTI=ID.rMuB3cF4vsYPk494lU17qkSfJ94ONfHvVg3xv1sHKT0]

We can then change the current context to use the oidc user.

$ kubectl config set-context \
--current \
--user=oidc \
--kubeconfig=config

And now we do not need to supply the --user=oidc argument.

$ kubectl auth whoami --kubeconfig=config
ATTRIBUTE                                           VALUE
Username                                            john@larkintuckerllc.com
Groups                                              [vcluster-admins system:authenticated]
Extra: authentication.kubernetes.io/credential-id   [JTI=ID.rMuB3cF4vsYPk494lU17qkSfJ94ONfHvVg3xv1sHKT0]

One Last Thing

What may or may not be obvious is that we can create multiple vClusters with this same OIDC authentication setup and use the same credentials to authenticate to them.

We can then control access to the various vClusters by configuring RBAC authorization per vCluster as appropriate.

Whew; that was a bit more than expected.