Kubernetes Authentication

Kubernetes Authentication means validating the identity of who or what is issuing the request. For example, if you want to access a Linux box through SSH, the SSH daemon must verify that the username and password you are using for login matches an account that lives in /etc/passwd and /etc/shadow files. For them to be Combined together, those files represent the authentication database for Linux. On the other hand, authorization refers to what you are allowed to do once you gain access. So, back to our Linux example, the user can gain access to the operating system by providing valid credentials, but they cannot view or modify the contents of sensitive files like /etc/shadow. Only the root user or someone with root privileges can do this.

In Kubernetes, the API server needs to authenticate every request that it receives. A request may originate from a human user or a program (like a Pod for example). Currently, human user accounts need to be managed outside the cluster through one of the supported authentication strategies. Notice that a user account is not namespaced, which means that it must be unique cluster-wide. But more often than not, you also need to enable programs to get authenticated and send requests to the API server. For a resource to be able to authenticate to the API server, it needs to possess a service account.

The Service Account

A service account is another Kubernetes resource (like pods, deployments, etc.). Unlike user accounts, the service account is namespaced so you can create multiple service accounts with the same name as long as they live in different namespaces.

By default, the service account credentials are mounted to the pods through a secret. Let’s have a quick example:

$ kubectl run -it --rm testpod --restart=Never --image=alpine -- sh
If you don't see a command prompt, try pressing enter.
/ # ls -l /var/run/secrets/kubernetes.io/serviceaccount/
total 0
lrwxrwxrwx    1 root     root            13 Nov  8 11:45 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root            16 Nov  8 11:45 namespace -> ..data/namespace
lrwxrwxrwx    1 root     root            12 Nov  8 11:45 token -> ..data/token

This is a one-time pod that hosts an Alpine image. Opening a shell to the container, we could easily view the files that were mounted to the pod from an outside secret under the /data mount point. The authentication process is done through the token file. Let’s have a look at the contents of this file:

/ # cat /var/run/secrets/kubernetes.io/serviceaccount/token && echo
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tcHh6ZjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUzMzFmNTU1LTAxOTktMTFlYS1hYTVhLTQyMDEwYTgwMDBhYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.I3jFqRd7ANj4BidzoW-v_YO1N_z2p33Ex1biaz_J97QK9CRlIFmK2KC1k0RbQux3zo72xsLhbPjNH40pBzqr2TzVjZKFD7T8-ihDj1Og5L_BAud-CtT4e-0zcvC5rTKcWzkUn5a64TUSwF5Q6I3KjeaE3pmDFwG4I6XLuODCVOwRkfn0V_LzjqLy3nXfUK8FpIkeBrRd9QN68PG9YFH9lKMgGZLvuG_m6K6EFxqBpVMec8SPsG77GGRezC9Mjsyxp2Wie-j8vUVb5et2o4ShKa8sp-Nqum4bpBxGQ9NBo3qlefuqYGYivGZBnL8dienBLFbO5swSHUL6vxTgaRg2kA
/ #

The contents of the token is of the JSON Web Token (JWT) format. It can be decoded using the jwt command-line tool or online on the jwt.io website. The following is the output of the decoded operation on the web likewise on the command-line:

And on the command line:

$ jwt eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tcHh6ZjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUzMzFmNTU1LTAxOTktMTFlYS1hYTVhLTQyMDEwYTgwMDBhYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.I3jFqRd7ANj4BidzoW-v_YO1N_z2p33Ex1biaz_J97QK9CRlIFmK2KC1k0RbQux3zo72xsLhbPjNH40pBzqr2TzVjZKFD7T8-ihDj1Og5L_BAud-CtT4e-0zcvC5rTKcWzkUn5a64TUSwF5Q6I3KjeaE3pmDFwG4I6XLuODCVOwRkfn0V_LzjqLy3nXfUK8FpIkeBrRd9QN68PG9YFH9lKMgGZLvuG_m6K6EFxqBpVMec8SPsG77GGRezC9Mjsyxp2Wie-j8vUVb5et2o4ShKa8sp-Nqum4bpBxGQ9NBo3qlefuqYGYivGZBnL8dienBLFbO5swSHUL6vxTgaRg2kA

To verify on jwt.io:

https://jwt.io/#id_token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tcHh6ZjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUzMzFmNTU1LTAxOTktMTFlYS1hYTVhLTQyMDEwYTgwMDBhYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.I3jFqRd7ANj4BidzoW-v_YO1N_z2p33Ex1biaz_J97QK9CRlIFmK2KC1k0RbQux3zo72xsLhbPjNH40pBzqr2TzVjZKFD7T8-ihDj1Og5L_BAud-CtT4e-0zcvC5rTKcWzkUn5a64TUSwF5Q6I3KjeaE3pmDFwG4I6XLuODCVOwRkfn0V_LzjqLy3nXfUK8FpIkeBrRd9QN68PG9YFH9lKMgGZLvuG_m6K6EFxqBpVMec8SPsG77GGRezC9Mjsyxp2Wie-j8vUVb5et2o4ShKa8sp-Nqum4bpBxGQ9NBo3qlefuqYGYivGZBnL8dienBLFbO5swSHUL6vxTgaRg2kA

✻ Header
{
  "alg": "RS256",
  "kid": ""
}

✻ Payload
{
  "iss": "kubernetes/serviceaccount",
  "kubernetes.io/serviceaccount/namespace": "default",
  "kubernetes.io/serviceaccount/secret.name": "default-token-pxzf7",
  "kubernetes.io/serviceaccount/service-account.name": "default",
  "kubernetes.io/serviceaccount/service-account.uid": "e331f555-0199-11ea-aa5a-42010a8000aa",
  "sub": "system:serviceaccount:default:default"
}

✻ Signature I3jFqRd7ANj4BidzoW-v_YO1N_z2p33Ex1biaz_J97QK9CRlIFmK2KC1k0RbQux3zo72xsLhbPjNH40pBzqr2TzVjZKFD7T8-ihDj1Og5L_BAud-CtT4e-0zcvC5rTKcWzkUn5a64TUSwF5Q6I3KjeaE3pmDFwG4I6XLuODCVOwRkfn0V_LzjqLy3nXfUK8FpIkeBrRd9QN68PG9YFH9lKMgGZLvuG_m6K6EFxqBpVMec8SPsG77GGRezC9Mjsyxp2Wie-j8vUVb5et2o4ShKa8sp-Nqum4bpBxGQ9NBo3qlefuqYGYivGZBnL8dienBLFbO5swSHUL6vxTgaRg2kA

The interesting part of the output is the Payload. As you can see, we have the service-account.name set to “default”. This is the default service account that gets created and mounted automatically to every pod upon initialization. In most cases, you shouldn’t use the default account when you need to give your pod more privileges. Let’s create a service account and assign it to a pod:

$ kubectl create serviceaccount testsa                                                                                                             
serviceaccount/testsa created

Notice that we did not specify a namespace here so our testsa service account is created in the default namespace. If we take a look at the Secrets in this namespace we shall find two secrets:

$ kubectl get secrets
NAME                  TYPE                                  DATA   AGE
default-token-pxzf7   kubernetes.io/service-account-token   3      16h
testsa-token-zwhvm    kubernetes.io/service-account-token   3      11s

The first secret is the one created for the default service account that we examined earlier. The second one holds the credentials for the testsa account that we’ve just created.

So, now that we have our service account ready, let’s attach it to a pod. The following definition creates a pod that uses the testsa as a service account:

apiVersion: v1
kind: Pod
metadata:
  name: testpod
spec:
  serviceAccountName: testsa
  containers:
    - name: mycontainer
      image: alpine:latest
      command:
        - "sh"
        - "-c"
        - "sleep 1000"

Perhaps the interesting part in this definition is in line 6 where we specify the serviceAccountName parameter. The container itself runs the Alpine image and we instruct it to sleep for 1000 seconds, enough for us to examine the container contents before the process exits. Now, if you apply this definition to your cluster, you can log in to the pod using a command like the following:

kubectl exec -it testpod -- sh

If you examine the contents of /var/run/secrets/kubernetes.io/serviceaccount/ you would have a token file but (when decrypted using the jwt tool) contains different service account name and credentials.

Kubernetes Authentication Methods

At the start of this article, we had Linux as an example of how authentication happens. Linux used /etc/shadow and /etc/passwd for storing user credentials. While you can use other authentication methods with Linux like LDAP, you don’t have many other options. On the other hand, Kubernetes designers wanted to give the admins more flexibility when choosing how they want to authenticate their users. Different authentication methods can be enabled/disabled through plugins. Let’s have a look at them:

• Static Password or Token: this is the simplest (yet the least flexible) method. Each of your users is provided a fixed password or token. The API server authenticates the user through HTTP basic authentication where the username and password are passed through the authorization header of the HTTP request. For example, Authorization basic bXl1c2VyOm15cGFzc3dvcmQK. Notice that the set of characters that follow the authentication type (basic) are the base64 encoded version of myuser:mypassword. The basic HTTP authentication also supports token-based credentials where the user only needs to supply a token to get authenticated. In this case, the authorization header of the HTTP request looks like this:

  Authorization Bearer: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.2Jy0uRBfQC3EIo-_iv3QE0qQMDKYLvBpkK82_7J6q0M

  Needless to say, both types of authentication must be sent over HTTPS. Otherwise, you’re locking your door while handing the keys to the intruder. This method requires that you maintain a static file for each user containing their credentials. It also requires direct access to the API server, which makes it highly inflexible. It should only be used in test/development environment where authentication is not much of a concern.
• X.509 Certificates: you can use certificates for authenticating to the API server. The workflow goes as follows: you create a certificate request, sign it through a Certificate Authority (CA) and present it to the API server during the authentication phase. The API server consults the CA server to validate the certificate and, accordingly, approves or denies the request. Notice that this mechanism has its similarities with the traditional workflow used by browsers when they need to contact websites over HTTPS.
• Single Sign-On using OpenID Connect: you can use one of the OAuth 2.0 providers like Google, Azure Active Directory, etc. to authenticate yourself to the API server. In this scenario, the user needs to login first to Google (if we decide to use Google as the OIDC providers) and - once authenticated - they send a bearer token containing the identity information. Sending this token means that the user has been successfully authenticated by the provider. The Bearer token is a JSON Web Token (discussed earlier) that contains information about the user. The API server needs the following flags:
• --oidc-issuer-url: the discovery URL provided by the authentication provider without the path. So, for example, if the discovery URL is https://accounts.google.com/.well-known/openid-configuration, the value should be https://accounts.google.com. Notice that only URLs that use https are accepted.
• --oidc-client-id: the client id for which the token will be issued. For example, kubernetes.

However, web certificates have a common name that is used to display the website name (for identity verification) and the subject for storing information related to the locality of the resource, like the country, city, division, etc. Kubernetes uses the common name field for the username and the subject for adding any namespaces that this user is part of. In this authentication method, the cluster operator/administrator is responsible for creating and distributing certificates among the users. Also, monitoring certificate expiry and revocation is part of the workflow.

The above flags are required for this authentication method to work. However, there are other optional flags that you can supply to customize the authentication process to your preference.

The above strategies are provided out of the box. However, you may need to tie the API authentication process to your existing LDAP or Kerberos authentication providers. Kubernetes offers two methods for integrating with external authentication providers:

• Authentication Proxy: using this method, the API server can extract the information it needs about the user’s identity using the HTTP headers that you specify. Let’s have a quick example to demonstrate this concept:

Assuming that you choose the X-Remote-User as the header containing the username, the X-Remote-Group for specifying the user group (or namespace), and any headers that start with the X-Remote-Extra- prefix for other additional headers that may contain extra information about the user. Now, the API server can have the following flags to enable this configuration:

--requestheader-username-headers=X-Remote-User
--requestheader-group-headers=X-Remote-Group
--requestheader-extra-headers-prefix=X-Remote-Extra-

And a typical HTTP request that holds the required information could look like this:

GET / HTTP/1.1
X-Remote-User: fido
X-Remote-Group: dogs
X-Remote-Group: dachshunds
X-Remote-Extra-Acme.com%2Fproject: some-project
X-Remote-Extra-Scopes: openid
X-Remote-Extra-Scopes: profile

However, there’s nothing that prevents an attacker from impersonating a legitimate user by using their names and namespaces. Hence, the API proxy validates the requests using the Certificate Authority. The following flag is added to the API server options:
--requestheader-client-ca-file which is a PEM-encoded certificate bundle. Any request must have a valid certificate that the API server validates first against the Certificate Authority.

• Webhook Token Authentication: in this method, you can use an external service to handle authentication to the API server. A good example of using this mechanism is through GitHub. You can allow your users to authenticate themselves to the API server by generating their own tokens on GitHub and supplying them through the authentication request. The API server checks with GitHub to ensure that the tokens are valid before approving or denying the authentication attempt. The advantage of this approach is that users can generate/revoke their own tokens instead of handing this role to the cluster administrator as in the X.509 method. To enable this method, you need to add the following flags to the API server startup command:
  --authentication-token-webhook-config-file: the configuration file that instructs the API server about how to access the external service (for example, GitHub). The configuration file follows the kubeconfig format. The following is the official example provided in the Kubernetes documentation:

# Kubernetes API version
apiVersion: v1
# kind of the API object
kind: Config
# clusters refers to the remote service.
clusters:
  - name: name-of-remote-authn-service
    cluster:
      certificate-authority: /path/to/ca.pem         # CA for verifying the remote service.
      server: https://authn.example.com/authenticate # URL of remote service to query. Must use 'https'.

# users refers to the API server's webhook configuration.
users:
  - name: name-of-api-server
    user:
      client-certificate: /path/to/cert.pem # cert for the webhook plugin to use
      client-key: /path/to/key.pem          # key matching the cert

# kubeconfig files require a context. Provide one for the API server.
current-context: webhook
contexts:
- context:
    cluster: name-of-remote-authn-service
    user: name-of-api-sever
  name: webhook

--authentication-token-webhook-cache-ttl: the grace period that the API server grants to the users before requiring them to re-supply their tokens (think of it as when you use sudo for the first time and then keep on using it for some time without needing to supply your password). The default is two minutes.

TL;DR

• Authentication refers to verifying the identity of the user or service trying to access a resource.
• Kubernetes is flexible enough to allow you to choose the authentication mechanism that suits you and your organization.
• The simplest authentication method is to use static files containing passwords or tokens for the users. This method is not recommended in production environments because you need to maintain a file for each user or service and it requires that you access the API server.
• The X.509 method is a popular authentication mechanism in which you create a certificate for each user containing the username and the groups that each user belongs to. You need to deploy a Certificate Authority and maintain the certificates generation, revocation, and expiry. However, it is still better than the static-files method.
• Using any platform that supports OIDC (OpenID Connect) like Google, you can configure the API server to accept the Bearer token that the user acquired from a successful authentication to the OIDC service.
• If you have your own authentication services like an LDAP server or a Kerberos based mechanism, you can still integrate it with Kubernetes to authenticate the users through them. Kubernetes uses the Authentication Proxy and the Webhook token authentication for those scenarios.
• The Authentication Proxy lets you select specific headers in the HTTP request to extract the required authentication information like the username and namespaces.
• The Webhook authentication allows your users to generate and use their own tokens through the external service (for example, GitHub) and use them when authenticating to the API server.