Weaveworks 2022.03 release featuring Magalix PaC | Learn more
Balance innovation and agility with security and compliance
risks using a 3-step process across all cloud infrastructure.
Step up business agility without compromising
security or compliance
Everything you need to become a Kubernetes expert.
Always for free!
Everything you need to know about Magalix
culture and much more
Kubernetes Authentication means validating the identity of who or what is issuing the request. For example, if you want to access a Linux box through SSH, the SSH daemon must verify that the username and password you are using for login matches an account that lives in /etc/passwd and /etc/shadow files. For them to be Combined together, those files represent the authentication database for Linux. On the other hand, authorization refers to what you are allowed to do once you gain access. So, back to our Linux example, the user can gain access to the operating system by providing valid credentials, but they cannot view or modify the contents of sensitive files like /etc/shadow. Only the root user or someone with root privileges can do this.
In Kubernetes, the API server needs to authenticate every request that it receives. A request may originate from a human user or a program (like a Pod for example). Currently, human user accounts need to be managed outside the cluster through one of the supported authentication strategies. Notice that a user account is not namespaced, which means that it must be unique cluster-wide. But more often than not, you also need to enable programs to get authenticated and send requests to the API server. For a resource to be able to authenticate to the API server, it needs to possess a service account.
A service account is another Kubernetes resource (like pods, deployments, etc.). Unlike user accounts, the service account is namespaced so you can create multiple service accounts with the same name as long as they live in different namespaces.
By default, the service account credentials are mounted to the pods through a secret. Let’s have a quick example:
$ kubectl run -it --rm testpod --restart=Never --image=alpine -- sh
If you don't see a command prompt, try pressing enter.
/ # ls -l /var/run/secrets/kubernetes.io/serviceaccount/
total 0
lrwxrwxrwx 1 root root 13 Nov 8 11:45 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root 16 Nov 8 11:45 namespace -> ..data/namespace
lrwxrwxrwx 1 root root 12 Nov 8 11:45 token -> ..data/token
This is a one-time pod that hosts an Alpine image. Opening a shell to the container, we could easily view the files that were mounted to the pod from an outside secret under the /data mount point. The authentication process is done through the token file. Let’s have a look at the contents of this file:
/ # cat /var/run/secrets/kubernetes.io/serviceaccount/token && echo
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tcHh6ZjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUzMzFmNTU1LTAxOTktMTFlYS1hYTVhLTQyMDEwYTgwMDBhYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.I3jFqRd7ANj4BidzoW-v_YO1N_z2p33Ex1biaz_J97QK9CRlIFmK2KC1k0RbQux3zo72xsLhbPjNH40pBzqr2TzVjZKFD7T8-ihDj1Og5L_BAud-CtT4e-0zcvC5rTKcWzkUn5a64TUSwF5Q6I3KjeaE3pmDFwG4I6XLuODCVOwRkfn0V_LzjqLy3nXfUK8FpIkeBrRd9QN68PG9YFH9lKMgGZLvuG_m6K6EFxqBpVMec8SPsG77GGRezC9Mjsyxp2Wie-j8vUVb5et2o4ShKa8sp-Nqum4bpBxGQ9NBo3qlefuqYGYivGZBnL8dienBLFbO5swSHUL6vxTgaRg2kA
/ #
The contents of the token is of the JSON Web Token (JWT) format. It can be decoded using the jwt command-line tool or online on the jwt.io website. The following is the output of the decoded operation on the web likewise on the command-line:
And on the command line:
$ jwt eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tcHh6ZjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUzMzFmNTU1LTAxOTktMTFlYS1hYTVhLTQyMDEwYTgwMDBhYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.I3jFqRd7ANj4BidzoW-v_YO1N_z2p33Ex1biaz_J97QK9CRlIFmK2KC1k0RbQux3zo72xsLhbPjNH40pBzqr2TzVjZKFD7T8-ihDj1Og5L_BAud-CtT4e-0zcvC5rTKcWzkUn5a64TUSwF5Q6I3KjeaE3pmDFwG4I6XLuODCVOwRkfn0V_LzjqLy3nXfUK8FpIkeBrRd9QN68PG9YFH9lKMgGZLvuG_m6K6EFxqBpVMec8SPsG77GGRezC9Mjsyxp2Wie-j8vUVb5et2o4ShKa8sp-Nqum4bpBxGQ9NBo3qlefuqYGYivGZBnL8dienBLFbO5swSHUL6vxTgaRg2kA
To verify on jwt.io:
https://jwt.io/#id_token=eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tcHh6ZjciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImUzMzFmNTU1LTAxOTktMTFlYS1hYTVhLTQyMDEwYTgwMDBhYSIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.I3jFqRd7ANj4BidzoW-v_YO1N_z2p33Ex1biaz_J97QK9CRlIFmK2KC1k0RbQux3zo72xsLhbPjNH40pBzqr2TzVjZKFD7T8-ihDj1Og5L_BAud-CtT4e-0zcvC5rTKcWzkUn5a64TUSwF5Q6I3KjeaE3pmDFwG4I6XLuODCVOwRkfn0V_LzjqLy3nXfUK8FpIkeBrRd9QN68PG9YFH9lKMgGZLvuG_m6K6EFxqBpVMec8SPsG77GGRezC9Mjsyxp2Wie-j8vUVb5et2o4ShKa8sp-Nqum4bpBxGQ9NBo3qlefuqYGYivGZBnL8dienBLFbO5swSHUL6vxTgaRg2kA
✻ Header
{
"alg": "RS256",
"kid": ""
}
✻ Payload
{
"iss": "kubernetes/serviceaccount",
"kubernetes.io/serviceaccount/namespace": "default",
"kubernetes.io/serviceaccount/secret.name": "default-token-pxzf7",
"kubernetes.io/serviceaccount/service-account.name": "default",
"kubernetes.io/serviceaccount/service-account.uid": "e331f555-0199-11ea-aa5a-42010a8000aa",
"sub": "system:serviceaccount:default:default"
}
✻ Signature I3jFqRd7ANj4BidzoW-v_YO1N_z2p33Ex1biaz_J97QK9CRlIFmK2KC1k0RbQux3zo72xsLhbPjNH40pBzqr2TzVjZKFD7T8-ihDj1Og5L_BAud-CtT4e-0zcvC5rTKcWzkUn5a64TUSwF5Q6I3KjeaE3pmDFwG4I6XLuODCVOwRkfn0V_LzjqLy3nXfUK8FpIkeBrRd9QN68PG9YFH9lKMgGZLvuG_m6K6EFxqBpVMec8SPsG77GGRezC9Mjsyxp2Wie-j8vUVb5et2o4ShKa8sp-Nqum4bpBxGQ9NBo3qlefuqYGYivGZBnL8dienBLFbO5swSHUL6vxTgaRg2kA
The interesting part of the output is the Payload. As you can see, we have the service-account.name set to “default”. This is the default service account that gets created and mounted automatically to every pod upon initialization. In most cases, you shouldn’t use the default account when you need to give your pod more privileges. Let’s create a service account and assign it to a pod:
$ kubectl create serviceaccount testsa
serviceaccount/testsa created
Notice that we did not specify a namespace here so our testsa service account is created in the default namespace. If we take a look at the Secrets in this namespace we shall find two secrets:
$ kubectl get secrets
NAME TYPE DATA AGE
default-token-pxzf7 kubernetes.io/service-account-token 3 16h
testsa-token-zwhvm kubernetes.io/service-account-token 3 11s
The first secret is the one created for the default service account that we examined earlier. The second one holds the credentials for the testsa account that we’ve just created.
So, now that we have our service account ready, let’s attach it to a pod. The following definition creates a pod that uses the testsa as a service account:
apiVersion: v1
kind: Pod
metadata:
name: testpod
spec:
serviceAccountName: testsa
containers:
- name: mycontainer
image: alpine:latest
command:
- "sh"
- "-c"
- "sleep 1000"
Perhaps the interesting part in this definition is in line 6 where we specify the serviceAccountName parameter. The container itself runs the Alpine image and we instruct it to sleep for 1000 seconds, enough for us to examine the container contents before the process exits. Now, if you apply this definition to your cluster, you can log in to the pod using a command like the following:
kubectl exec -it testpod -- sh
If you examine the contents of /var/run/secrets/kubernetes.io/serviceaccount/ you would have a token file but (when decrypted using the jwt tool) contains different service account name and credentials.
At the start of this article, we had Linux as an example of how authentication happens. Linux used /etc/shadow and /etc/passwd for storing user credentials. While you can use other authentication methods with Linux like LDAP, you don’t have many other options. On the other hand, Kubernetes designers wanted to give the admins more flexibility when choosing how they want to authenticate their users. Different authentication methods can be enabled/disabled through plugins. Let’s have a look at them:
Authorization Bearer: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.2Jy0uRBfQC3EIo-_iv3QE0qQMDKYLvBpkK82_7J6q0M
Needless to say, both types of authentication must be sent over HTTPS. Otherwise, you’re locking your door while handing the keys to the intruder. This method requires that you maintain a static file for each user containing their credentials. It also requires direct access to the API server, which makes it highly inflexible. It should only be used in test/development environment where authentication is not much of a concern.However, web certificates have a common name that is used to display the website name (for identity verification) and the subject for storing information related to the locality of the resource, like the country, city, division, etc. Kubernetes uses the common name field for the username and the subject for adding any namespaces that this user is part of. In this authentication method, the cluster operator/administrator is responsible for creating and distributing certificates among the users. Also, monitoring certificate expiry and revocation is part of the workflow.
The above flags are required for this authentication method to work. However, there are other optional flags that you can supply to customize the authentication process to your preference.
The above strategies are provided out of the box. However, you may need to tie the API authentication process to your existing LDAP or Kerberos authentication providers. Kubernetes offers two methods for integrating with external authentication providers:
Assuming that you choose the X-Remote-User as the header containing the username, the X-Remote-Group for specifying the user group (or namespace), and any headers that start with the X-Remote-Extra- prefix for other additional headers that may contain extra information about the user. Now, the API server can have the following flags to enable this configuration:
--requestheader-username-headers=X-Remote-User
--requestheader-group-headers=X-Remote-Group
--requestheader-extra-headers-prefix=X-Remote-Extra-
And a typical HTTP request that holds the required information could look like this:
GET / HTTP/1.1
X-Remote-User: fido
X-Remote-Group: dogs
X-Remote-Group: dachshunds
X-Remote-Extra-Acme.com%2Fproject: some-project
X-Remote-Extra-Scopes: openid
X-Remote-Extra-Scopes: profile
However, there’s nothing that prevents an attacker from impersonating a legitimate user by using their names and namespaces. Hence, the API proxy validates the requests using the Certificate Authority. The following flag is added to the API server options:
--requestheader-client-ca-file which is a PEM-encoded certificate bundle. Any request must have a valid certificate that the API server validates first against the Certificate Authority.
# Kubernetes API version
apiVersion: v1
# kind of the API object
kind: Config
# clusters refers to the remote service.
clusters:
- name: name-of-remote-authn-service
cluster:
certificate-authority: /path/to/ca.pem # CA for verifying the remote service.
server: https://authn.example.com/authenticate # URL of remote service to query. Must use 'https'.
# users refers to the API server's webhook configuration.
users:
- name: name-of-api-server
user:
client-certificate: /path/to/cert.pem # cert for the webhook plugin to use
client-key: /path/to/key.pem # key matching the cert
# kubeconfig files require a context. Provide one for the API server.
current-context: webhook
contexts:
- context:
cluster: name-of-remote-authn-service
user: name-of-api-sever
name: webhook
--authentication-token-webhook-cache-ttl: the grace period that the API server grants to the users before requiring them to re-supply their tokens (think of it as when you use sudo for the first time and then keep on using it for some time without needing to supply your password). The default is two minutes.
Empower developers to delivery secure and compliant software with trusted application delivery and policy as code. Learn more.
Automate your deployments with continuous application delivery and GitOps. Read this blog to learn more.
This article explains the differences between hybrid and multi-cloud model and how GitOps is an effective way of managing these approaches. Learn more.
Implement the proper governance and operational excellence in your Kubernetes clusters.
Comments and Responses