14-days FREE Trial

 

Right-size Kubernetes cluster, boost app performance and lower cloud infrastructure cost in 5 minutes or less

 

GET STARTED

  Blog

Kubernetes Secrets 101

 

What is a Kubernetes Secret?

There are many times when a Kubernetes Pod needs to use sensitive data. Think for examples of:

  • SSH keys
  • Database passwords
  • OAuth tokens
  • API keys
  • Image registry keys

Kubernetes is designed to have a declarative syntax. Object definitions are stored in YAML (or JSON) files and - typically - placed under version control. Adding confidential information to a version-controlled file (that anyone can view) is against any security best practice. For that reason, Kubernetes includes Secrets.

A Secret is just another Kubernetes object that stores restricted data so that it can be used without being revealed. Kubernetes users can create Secrets, and also the system itself establishes and uses Secrets.

You can find Secrets referenced through a file attached to the pod through a volume. The kubelet also makes use of Secrets when it needs to pull an image from an Image Registry that requires authentication (for example, a private Docker Hub account, AWS ECR, or Google GCR). Additionally, Kubernetes makes use of Secrets internally to enable Pods to access and communicate with the apiserver component. The system automatically manages API tokens through Secrets attached to the Pods.

Creating Kubernetes Secrets Objects

There are several ways to create Secrets in Kubernetes. Your choice highly depends on the type of scenario you’re in.

Using The Command Line and Text Files

Let’s have a quick example: you have a third-party service that you need your application to access. This service requires a username and password to authenticate requests. You need to create a Secret object that contains this information. Creating this secret using a file can be done as follows:

echo -n 'superuser' > ./username.txt
echo -n 'Q%FvqS$*F$k^6i' > ./password.txt
kubectl create secret generic app-user-cred --from-file=./username.txt --from-file=./password.txt

In this example, we used echo with the -n option to ensure that no newline character is added to our text (if we include a newline character it will be part of the username and password and authentication may fail). We created a file for the username and another for the password. Then we used kubectl to create a Secret object named “app-user-cred” that uses the data from the files we’ve just created.

Using The Command Line and Literal Input

We could’ve equally generated the same password object without using files with a command like the following:

kubectl create secret generic app-user-cred --from-literal=username=devuser --from-literal=password=Q\\%FvqS\\$\\*F\\$k\\^6i

 

However, notice that we had to add some extra characters to the password. Whenever the secret text contains special characters, we need to escape them by preceding each unique character with \\, which is only required when adding Secrets through the --from-literal option. Hence, it is much easier to use content from text files when defining Secrets to avoid confusion.

Using Definition Files

Like any other Kubernetes object, Secretes can be created using a definition file written in YAML or JSON.

Using Base64 and The Data Parameter

You can define the secret text as a base64-encoded string. For example, the following procedure creates and uses a YAML file to create a Secret to save our credentials:

$ echo -n 'superuser' | base64
c3VwZXJ1c2Vy
$ echo -n 'Q%FvqS$*F$k^6i' | base64
USVGdnFTJCpGJGteNmk=

Now that we have our credentials encoded in base64, let’s create the definition file mysecret.yaml:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  superuser: YWRtaW4=
  password: USVGdnFTJCpGJGteNmk=

Apply the file as usual using kubectl as follows:

kubectl apply -f mysecret.yaml

Using Clear Text and The StringData Parameter

You can skip encoding the strings and have Kubernetes does that for you automatically:

apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
stringData:
  superuser: 'superuser'
  password: 'Q%FvqS$*F$k^6i'

Apply this definition similarly using kubectl:

kubectl apply -f mysecret.yaml

Learn how to continuously optimize your k8s cluster

You can double check that Kubernetes did the base64 encoding for you by querying the secret data as follows:

$ kubectl get secret mysecret -o yaml
apiVersion: v1
data:
  password: USVGdnFTJCpGJGteNmk=
  superuser: c3VwZXJ1c2Vy
kind: Secret
metadata:
  annotations:
	kubectl.kubernetes.io/last-applied-configuration: |
  	{"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"mysecret","namespace":"default"},"stringData":{"password":"Q%FvqS$*F$k^6i","superuser":"superuser"},"type":"Opaque"}
  creationTimestamp: "2019-07-21T14:14:56Z"
  name: mysecret
  namespace: default
  resourceVersion: "708718"
  selfLink: /api/v1/namespaces/default/secrets/mysecret
  uid: eaddc88c-abc1-11e9-8bff-025000000001
type: Opaque

 

Notice that you can use both stringData and data parameters when adding text to the Secret definition file. However, if you use stringData and data for defining the same parameter (for example, superuser), the stringData takes precedence.

Getting The Secret Data Back

As depicted in the following image, the container accesses the secret through a volume in pretty much the same way as it does with configMap volumes:

 

secrets 1

 

You can get a Secret from Kubernetes the same way you get other objects using the get subcommand. For example:

$ kubectl get secret mysecret -o yaml                                                                                                                                                       
apiVersion: v1
data:
  password: USVGdnFTJCpGJGteNmk=
  superuser: c3VwZXJ1c2Vy
kind: Secret
metadata:
  annotations:
	kubectl.kubernetes.io/last-applied-configuration: |
  	{"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"mysecret","namespace":"default"},"stringData":{"password":"Q%FvqS$*F$k^6i","superuser":"superuser"},"type":"Opaque"}
  creationTimestamp: "2019-07-21T14:14:56Z"
  name: mysecret
  namespace: default
  resourceVersion: "708718"
  selfLink: /api/v1/namespaces/default/secrets/mysecret
  uid: eaddc88c-abc1-11e9-8bff-025000000001
type: Opaque

Now, you can easily decode the base64-encoded strings as follows:

$ echo "superuser: $(base64 -D <<< 'c3VwZXJ1c2Vy')"                                                                                                                                         
superuser: superuser
$ echo "password: $(base64 -D <<< 'USVGdnFTJCpGJGteNmk=')"                                                                                                                                  
password: Q%FvqS$*F$k^6i

How You Can Use Your Defined Secrets in Kubernetes

Now that you know how to define and decode a Secret, let’s see how we can use it for different purposes like authentication.

Accessing Secrets Through Files Inside a Volume

One way of using Secrets is to attach them to the Pod that needs them as volumes. If the Pod has more than one container, then each container must define a volume mount that points to the Secret volume. Each entry in a Secret is represented by a file. Let’s have an example:

---
apiVersion: v1
kind: Secret
metadata:
  name: db-creds
type: Opaque
stringData:
  user: 'root'
  password: 'mypassword'
  host: 'mydb.example.com'
---
apiVersion: v1
kind: Pod
metadata:
  name: mysqlclient
spec:
  containers:
  - name: mysql
	image: mysql
	command: ["/bin/sh"]
	args: ["-c","mysql -u `cat /mnt/db-creds/user)` -p`cat /mnt/db-creds/password)` -h `cat /mnt/db-creds/host)`"]
	volumeMounts:
	- name: creds
  	  mountPath: "/mnt/db-creds"
  	  readOnly: true
  volumes:
  - name: creds
    secret:
  	secretName: db-creds

 

Magalix trial

The above YAML file includes a Secret, db-creds that defines the user, password, and host for a hypothetical database mydb.example.com. In the same file, we describe the Pod that will use this Secret. Setting all the related resources in the same file is common Kubernetes practice.

To be able to access the Secret, the Pod has several important attributes:

  • We define a volume that contains the Secret. The Secret name must match the one we set in the Secret definition.
  • In the containers part, we define a volumeMounts entry for each container in the Pod. If you have more than one Secret, then each one has its volume and corresponding volume mount.
  • The container should not have write access to the Secret volume. Hence, we set the readonly parameter to true.

The mountPath can point to any empty directory on the container’s filesystem. Any entries defined in the Secret are automatically converted to files (one file per entry) under this directory. The file is named after the key used to define the Secret.

Using the above configuration, we could access our remote database using the MySQL client command defined in the command and args part of the container. The command reads the contents of user, password, and host files found under the mounted path. For a real-world use case, this Pod can be deployed through a Kubernetes Job to execute SQL commands against the DB and gets terminated.

You can even override the default behavior of naming the Secret file after the name of the key by using .volumes[].secret.items array as follows:

 

 

 

volumes:
  - name: creds
	secret:
  	  secretName: db-creds
  	  items:
  	  - key: user
    	    path: db-username
  	  - key: password
    	    path: db-password
  	  - key: host
    	    path: db-host

 

Now you can refer to the files by their new names under the same mounted volume.

An important thing to notice here is that if you decide to use the .volumes[].secrets.secretName.items[] in a pod definition, then only the file paths defined in the items array will be used. If an entry was not set, it wouldn’t be accessible to the container, which can be a way to filter which secret listings are exposed to a container.

Accessing Secrets Through Environment Variables

Mounting volumes and using files that contain the Secret entries are more suited to scenarios where your application uses a file to authenticate itself to other applications. For example, key-based SSH authentication. But, for applications that use text on the command itself (like the mysql client used in our example), it is much easier and more convenient to use environment variables.

Let’s modify our Pod definition to access the Secret entries using environment variables instead of volumes. Notice that we won’t make any changes to the Secret definition.

---
apiVersion: v1
kind: Pod
metadata:
 name: mysqlclient
spec:
 containers:
 - name: mysql
   image: mysql
   env:
   - name: USER
     valueFrom:
       secretKeyRef:
         name: db-creds
         key: user
   - name: PASSWORD
     valueFrom:
       secretKeyRef:
         name: db-creds
         key: password   
   - name: HOST
     valueFrom:
       secretKeyRef:
         name: db-creds
         key: host
   command: ["/bin/sh"]
   args: ["-c","mysql -u $USER -p$PASSWORD -h $HOST"]

The changes we did here is removing the volumes and volumeMounts part and adding the env array to the container. The env array defines one or more entries that are visible to the container as environment variables. An env entry normally has a name (variable name) and the value that the variable holds. When getting its value from a secret, the env entry uses the valueFrom parameter, which searches for the Secret name and the Secret entry key through the secretKeyRef parameter.

Once all the Secret entries are exposed through environment variables to the container, they are accessed like any other environment variable. For example, $USER, $PASSWORD, and $HOST correspond to the user, password, and host Secret entries respectively.

Using ImagePullSecrets To Authenticate To Private Image Registries

Docker Hub and Google GCR are examples of image repositories that offer public as well as private image hosting. When you host your images that contain your proprietary application code, you usually choose to store those images privately. When you pull those private images on your local machine, you need to authenticate yourself first to the service. For example, Docker offers the docker login command that stores a file containing your credentials locally. This file is used whenever you need to pull or push an image to your repository. Google GCR offers a similar mechanism through its gcloud helper command. However, when Kubernetes pulls those images, the kubelet needs this file to be able to access the private registry. The imagePullSecrets parameter uses a Secret that holds the contents of the authentication file to enable the kubelet to pull the image.

Using ImagePullSecrets When You Don’t Have The Authentication File

You don’t need to create the authentication file first before injecting it into a Secret as this can be done in one step:

kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

The above command automatically creates a secret that contains the necessary data required to authenticate to the image registry. Notice that although the command mentions “docker-registry” as a parameter, it can be equally used to authenticate Kubernetes to other image registries like GCR. Just make sure you follow the image-registry’s specific authentication information.

Now, once the Secret is created, we can use it to pull our private image like in the following example:

apiVersion: v1
kind: Pod
metadata:
 name: myprivateapp
spec:
 containers:
   - name: myprivateapp
     image: magalix/privateapp:v1
 imagePullSecrets:
   - name: myregistrykey

Using ImagePullSecrets When You Already Have The Authentication File

You may be already working with the private image registry and you just need Kubernetes to use your existing authentication file. You can create a Secret object that uses the contents of your authentication file as follows:

kubectl create secret generic myregistrykey     --from-file=.dockerconfigjson=~/.docker/config.json --type=kubernetes.io/dockerconfigjson

The above command looks similar to the ones we used earlier in this article to create Secrets from files. However, there are some required settings that cannot change:

  • The file entry must have the entry name set to .dockerconfigjson
  • The type must be set to kubernetes.io/dockerconfigjson

You can create this Secret through a definition file. However, instead of specifying the path to the authentication file, you provide its content in base64 encoding. For example:

apiVersion: v1
kind: Secret
metadata:
 name: myregistrykey
 namespace: myapps
data:
 .dockerconfigjson: UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxsbGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg==
type: kubernetes.io/dockerconfigjson

Magalix Trial

TL;DR

  • Kubernetes Secrets are a way to store confidential information. They are used by Pods, the kubelet, and the system itself.
  • Secrets can be created through the command line as well as through definition files. When creating Secrets on the command line, you can either point to existing files (and have their content stored), or you can directly enter the text. Any special characters must be escaped when entering the text directly.
  • When using definition files, you can add the data in a base64 encoded format or plain text form. Kubernetes encodes the Secret data in base64 format. When you need to reveal a Secret text, you must base64-decode it.
  • To enable containers to access Secrets, you have the option to mount the Secret as a volume. All the Secret entries are represented as files under the mounted volume; each file contains the Secret content of the respective entry. You also have the option to pass them to the container as environment variables.
  • The kubelet uses Secrets when it needs to pull images from private image registries that require authentication using the imagePullSecrets parameter. You can create the authentication file and the Secret in one step, or you can bring your authentication file and store it as a Secret.
Mohamed Ahmed

Nov 13, 2019