Kubernetes Secrets and their Security

Kubernetes has become the most popular container orchestrator in the industry. It’s powerful, scalable, and flexible enough to deploy on premises or in the cloud. With these benefits come some security concerns that are worth addressing. 

Kubernetes Secrets help you manage an application's sensitive data across your cluster, reducing the potential risks associated with managing non-centralized secrets. Securing access to cluster secrets is critical because unauthorized access to private data violates personal privacy and can lead to compliance violations. 

This article talks about Kubernetes secrets, why secrets are important, how you can make secrets secure, and the best practices.

What are K8s Secrets?

One of the most challenging aspects in containerized environments like Kubernetes is the safe transfer of credentials or secrets required by applications. Kubernetes secrets are secure objects that can store and manage sensitive data such as authentication tokens, passwords, SSH keys, or cloud access keys in your clusters. Secrets in Kubernetes are stored in base64 encoded format. Kubernetes stores secrets in etcd in plain text format unless you've specified an encryption provider. A secret in Kubernetes can be consumed using environmental variables or as files in a volume.

Secrets are a powerful tool in your Kubernetes toolkit. One of the most significant advantages of using secrets is that the cluster itself manages secrets. You don't have to manage them yourself or store them elsewhere — they're just there for you when you need them. There's no complicated setup process, and there isn't any extra work involved in creating and storing secrets: they live right alongside other Kubernetes objects like pods and services.

How Secure are your K8s Secrets?

Kubernetes offers network security resources such as the ability to control which containers and pods communicate over the network, and even some basic secrets management capabilities. However, it does not help secure secrets and only provides limited functionality.

Figure 1: Illustration of a Kubernetes Secret inside the “etcd”

One downside to Kubernetes secrets is that they aren't encrypted by default, thus exposing your system to potential security risks if someone gains access to the unencrypted data. Secrets can be leaked in several ways: source code management tools like Git, application’s source code, logs, etc. Hence, secrets should always be stored using proper encryption techniques.

You should not consider your data to be secure just by preserving the sensitive data in a secret object. Kubernetes has no robust mechanism to share, encrypt and manage secrets across the clusters. You must avoid preserving sensitive data that may include passwords, SSH keys, and authentication tokens as plain text.

How do growing development and ops teams manage a production ready cluster?

Download this white paper and find out what production ready means, the cultural changes you need to make on your team, as well as the most important requirements to consider when using and taking advantage of Kubernetes in production.

How Do Kubernetes Secrets Work?

Kubernetes secrets are one of the following two types:

• Built-in Secrets - Kubernetes service account automatically creates built-in secrets and attaches them to containers with API credentials. These secrets can be disabled or changed as necessary. 

• Custom Secrets – In Kubernetes, custom secrets help you to specify your sensitive data and then create a secret to protect it.

You can create secrets in Kubernetes both automatically and manually. You can also retrieve a secret using the "kubectl get secret" command and then decode the base64 encoding to plaintext.

Kubernetes Cluster Secrets Challenges

Kubernetes service accounts can automatically create built-in secrets and bind them with the containers. Note that they may be deactivated or overridden. Customized Secrets allow you to enter sensitive data and to create a secret to keep it. 

Kubernetes secret deployments are more secure than direct deployment in the Docker image. However, Kubernetes Secrets management also has its limitations. Kubernetes Secrets keeps passwords and usernames by encoding them in base-64 format. Keep in mind that text encoding is never secure. Furthermore, secrets are only accessible inside the cluster in which they are stored.

Secrets in Kubernetes are preserved in etcd, a high-performance, distributed key/value storage system. However, it lacks several critical features necessary for storing sensitive data, such as an audit log, insights into key age, and automated key rotation. However, etcd lacks essential security features such as automatic key rotation, audit log, etc. Moreover, since the secrets are stored in etcd unencrypted, anyone having access to the cluster can access those secrets.

Built-in Security Controls in Kubernetes

Kubernetes implements several built-in security controls to provide a baseline of security for secrets:

• One pod cannot access the secrets of another.

• A secret is made available to a node if a pod is scheduled on that node and needs it.

• The secret is saved in "tmpfs" so it is not written to persistent storage. This file system is RAM-backed and will not survive a reboot.

• The maximum permissible size of secrets can only be 1 MB. This helps protect the apiserver memory resources and kubelet from abuse.

• The local copy of the secret will also be deleted once the pod containing the secret has been removed.

• When there are many containers in a Pod, each container must explicitly request the secret volume to be visible inside the container.

Despite the fact that these controls are essential, they are by no means adequate for production-grade implementations. Kubernetes administrators must have extra safeguards in place, particularly when dealing with systems that are not part of the Kubernetes ecosystem.

Problems with the built-in Secrets Mechanism in Kubernetes

No zero-trust system - It would help if you decrypted secrets only when it is necessary. Kubernetes, on the other hand, does not natively offer this capability. As a result, any user with root access to a Kubernetes node may be able to see the secrets inside a node. Moreover, in a zero-trust system, one you’ve encrypted a secret there is no way in which you can decrypt it.

Albeit providing support for role based access, the existing permissions for Kubernetes secrets are just get and set. Although the secrets are encrypted at rest, there is no way to enable the user to get a secret encrypted. Hence a zero-trust system is imperative.

Insecure Storage of Secrets - In Kubernetes, secrets are stored inside a data file known as "etcd." Since the contents of "etcd" are not encrypted by default, the administrators must limit their access privileges appropriately. Additionally, Kubernetes secrets are not encrypted by default and are stored as base64 strings. 

So you cannot commit the files that contain these secrets into the source control as-is since anyone having access to the repository can get access to the secrets as well. Secrets can be exposed through application code, logs, debug records or even through source code management tools such as Gitlab and Github.

External Risks - Kubernetes secrets management and how other applications use your secrets also matters. Ensure that you don't expose your sensitive data by using tools and platforms with no security measures.

Kubernetes Secrets Management Best Practices

Securing Kubernetes is important for maintaining a safe environment where your data can be accessed only when it should be. Here are some ways to protect your data in Kubernetes environments with secrets management best practices.

Use a Centralized Secret Management Solution

A centralized secrets management solution such as Akeyless or Conjur simplifies audit, access control, and secrets management by providing enterprises with a consolidated view of their complete Kubernetes security environment. It gives organizations a centralized view of all aspects of Kubernetes security. This makes it possible to control and understand how containers and services interact.

Secret Management in a Container Environment

It is a recommended practice to ensure that secrets are only available to the containers that need them. You must ensure that secrets in a container are stored securely. It would help if you did not keep secrets in the disk; instead, secrets should be accessible only to the container that needs them, stored in the memory, and disappear as soon as the container shuts down.

Configuration Best Practices

Almost all data in the Kubernetes API, including secrets, is kept in a distributed data store. Setup the Kubernetes cluster configuration to encrypt the data at rest to be encrypted. You should use the EncryptionConfiguration object at the cluster level to allow encryption at rest and decrease the number of keys that must be managed.

Networking Best Practices

You can apply the principle of least-privilege to your network traffic so that sensitive cluster components can only be accessed from the services that need direct access. You can limit service-to-service traffic by using the traffic filtering mechanisms of your cloud provider. You can also restrict traffic between applications components by using NetworkPolicy tools.

Security Best Practices

Some of the best practices in securing your Kubernetes secrets include using a centralized secrets management solution, implementing role-based access control (RBAC), regular auditing, etc.

Secure internal communication in a Kubernetes Cluster by enabling TLS/SSL between the pods and Etcd.

• It would help if you encrypted the secret keys so that anyone who tries to steal your secret will be unable to see it. To access the secret keys, you would need to decrypt it using a private key. 
• You should harden the system by ensuring that the secrets stored in etcd are encrypted before they are written. Additionally, you should have strong key management for your symmetric encryption keys so that you can rotate keys often. Access to etcd should be restricted - only the APIserver should be allowed to access the etcd.
• There should be a certificate based mutual authentication in place. It is recommended that you use a different certificate authority (CA) for safeguarding access to etcd than the one you use for protecting access to Kubernetes.
• Since secrets inside a cluster might be exposed to all cluster users, you must exercise caution while uploading personal credentials, mainly if you operate in a shared workplace. It would help if you avoided including environmental variables in your logging statements since this might disclose the secret in plaintext inside the logs.
• You must enable Encryption-at-rest - this is a feature that encrypts the data contained inside "etcd." Since this feature is not enabled by default, you should enable it explicitly to encrypt the data saved in "etcd."
• Kubernetes leverages Etcd to store cluster data. Incidentally, "etcd" is a distributed, highly available, and consistent key-value store. This data may contain sensitive information regarding access to deployed services. So, you need to restrict access to "etcd" by giving access only to the admin users.

Conclusion

Kubernetes Secrets is a mechanism for managing and accessing an application's sensitive data across the cluster. However, they are accessible to unauthorized users if you don’t follow the recommended best practices to secure them. You should enable "Encryption at Rest", restrict access to "etcd", and follow other best practices outlined earlier in this article to secure your Kubernetes secrets.

Magalix allows organizations to create, manage and deploy custom governance policies using policy-as code. It also provides a powerful policy enforcement engine. Magalix codifies your policies to maximize cloud infrastructure security. The Magalix platform offers a wide range of features, including customizable policies, compliance checks, and automatic remediation.