Balance innovation and agility with security and compliance
risks using a 3-step process across all cloud infrastructure.
Step up business agility without compromising
security or compliance
Everything you need to become a Kubernetes expert.
Always for free!
Everything you need to know about Magalix
culture and much more
Kubernetes has become the most popular container orchestrator in the industry. It’s powerful, scalable, and flexible enough to deploy on premises or in the cloud. With these benefits come some security concerns that are worth addressing.
Kubernetes Secrets help you manage an application's sensitive data across your cluster, reducing the potential risks associated with managing non-centralized secrets. Securing access to cluster secrets is critical because unauthorized access to private data violates personal privacy and can lead to compliance violations.
This article talks about Kubernetes secrets, why secrets are important, how you can make secrets secure, and the best practices.
One of the most challenging aspects in containerized environments like Kubernetes is the safe transfer of credentials or secrets required by applications. Kubernetes secrets are secure objects that can store and manage sensitive data such as authentication tokens, passwords, SSH keys, or cloud access keys in your clusters. Secrets in Kubernetes are stored in base64 encoded format. Kubernetes stores secrets in etcd in plain text format unless you've specified an encryption provider. A secret in Kubernetes can be consumed using environmental variables or as files in a volume.
Secrets are a powerful tool in your Kubernetes toolkit. One of the most significant advantages of using secrets is that the cluster itself manages secrets. You don't have to manage them yourself or store them elsewhere — they're just there for you when you need them. There's no complicated setup process, and there isn't any extra work involved in creating and storing secrets: they live right alongside other Kubernetes objects like pods and services.
Kubernetes offers network security resources such as the ability to control which containers and pods communicate over the network, and even some basic secrets management capabilities. However, it does not help secure secrets and only provides limited functionality.
Figure 1: Illustration of a Kubernetes Secret inside the “etcd”
One downside to Kubernetes secrets is that they aren't encrypted by default, thus exposing your system to potential security risks if someone gains access to the unencrypted data. Secrets can be leaked in several ways: source code management tools like Git, application’s source code, logs, etc. Hence, secrets should always be stored using proper encryption techniques.
You should not consider your data to be secure just by preserving the sensitive data in a secret object. Kubernetes has no robust mechanism to share, encrypt and manage secrets across the clusters. You must avoid preserving sensitive data that may include passwords, SSH keys, and authentication tokens as plain text.
How do growing development and ops teams manage a production ready cluster?
Download this white paper and find out what production ready means, the cultural changes you need to make on your team, as well as the most important requirements to consider when using and taking advantage of Kubernetes in production.
Kubernetes secrets are one of the following two types:
Built-in Secrets - Kubernetes service account automatically creates built-in secrets and attaches them to containers with API credentials. These secrets can be disabled or changed as necessary.
Custom Secrets – In Kubernetes, custom secrets help you to specify your sensitive data and then create a secret to protect it.
You can create secrets in Kubernetes both automatically and manually. You can also retrieve a secret using the "kubectl get secret" command and then decode the base64 encoding to plaintext.
Kubernetes service accounts can automatically create built-in secrets and bind them with the containers. Note that they may be deactivated or overridden. Customized Secrets allow you to enter sensitive data and to create a secret to keep it.
Kubernetes secret deployments are more secure than direct deployment in the Docker image. However, Kubernetes Secrets management also has its limitations. Kubernetes Secrets keeps passwords and usernames by encoding them in base-64 format. Keep in mind that text encoding is never secure. Furthermore, secrets are only accessible inside the cluster in which they are stored.
Secrets in Kubernetes are preserved in etcd, a high-performance, distributed key/value storage system. However, it lacks several critical features necessary for storing sensitive data, such as an audit log, insights into key age, and automated key rotation. However, etcd lacks essential security features such as automatic key rotation, audit log, etc. Moreover, since the secrets are stored in etcd unencrypted, anyone having access to the cluster can access those secrets.
Kubernetes implements several built-in security controls to provide a baseline of security for secrets:
One pod cannot access the secrets of another.
A secret is made available to a node if a pod is scheduled on that node and needs it.
The secret is saved in "tmpfs" so it is not written to persistent storage. This file system is RAM-backed and will not survive a reboot.
The maximum permissible size of secrets can only be 1 MB. This helps protect the apiserver memory resources and kubelet from abuse.
The local copy of the secret will also be deleted once the pod containing the secret has been removed.
When there are many containers in a Pod, each container must explicitly request the secret volume to be visible inside the container.
Despite the fact that these controls are essential, they are by no means adequate for production-grade implementations. Kubernetes administrators must have extra safeguards in place, particularly when dealing with systems that are not part of the Kubernetes ecosystem.
Problems with the built-in Secrets Mechanism in Kubernetes
No zero-trust system - It would help if you decrypted secrets only when it is necessary. Kubernetes, on the other hand, does not natively offer this capability. As a result, any user with root access to a Kubernetes node may be able to see the secrets inside a node. Moreover, in a zero-trust system, one you’ve encrypted a secret there is no way in which you can decrypt it.
Albeit providing support for role based access, the existing permissions for Kubernetes secrets are just get and set. Although the secrets are encrypted at rest, there is no way to enable the user to get a secret encrypted. Hence a zero-trust system is imperative.
Insecure Storage of Secrets - In Kubernetes, secrets are stored inside a data file known as "etcd." Since the contents of "etcd" are not encrypted by default, the administrators must limit their access privileges appropriately. Additionally, Kubernetes secrets are not encrypted by default and are stored as base64 strings.
So you cannot commit the files that contain these secrets into the source control as-is since anyone having access to the repository can get access to the secrets as well. Secrets can be exposed through application code, logs, debug records or even through source code management tools such as Gitlab and Github.
External Risks - Kubernetes secrets management and how other applications use your secrets also matters. Ensure that you don't expose your sensitive data by using tools and platforms with no security measures.
Securing Kubernetes is important for maintaining a safe environment where your data can be accessed only when it should be. Here are some ways to protect your data in Kubernetes environments with secrets management best practices.
A centralized secrets management solution such as Akeyless or Conjur simplifies audit, access control, and secrets management by providing enterprises with a consolidated view of their complete Kubernetes security environment. It gives organizations a centralized view of all aspects of Kubernetes security. This makes it possible to control and understand how containers and services interact.
It is a recommended practice to ensure that secrets are only available to the containers that need them. You must ensure that secrets in a container are stored securely. It would help if you did not keep secrets in the disk; instead, secrets should be accessible only to the container that needs them, stored in the memory, and disappear as soon as the container shuts down.
Almost all data in the Kubernetes API, including secrets, is kept in a distributed data store. Setup the Kubernetes cluster configuration to encrypt the data at rest to be encrypted. You should use the EncryptionConfiguration object at the cluster level to allow encryption at rest and decrease the number of keys that must be managed.
You can apply the principle of least-privilege to your network traffic so that sensitive cluster components can only be accessed from the services that need direct access. You can limit service-to-service traffic by using the traffic filtering mechanisms of your cloud provider. You can also restrict traffic between applications components by using NetworkPolicy tools.
Some of the best practices in securing your Kubernetes secrets include using a centralized secrets management solution, implementing role-based access control (RBAC), regular auditing, etc.
Secure internal communication in a Kubernetes Cluster by enabling TLS/SSL between the pods and Etcd.
Kubernetes Secrets is a mechanism for managing and accessing an application's sensitive data across the cluster. However, they are accessible to unauthorized users if you don’t follow the recommended best practices to secure them. You should enable "Encryption at Rest", restrict access to "etcd", and follow other best practices outlined earlier in this article to secure your Kubernetes secrets.
Magalix allows organizations to create, manage and deploy custom governance policies using policy-as code. It also provides a powerful policy enforcement engine. Magalix codifies your policies to maximize cloud infrastructure security. The Magalix platform offers a wide range of features, including customizable policies, compliance checks, and automatic remediation.
Self-service developer platform is all about creating a frictionless development process, boosting developer velocity, and increasing developer autonomy. Learn more about self-service platforms and why it’s important.
More and more businesses are adopting GitOps. Learn about the 5 reasons why GitOps is important for businesses.