Basics of Securing Your Kubernetes Cluster

How to Approach the Security Topic?

When it comes to cloud-native technologies, a lot of people fall into the trap of overlooking security. They mistakenly think that, since their application is hosted on somebody else’s datacenter, that somebody shall take all the required precautions to protect their business against digital attacks. While this assumption holds some truth, we have to make an important distinction between what lies on you (the client) and what is handled by the cloud provider in terms of security. For example, let’s assume that you host your application on a number of EC2 instances on AWS (Amazon Web Service). The following are some of what AWS is responsible for:

• Ensuring that the underlying infrastructure on which EC2 instances operate is secure enough so that you cannot be attacked from another EC2 instance owned by another user.
• Taking all the required measures to guarantee that no one gains access to the private key used to login to your instance.

On the other hand, AWS cannot be held liable for the following (for example):

• Your application was compromised through a SQL injection attack. The code was poorly written and, thus, vulnerable.
• Sensitive parts of your application has weak or no passwords to protect users’ data. As a result, malicious attackers will be able to steal and/or manipulate this data causing damage to your business.

The above is just an over-simplified example of what can go wrong if you don’t take security as seriously as it should be taken. Also, having a third-party to host your application is - by no means - a guarantee that you’re 100% protected. And that leads to an important question:

How Secure is “Secure”?

If you research the web for security-related material, read books, attend training, you’ll never find a source claiming that you can achieve a 100% security level. The reason is that there’s, simply, nothing as a 100%-secured resource. What all information security professionals are working hard to achieve is to make the job harder for the attacker. Let’s take passwords for example. A little more than a decade ago, eight characters were sufficient to regard a password as strong, and MD5 was a powerful hashing algorithm. But now, you can easily figure out what an md5-hashed password was originally by using rainbow tables. Decrypting MD5 became very easy to the point that you can do it online (you can try https://www.md5online.org/md5-decrypt.html for an example). So, the point that I’m trying to drive home is that, no matter how hard you try, you’re only making the job harder for the attacker for the time being. You should always be updated to date with the latest security practices, technologies, and recommendations. In this series of articles, we are helping you achieve that with respect to Kubernetes. So, we discuss topics like configuration of Kubernetes components, controlling and limiting access to them. Also, we explain how to ensure that the code you’re hosting on the cluster is not vulnerable to attacks, how to restrict containers’ behavior so that they cannot be manipulated by attackers. Finally, we discuss the recommended ways of storing and retrieving credentials in your Kubernetes cluster.

In this article, we start the series by discussing two important defense principles: the multi-layered architecture and the least-privileged approach.

The Multi-Layered Architecture

What measures would you follow when you need to secure your house?

Naturally, you’d lock the front door at all times, perhaps install an alarm system so that even if a burglar was able to unlock the door the alarm will set off. You might also take additional measures like not leaving large amounts of cash in the house and putting them in a bank instead. Alternatively, you could buy a safe and use it to store your valuables. Why are you doing this? In fact, you’re following the multi-layered security architecture when deciding how to protect your house and belongings even if you don’t know it. Let’s have a look at the layers that you have in place:

1. The lock at the door.

2. The alarm system.

3. The safe.

Now, if the attacker was able to break the door, disable the alarm, he/she would have a hard time trying to open the safe. Certainly, it might happen that someone could unlock the door, bypass the alarm, and manage to open the safe. But the question is how likely is this to happen?

Similarly, you shouldn’t rely on a single practice/security layer when designing your Kubernetes cluster security.

The Least-Privileged Approach

Let’s return to the house-security example. Would you design your door lock to use the same key that’s used to open your safe? Of course not. The reason is that if that key gets lost or stolen, it’ll be used to bypass only one security layer not all. So, if the attacker has the door key, he still needs to figure out how to unlock the safe. If he’s got the safe key, he still needs to make his way through the front door.

If we apply this analogy to software and Kubernetes in particular, you shouldn’t grant a component more privileges than it actually needs. For example, an Ingress controller should be granted read access to the services and namespaces but shouldn’t be able to modify them. If the controller was compromised, the attacker will have read-only access to other components.

Reducing the Attack Surface

In information security, the attack surface refers to all the possible ways through which an attacker can get into a system. So, which do you think is more vulnerable to burglars: a house with only one door or one with three or four gates?

The more the entrance points to a resource, the more “exposed” it is to attacks. In the software world, the same principle still holds. Assume that you are coding a Python application that needs to connect to an external API. While you were testing different HTTP access methods, you came across the requests library. And then, after using it for some time, you decided that it’s not the best approach and you settled on another module. The problem is that you forgot to remove the import requests line from your code, which means that the library is still available to the code although it is not used. Now, an attacker can use any known vulnerabilities in requests to compromise your application. Such an attack could’ve been easily mitigated by simply not importing modules that you’re not using.

Let’s start applying those principles on Kubernetes, starting with the components. The core of Kubernetes is the API server so let’s start with it.

Securing the API Server

The API server offers a REST interface for Kubernetes controllers (and users) to interact with different cluster components. It is the first and the most important core component to consider when securing your cluster.

By default, the API server listens on port 8080 for HTTP requests. If you have this default configuration then all the communication between you and the API server is sent in cleartext. Any network sniffing program (for example, Wireshark) can reveal sensitive information about your cluster including application settings (in configMaps), Secrets and so on. If you are running your cluster as a service from a cloud provider, chances are the API server is already using the secure port 443 over TLS to encrypt communications. However, you can always check whether or not this is the case by trying to send an HTTP GET request to the API server and make sure you don’t receive a valid response. For example:

curl -X GET apiserver:8080

If you have a valid output, then you need to close that port by using the --insecure-port=0 when starting the API server. Note that Kubernetes versions that are later than 1.10 may already have this option deprecated or totally removed so that the system starts at the secure port 443 by default. In all cases, you should do this check.

For your reference, securing the API server falls under the multi-layered security principle. The next layer that we should protect is the kubelet component.

Securing the Kubelet

The kubelet is an agent that sits at the top of every node in the cluster. It is responsible for receiving and executing commands against the containers running on the nodes. To do this, the kubelet has its own API interface through which it receives the instructions through HTTP requests. Securing the kubelet component involves multiple layers of its own:

• Deny anonymous access: we need to ensure that no requests are allowed to the kubelet except authenticated ones. This can be done by setting the --anonymous-auth=false flag when starting the kubelet. However, this may prevent legitimate requests from clients like the API server from getting executed. Hence, you need to also set the --kubelet-client-certificate and kubelet-client-key flags when starting the API server. Those flags ensure that the API uses a certificate to identify itself to the kubelet.
• Authorize all requests: make sure that all requests to the kubelet are authorized first. This can be done by setting the --authorization flag to one of the supported values and make sure it is NOT set to AlwaysAllow.
• Limit the kubelet’s access level: this belongs to the least-privileged principle that we discussed earlier. The kubelet component should not be able to control containers and pods on other nodes than its own. Notice that this is done on the API server rather than the kubelet itself. For example, kube-apiserver --enable-admission-plugins=...,NodeRestriction.
• Disable the read-only port: although this port (10255) is no longer supported in newer versions of Kubernetes, you should always ensure that it is not open (depending on which version of Kubernetes you are running). To do this, you add the --read-only-port=0 to the set of flags used to start the API server.

Securing the Etcd

The etcd is the distributed database that contains all the information necessary for the cluster to function correctly. If an attacker could gain write access to this database, they are effectively controlling the cluster. The following startup flags will help mitigate several risks related to running etcd:

• HTTPS connection must be enabled by setting the --cert-file and --key-file.
• Access to etcd must be limited to authorized requests only. This can be done by setting the --client-cert-auth=true. Now any client must provide a valid certificate for connecting to etcd. Additionally, you must instruct etcd on how to verify the certificates that the clients present upon attempting to initiate a connection. This can be done by setting the --trusted-ca-file to specify the certificate authority that was used to sign the certificates.
• Deny accepting any self-signed certificates by setting the --auto-tls=false.
• Configure the nodes running etcd to also use HTTPS in their intercommunication. This can be done by setting the following flags:
• --peer-client-cert-auth=true to deny requests that do not use HTTPS and present a valid certificate.
• --peer-auto-tls=false to disable self-signed certificates.
• --peer-cert-file the path to the certificate file used for establishing a secure connection.
• --peer-key-file the key for the certificate. It must be unencrypted.
• --peer-trusted-ca-file the trusted certificate authority that is used to sign the certificates.

Since the API server also communicates with etcd, a similar configuration must be applied to it so it can communicate with etcd through HTTPS.

TL;DR

• Security is a very important concern that should never be overlooked, especially in production environments.
• You can never achieve a 100% protection level. You’re only making the attacker’s job harder given the current hardware resources (fastest CPUs available, computer algorithms,etc.). A security measure that’s considered “bullet-proof” today will soon be obsolete a few years from now (or less).
• Security is approached through multiple angles: multi-layered architecture, the least-privileged approach, and reducing the attack surface.
• This is the first of a series of articles that discuss the best practices for securing your Kubernetes cluster. In this one, we started with hardening the core Kubernetes components.
• The core components include the API server, the kubelet, and the etcd database. Each of them has its own set of flags that control how it behaves. Those flags should be set in a way that mitigates security risks.
• The common hardening practice among the core components is to authenticate and authorize requests. Additionally, HTTPS should always be used over HTTP to encrypt connections between different components, and also ensure that identity is verified through certificates and a certificate authority.