Security & DevOps with Code-to-Production Security for Kubernetes Lifecycle

In this article, we will talk about the security of applications throughout the Kubernetes development and deployment lifecycle. While threats occur in different stages of the Kubernetes application lifecycle, we can take action to countermeasure these threats. We know containers are everywhere, and Kubernetes can manage these numerous and closely connected microservices. It also adds complexity to add security and ensure all microservices have the exact minimum permissions required.

Security is something we can’t toss in at the end of DevOps. We have to ensure security at the building, deploy, and runtime phases to provide a protected environment. In other words, DevOps must be evolved within DevSecOps. As DevSecOps, we have to integrate security with each step of DevOps. Firstly, we explore how we can incorporate the security layer into our DevOps processes.

Security in the DevOps Process

The whole idea of DevOps is to prepare developers to develop and deploy applications to the server. As we move forward, the need for security for DevOps rises, which introduces us to the term DevSecOps. DevSecOps is the set of practices that combines development, deployment and integrating security into the Code to Production Lifecycle. The application team is also responsible for integrating security in DevOps processes, as well as training DevOps engineers to write secure code.

DevOps is of no value if the program does not increase release velocity, however, manually checking for security can be overwhelming. We can eliminate the threat by integrating security in DevOps processes.

Collaboration is a critical part of DevOps culture. Developer and Operations teams are working collectively, and we have to include security people to put their thoughts on multiple aspects, such as:

• Providing security and a checklist for a developer to integrate with their code.
• Offering best practices, developing secure code to prevent typical attacks like SQL injection or cross-site scripting.
• Help the operation team keep up with a secure configuration for the cloud, such as using secrets.
• Focusing on providing security with agility. Back then, security was creating a bottleneck and making long releases span.
• Creating strategies to integrate security in DevOps processes.

Security with Observability

We can secure the infrastructure by monitoring and auditing the cluster through its application lifecycle. We have to put an automated mechanism in place to pass certain checks when deploying it into infrastructure. We can secure the cluster with the following tactics:

• Placing Monitoring and Auditing at the service level so it can overlook pods, containers, and clusters.
• Using the alert mechanism to create a message on observing compliance or security-related issues.
• Having real-time visibility on data or config changing.

Automating Security

Automating security helps reduce human manual intervention, making the process less error-prone. Automating security also helps remove bottleneck problems, which are created by manual checks and approvals. When talking about the Cloud, Kubernetes, hundreds of microservices and clusters spread across multiple clouds, it is nearly impossible to manually validate everything. We can use the following strategies to automate security:

• Use an application security testing tool that can look for known vulnerability and policy violations, such as preventing the use of specific libraries.
• Automating infrastructure provisioning.
• Controlling container registries access.
• Automatically updating security patches.
• Integrating security into the CI/CD pipeline to ensure all container images are checked for a production push.

Now that we’ve seen how security has been a part of the DevOps process and how to use Kubernetes security from code to production lifecycle, let’s move forward with the actual stages such as the build, deploy, and runtime of the Kubernetes application.

Stage 1: Build Phase

The first stage of the Kubernetes lifecycle is building, or packaging containers after the code part is completed. There are two parts to secure this stage- building a secure image and scanning these images for known threats. Utilizing tight security at the start eliminates threats and saves us from solving the problem at a later stage. Here are some recommendations to eliminate hazards:

• Compact base images: Avoid using the OS package manager or shell, which may contain vulnerabilities. Consider using distro-less images.
• Remove unnecessary components: Remove unnecessary tools such as debugging tools or curl, which are of no use in a production environment.
• Latest image: Enable the usage of the updated image and ensure these images are using the latest components.
• Use an Image Scanner: Using an image scanner and Integrating it with CI/CD helps remove vulnerabilities at early stages.
• Minimizing surface attack: Implementing security at the CI/CD level or enabling security people to collaboratively participate in a process to plan the need for security enables the DevOps team to eliminate threats before production push.

Stage 2: Deploy Phase

The second phase is a deployment where we deploy our workloads to Kubernetes. When deploying your workload to the Kubernetes server, we have to be aware of what has been deployed or compliant with organizational policies, or that permissions given to these containers are not overly permissive.

Firstly we need to create a secure network by deep visibility and protection. This layer is the first defense for keeping threats away, as well as the line of defense for data breaches. Let’s discuss best practices to avoid vulnerabilities in the deployment phase:

• Use Network policies to isolate workload: By default, Kubernetes allows each pod to talk to every other pod in the cluster. Fortunately, we can use Network policies to isolate or define which pod can talk to other pods.
• Container privileges: Privileges given to containers directly impact the security of the Kubernetes deploy phase. There is a rule while giving permission, “Least is the best”. Try to provide the least permission to a container required to perform its intended task.
• We will be using pod security policies to control the security of the network.
  • Don’t allow privilege escalation.
  • Don’t use the host network or process space.
  • Don’t run application processes as root.
  • Give each application its own Kubernetes Service Account.
  • Don’t mount service account credentials if it does not need to access the Kubernetes API.
  • Don’t select unused and unnecessary Linux capabilities.
• Use trusted registries: Always use trusted registries, and never deploy code from unknown code registries, as this can expose data and security loopholes.
• Use labels and annotation: This doesn’t directly help to solve security issues but makes it easier to create alerts for specific problems.
• Use Kubernetes RBAC: Kubernetes Role-Based Access Control allows controlling authorization to access the Kubernetes API server for user service accounts and is highly configurable.

We have seen many ways to tackle threats, and there are more ways to tackle build phase security, however, we can’t discuss all of them in a single post.

Now, let's move to the last phase, i.e., the Runtime phase.

Stage 3: Runtime Phase

This phase exposes many threats and vulnerabilities as we are now serving traffic. The goal is to gain visibility into your server and detect threats. If we have securely created the server by using best practices at the build and deploy phase, it’s unlikely that threats will occur, but what little threats arise, we have to detect and respond to them right away. Monitoring the container is easy because of its declarative nature. This also allows us to easily introspect into what you have deployed and its activity. Here are some best practices to avoid vulnerabilities in the runtime phase:

• Scan for vulnerabilities in running workload: Use tools like Kubei and Kube-scan to scan for vulnerabilities and expose threats to the server, then eliminate these threats. You can also use tools like Magalix, which helps in the visibility of your cluster in runtime.
• Monitor Network traffic: Monitor your network and compare the traffic allowed with the network policies as containerized application is extensively using networking, monitoring the active flow of traffic and identifying unexpected communication.
• If breached, kill a pod: Use Kubernetes controls to contain a successful breach by automatically instructing Kubernetes to scale suspicious pods down to zero or kill then restart the workload of breached applications.
• Use Kubernetes Built-in tools: Use Kubernetes native control, using a security context for a pod to limit their capabilities. This type of control helps eliminate the entire class of threat that depends on privilege access. Here’s an example: limiting filesystem access can prevent any threat that depends upon installing malicious code or writing into the filesystem.

We have discussed the different types of threats that can slow down the cluster at various stages and proactively how to secure them. Let’s discuss some final Thoughts.

Final Thoughts

So far, we have seen how to secure the workloads, but we also need to secure infrastructure by securing its individual Kubernetes master components as far as security concerns.

• Always update your Kubernetes software; every update comes with the latest security patches.
• Make sure to disable anonymous access and use TLS encryption for connection between Kubelete and API server.
• Secure etc. database, which is the source of truth for the server. Make sure this allows traffic only from the TLS connection.

Magalix OPA

You can also use Magalix OPA, which enables us to define organizational policies as code so we won’t be compromising on security. It also helps in monitoring workload issues that are violating these rules. Magalix is also used to monitor resources for over or under-utilization of computed resources.

Conclusion

• Security is the main concern for robust application environment
• DevOps makes the release faster, which helps in gaining an advantage in the market, but not prioritizing security will lead you to pay a higher cost during a later stage of the application.
• Integrating security elements in DevOps processes is known as DevSecOps.
• DevSecOps make Kubernetes clusters secure and faster.
• To enhance security, we have to integrate security into the Kubernetes lifecycle, which comprises three stages: build, deploy and runtime.
• Integrating security slows down the process but helps avoid future threats and vulnerabilities.