Enforcing A Min And Max Number Of Kubernetes Pod Replicas Using OPA

Welcome to a new article in our OPA series. If you haven’t already done so, please go through our previous articles in this series to gain an understanding of what the Open Policy Agent is all about, how you can integrate it with Kubernetes, and how it can help you enforce security policies and best practices.

In this article, we’ll see how we can apply a minimum number of replicas per application.

Requirement: Ensure High Availability For Specific Applications

While the Deployment resource ensures high availability by Allowing you to specify the number of Pod replicas, it does nothing to enforce this requirement. The number of replicas remains an option that you may or may not choose while writing the definition. However, in many cases, it’s not acceptable that your core microservices survive in the cluster with only one, or even two replicas. Due to both the expected high-volume of traffic, and the criticality of the tasks performed by this service, you must ensure that at least a specific number of replicas are maintained by deployment for this application.

On the other hand, you may have services of less importance, for example, log collection and analysis. You don’t want precious cluster resources wasted on maintaining an unnecessarily high number of replicas of them.

We always state the purpose of our policy in human-readable language first:

“I need to enforce a minimum number of replicas for specific applications and a maximum limit of replicas for other ones.”

In most cases, it will probably be a number of applications that share the same level of criticality, rather than one single application for which you need to enforce a minimum or maximum replica count. For this reason, it is recommended that you define the policy input parameters as Namespace annotations. That way you can say that all applications in the core namespace should have this replicas-count floor, or that all Deployments in the logging namespace should not exceed the replicas count ceiling.

Let’s depict the desired workflow in the following diagram:

Describing The Policy In Rego

As you probably know, Rego is a language designed specifically for writing OPA policies. Based on the above requirements, our enforce-replica-count.rego file may look as follows:

package kubernetes.admission
# Acquire the existing namespaces defined in the cluster
import data.kubernetes.namespaces
# Get the number of requested replicas from the request
replicas = input.request.object.spec.replicas
# Get the namespace name from the request
namespace := input.request.object.metadata.namespace
# Get the minimum limit (if any) from the deployment's namespace annotation
minimum = to_number(namespaces[namespace].metadata.annotations["replicas-min"])
# Get the maximum limit (if any) from the deployment's namespace annotation
maximum = to_number(namespaces[namespace].metadata.annotations["replicas-max"])
# Get the deployment name from the request
deployment_name = input.request.object.metadata.name
deny[msg]{
	# We are interested in Deployment requests only
	input.request.kind.kind == "Deployment"
    # Evaluate the requested replica count against the one defined in the namespace annotation
    replicas < minimum
	msg := sprintf("The Deployment %v could not be created because it requests %v replicas which is less than the minimum %v",[deployment_name,replicas,minimum])
}
deny[msg]{
	# We are interested in Deployment requests only
	input.request.kind.kind == "Deployment"
    # Evaluate the requested replica count against the one defined in the namespace annotation
    replicas > maximum
	msg := sprintf("The Deployment %v could not be created because it requests %v replicas which is more than the maximum %v",[deployment_name,replicas,maximum])
}

A few points to be understand about this policy definition:

• We define several variables in the global scope (lines 5 - 11). That is, those variables are available to any policy in the document.
• Two policies with the same name were created. In Rego, policies with the same name are combined with a logical OR. This allows us to evaluate the Deployment replica count regardless of the limit type imposed by the namespace (maximum or minimum).
• We use the to_number built-in function in Rego to convert the replica count obtained from the namespace annotation to a number so that we can perform logical comparisons against it.

Let’s apply the policy by creating a ConfigMap from the Rego file:

kubectl create configmap  enforce-replica-count -n opa --from-file=enforce-replica-count.rego

And ensure that the policy was accepted with no issues:

$ kubectl get cm enforce-replica-count -o json | jq '.metadata.annotations'
{
  "openpolicyagent.org/policy-status": "{\"status\":\"ok\"}"
}

Hint: Modifying The Configmap Without Having To Delete And Recreate It

Sometimes, you may want to make changes to the Rego file and apply it to the ConfigMap without having to delete and recreate it. To do that, you can use a command like the following:

kubectl create configmap  enforce-replica-count -n opa--from-file=enforce-replica-count.rego -o yaml --dry-run=client | kubectl replace -f -

Exercising The OPA Policy

The first thing we need to do is define our constraints. As discussed before, a good place to specify our minimum and maximum replica count is through the Namespace annotations. Let’s create two namespaces productions and test and impose our limits. The namespaces.yaml file may look as follows:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    replicas-min: "6"
  name: production
---
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    replicas-max: "4"
  name: test

Now, let’s create a deployment file for our production environment that abides by our limits. Our deployment.yaml file should look as follows:

apiVersion: apps/v1 
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: production
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 7
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Application should not be a problem:

$ kubectl apply -f deployment.yaml
deployment.apps/nginx-deployment created
$ kubectl get pods -n production
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-6b474476c4-8p7lr   1/1     Running   0          46s
nginx-deployment-6b474476c4-8pglz   1/1     Running   0          46s
nginx-deployment-6b474476c4-f9dzl   1/1     Running   0          46s
nginx-deployment-6b474476c4-gbk7c   1/1     Running   0          46s
nginx-deployment-6b474476c4-gqxqh   1/1     Running   0          46s
nginx-deployment-6b474476c4-hf94v   1/1     Running   0          46s
nginx-deployment-6b474476c4-kxkkn   1/1     Running   0          46s

Now, let’s see what happens when we change the number of replicas to 5:

$ kubectl edit deployments.apps nginx-deployment -n production
error: deployments.apps "nginx-deployment" could not be patched: admission webhook "validating-webhook.openpolicyagent.org" denied the request: The Deployment nginx-deployment could not be created because it requests 5 replicas which is less than the minimum 6
You can run `kubectl replace -f /var/folders/9y/yl3b764s2s3bqft88cwyk_9c0000gn/T/kubectl-edit-ailee.yaml` to try this update again.

Let’s try creating the deployment in the test namespace with more replicas than allowed. Our deployment-test.yaml file should look as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: test
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 7
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Applying this definition should violate the policy:

$ kubectl apply -f deployment.yaml
Error from server (The Deployment nginx-deployment could not be created because it requests 7 replicas which is more than the maximum 4): error when creating "deployment.yaml": admission webhook "validating-webhook.openpolicyagent.org" denied the request: The Deployment nginx-deployment could not be created because it requests 7 replicas which is more than the maximum 4

Already working in production with Kubernetes? Want to know more about kubernetes application patterns?

👇👇

Can We Apply Maximum And Minimum Constraints At The Same Time?

Sometimes, you may have applications that need to have a minimum number of replicas due to their criticality. But, at the same time, you don’t want too many replicas that eat up all the available cluster resources, preventing other applications from running. In this case, we only need to modify the namespace annotations to cover both constraints. For example, our production namespace may be defined as follows:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    replicas-min: "4"
    replicas-max: "10"
  name: production

Additionally, we need to change the policy message to reflect the new violation. So, our policy file should now look like this:

package kubernetes.admission
# Acquire the existing namespaces defined in the cluster
import data.kubernetes.namespaces
# Get the number of requested replicas from the request
replicas = input.request.object.spec.replicas
# Get the namespace name from the request
namespace := input.request.object.metadata.namespace
# Get the minimum limit (if any) from the deployment's namespace annotation
minimum = to_number(namespaces[namespace].metadata.annotations["replicas-min"])
# Get the maximum limit (if any) from the deployment's namespace annotation
maximum = to_number(namespaces[namespace].metadata.annotations["replicas-max"])
# Get the deployment name from the request
deployment_name = input.request.object.metadata.name
deny[msg]{
	# We are interested in Deployment requests only
	input.request.kind.kind == "Deployment"
    # Evaluate the requested replica count against the one defined in the namespace annotation
    replicas < minimum
	msg := sprintf("The Deployment %v could not be created because it requests %v replicas which violates the minimum replicas count of %v or the maximum of ",[deployment_name,replicas,minimum,maximum])
}
deny[msg]{
	# We are interested in Deployment requests only
	input.request.kind.kind == "Deployment"
    # Evaluate the requested replica count against the one defined in the namespace annotation
    replicas > maximum
	msg := sprintf("The Deployment %v could not be created because it requests %v replicas which violates the minimum replicas count of %v or the maximum of ",[deployment_name,replicas,minimum,maximum])
}

TL;DR

• ReplicaSets and, subsequently, Deployments are there to ensure your application’s high availability and resilience.
• As a best practice, you should always define at least two replicas for all your applications (of course there may be exceptions).
• Some critical applications may require more than two replicas at minimum to make sure that they can handle high traffic volumes.
• OPA can help you enforce a minimum replica count at the application level, or at the namespace level.
• Optionally, you can also set a maximum replica count.
• Using Rego’s flexibility, you can enforce that the Deployment must have a minimum replica count, a maximum replica count, or a value that lies between both.