How To Force Kubernetes Namespaces To Have ResourceQuotas Using OPA

DevOps, Kubernetes, K8s, opa, open policy agent, Resource Quotas
How To Force Kubernetes Namespaces To Have ResourceQuotas Using OPA
DevOps, Kubernetes, K8s, opa, open policy agent, Resource Quotas

The Problem With Not Setting Container Requests And Limits

In Kubernetes clusters, effective resource utilization should not be taken lightly. Even in the smallest environments, you need to ensure that:

  1. You are consuming the maximum capacity of the nodes (no underutilization).
  2. No Pods are halted (pending) waiting for an available node (no overutilization).

One way of achieving this is by ensuring that all of your Pods (or at least most of them) define CPU and memory resource limits. Simply put, resource limits define the minimum and maximum amount of resources pods may need. This way, the scheduler knows beforehand which node Pods should be scheduled on, and what behavior is expected. Now that you know the recommendation, you need to enforce it, as the policy is only as good as its application.

The ResourceQuota Approach To Address The Problem

For this reason, Kubernetes offers Resource Quotas. A Resource Quota does two things:

  1. Defines a maximum total CPU and memory for a given namespace.
  2. Denies Pod creation unless the Pod definition supplies resource requests and limits that do not cause the global limit to be exceeded.

A quick example: assume that you set a ResourceQuota on the QA namespace to have the total CPU requests (minimum) to 1 and the limits (maximum) to 2. Now there are two Pod creation requests. Both them define requests and limits to be 400m and 800m respectively. As you might guess, the first Pod will be allowed, while the second Pod will be denied because it will violate the total CPU limit imposed by the ResourceQuota.

The definition files for the ResourceQuota and the Pods may look as follows:

quota-mem-cpu.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-demo
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
quota-mem-cpu-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: quota-mem-cpu-demo
spec:
  containers:
  - name: quota-mem-cpu-demo-ctr
    image: nginx
    resources:
      limits:
        memory: "800Mi"
        cpu: "800m"
      requests:
        memory: "600Mi"
        cpu: "400m"

Enforce ResourceQuota Existence Using OPA Policies

How to force Kubernetes namespaces to have ResourceQuotas defined using OPA 1-1

The ResourceQuota is one of Kubernetes’s admission controllers. It works by intercepting the HTTP request to the API server, and validating it against its policy definition. However, what if the ResourceQuota was never created in the first place? What if it was accidentally deleted? It simply means that there will be nothing to intercept Pod creation requests, and cluster resources will be inefficiently distributed. Here’s where the Open Policy Agent can help.

This article is part of our OPA series. We assume that you have read at least our OPA introduction and deployment to Kubernetes, and that you have a running Kubernetes cluster with OPA installed.

Expressing The Policy In A Human-readable Manner

The first step in creating the policy is to express it in plain English:

"We want to deny the creation of any Pod (even if it has resource requests and limits defined), unless the namespace where it belongs has a ResourceQuota object associated."

Applying this policy effectively ensures that namespaces have ResourceQuotas defined for them. Otherwise, no Pods will be allowed to exist.

Describing The Policy In Rego

The next step is to convert what we’ve just said to the Rego language. OPA follows the Policy as Code approach in which you use programming concepts and techniques to define and apply policies. Our ensure-rq-exists.rego file should look as follows:

package kubernetes.admission

import data.kubernetes.resourcequotas
deny[msg]{
    # We are only interested in requests that creates Pods
    input.request.kind.kind == "Pod"
    # Extract the the namespace from the request information
    requestns := input.request.object.metadata.namespace
    # Is it part of the existing resource quotas?
    existingrqs := {e | e := resourcequotas[_][_].metadata.namespace}
    not ns_exists(requestns,existingrqs)
    msg = sprintf("The Pod %v could not be created because the %v namespace does not have ResourceQuotas defined",[input.request.object.metadata.name,input.request.object.metadata.namespace])
}
ns_exists(ns,arr){
	arr[_] = ns
}

If you haven’t seen Rego code before, this code may look intimidating at first. But when you take a closer look, you will see that Rego makes several shortcuts that allow you to write less code to describe your policy than you would in another language. Let’s take a brief look at the above snippet:

  • As usual, our Rego code must start with defining the kubernetes.admission package.
  • Line 3 acquires all the ResourceQuota objects that are already defined in the cluster. We grant OPA access to this data when we modify the OPA deployment later in this lab.
  • Line 4 starts the policy. The head of the policy contains the msg variable that will hold our feedback to the user when the policy is violated.
  • Line 6 specifies the first condition that invokes this policy - we are interested in Pod requests.
  • Line 8 creates a variable that holds the ResourceQuotas that we already obtained in line 3.
  • Line 10 is where the actual verification takes place. Notice the use of the underscore _ character, that Rego uses to iterate the array. Other languages must create a loop with an iterator to extract relevant data from the array. But Rego uses a more elegant method - if you are not interested in the index of the array items, you just place an underscore. Also, notice the use of list-comprehension syntax in which we extract the namespace part of the ResourceQuota and assign it to a temporary variable, e. When the loop is over, existingrqs will be an array of namespaces.
  • Line 11 uses a helper function defined in lines 14 - 16. The function accepts an element and an array and returns true or false depending on whether this element was found in the array. Notice that you must use helper functions whenever you want to check for item existence in an array due to the way negation works in Rego.
  • Line 12 finally constructs the error message that the Pod creator shall see when he tries to pass the request to the API server.

Applying The Policy To The Cluster

In order to apply this policy to the cluster, we first need to make a small change to the OPA deployment. We are going to add an argument to the kube-mgmt sidecar container that gives OPA access to our ResourceQuota objects. The kube-mgmt sidecar container part of the deployment definition should look as follows:

- name: kube-mgmt
          image: openpolicyagent/kube-mgmt:0.8
          args:
            - "--replicate=v1/resourcequotas"

Notice that you can always deduce the value that you need to supply to the --replicate flag:

The API version of the resource / the resource name in the small case and in the plural.

Once the change is applied to the deployment and the Pods get recreated, we need to create a ConfigMap object in the opa namespace from our Rego file:

kubectl create configmap ensure-rq-exists --from-file=ensure-rq-exists.rego

It’s always a good practice to ensure that your policy didn’t have any issues after it was acquired. This can be done by examining the status of the ConfigMap:

kubectl get cm ensure-rq-exists -o json | jq '.metadata.annotations'
{
  "openpolicyagent.org/policy-status": "{\"status\":\"ok\"}"
}

We used jq to filter the JSON output but you can use just grep or any other tool. The most important thing is to ensure that the status is correct.


Already working in production with Kubernetes? Want to know more about kubernetes application patterns?

👇👇

Download Kubernetes Application Patterns E-Book


Exercising The Policy

To ensure that our policy is working as expected, we create a ResourceQuota for the default namespace. Then, we create another namespace (let’s call it freens) and try to create a Pod in that namespace. Because freens does not have a ResourceQuota object defined for it, the Pod creation will be denied. If we create the same Pod but in the default namespace, it should work.

$ kubectl apply -n default -f - <<EOT
apiVersion: v1
kind: ResourceQuota
metadata:
  name: mem-cpu-demo
spec:
  hard:
    requests.cpu: "1"
    requests.memory: 1Gi
    limits.cpu: "2"
    limits.memory: 2Gi
EOT
resourcequota/mem-cpu-demo created

$ kubectl apply -n default -f - <<EOT                                                                          ⎈ minikube/opa
heredoc> apiVersion: v1
kind: Pod
metadata:
  name: quota-mem-cpu-demo
spec:
  containers:
  - name: quota-mem-cpu-demo-ctr
    image: nginx
    resources:
      limits:
        memory: "800Mi"
        cpu: "800m"
      requests:
        memory: "600Mi"
        cpu: "400m"
heredoc> EOT
pod/quota-mem-cpu-demo created

$ kubectl create ns freens
namespace/freens created

$ kubectl apply -n freens -f - <<EOT
apiVersion: v1
kind: Pod
metadata:
  name: quota-mem-cpu-demo
spec:
  containers:
  - name: quota-mem-cpu-demo-ctr
    image: nginx
    resources:
      limits:
        memory: "800Mi"
        cpu: "800m"
      requests:
        memory: "600Mi"
        cpu: "400m"
EOT
Error from server (The Pod quota-mem-cpu-demo could not be created because the freens namespace does not have ResourceQuotas defined): error when creating "STDIN": admission webhook "validating-webhook.openpolicyagent.org" denied the request: The Pod quota-mem-cpu-demo could not be created because the freens namespace does not have ResourceQuotas defined

TL;DR

  • In this article we demonstrated how we can use OPA to enforce the presence of a ResourceQuota object in the namespace where a Pod is requesting to be created.
  • For this policy to work, it had to obtain a list of the existing ResourceQuotas in the cluster. This can be done by first modifying the kube-mgmt sidecar container to replicate the ResourceQuota and second by importing the data supplied through the container.
  • Rego is the language used to describe OPA policies. It makes it easy to work with JSON objects, and borrows some programming concepts from other languages, like list-comprehension from Python.

Comments and Responses

Related Articles

DevOps, Kubernetes, cost saving, K8s
Kubernetes Cost Optimization 101

Over the past two years at Magalix, we have focused on building our system, introducing new features, and

Read more
The Importance of Using Labels in Your Kubernetes Specs: A Guide

Even a small Kubernetes cluster may have hundreds of Containers, Pods, Services and many other Kubernetes API

Read more
How to Deploy a React App to a Kubernetes Cluster

Kubernetes is a gold standard in the industry for deploying containerized applications in the cloud. Many

Read more

start your 14-day free trial today!

Automate your Kubernetes cluster optimization in minutes.

Get started View Pricing
No Card Required