Enforce Kubernetes Container Images To Have Label That is Not “Latest”

Welcome to a new article in our OPA series. This article is part of a series. To follow along, you should at least know what Open Policy Agent (OPA) is and how you can use Rego to describe policies. If all this seems foreign to you, we’ve got you covered. In the first few articles of this series, we cover what OPA is, how Rego language is used, and also how to integrate the agent with Kubernetes using the OPA Gatekeeper project as well as through kube-mgmt sidecar containers. So, shall we get started?

The Problem With Using Naked Container Images Or The :latest tag

Let’s examine the following very common scenario: you have an application named xyz-app. The developers are constantly updating it, adding features, solving bugs, etc. Every two weeks or so, your CI/CD pipeline pushes the latest image to your Docker registry. By default, Docker registries tag the most recent version of an image with “latest”. If you pull the image without specifying a specific tag (naked image), you will automatically get the latest one. While this workflow sounds fine at first, it’s very risky. How do you know (and ensure) that your Kubernetes cluster is getting the newer image version? This simple diagram demonstrates what happens:

As per the diagram, the workflow is as follows:

1. After passing the tests, the latest code change is pushed to your CI/CD tool of choice.
2. The pipeline pushes the image to the container registry without specifying the tag, or explicitly appending “:latest” to the image name.
3. Finally, the pipeline contacts the cluster (perhaps through kubectl, helm, etc.) to make Kubernetes pull and use the new image. But how does Kubernetes know that you need to replace the image if the one it is already running is tagged latest, and the one you’re asking it to run it also :latest tagged?

Of course, one possible solution is to set imagePullPolicy to Always to force Kubernetes not to use the cached image and contact the container registry. But, then you need to kill the container so that the Deployment recreates it, and consequently pulls the latest image. Do we need to go that far just to deploy our newest code changes?

From the above undesirable scenario, it’s always strongly recommended that you use the appropriate image tag when building and pushing the container image. CI/CD tools make this very easy as they already provide unique values like the Git commit ID through environment variables. Following this best practice, we can re-depict our workflow as follows:

Now, it’s very straightforward to apply the actual latest image to your cluster by just patching the deployment with the changed image tag. The Deployment automatically pulls the newer image and, through rolling-updates, introduces the new application version with zero downtime.

However, this best practice is only valid if it’s applied. So, how do we ensure (and enforce) that no user or service will create a Pod (no matter which parent controller is used) that uses a naked or latest-tagged image? This is where OPA shines.

Describing The Policy In Rego Language

It’s highly recommended that you state your policy in plain English before starting to write code. In our case: “Any Pod that uses a container image with no tags at all or with the latest tag should not be created.”

Our enforce-image-tag.rego file may look as follows:

package kubernetes.admission
requested_images = {img | img := input.request.object.spec.containers[_].image}
deny[msg] {
	# We are intersted in Pod requests only
	input.request.kind.kind == "Pod"
	# Combine the results of both "ensure" policies with a logical OR
	ensure
	# If the evaulation result is true, deny the request and send this message to the requestor
	msg := sprintf("Pod %v could not be created because it uses images that are tagged latest or images with no tags",[input.request.object.metadata.name])
}
ensure {
	# Does the image tag is latest? this should violate the policy
	has_string(":latest",requested_images)
}
ensure {
	# OR Is this a naked image? this should also violate the policy
	not has_string(":",requested_images)

}
has_string(str,arr){
	contains(arr[_],str)
}

We added comments whenever we could so that the code is self-explanatory. However, there are some points that require discussion:

• Line 2 creates a global variable where we extract the image names from all the containers in the Pod definition. We call this variable several times in the script.
• We use the underscore character _ as an iterator.
• Lines 11 and 15 define the ensure policy. Rego allows you to define duplicate policy names. When executed, the Boolean results are combined with a logical OR. This translates to: if the image name contained “:latest” or if it didn’t have any tags at all, the policy should be violated. Notice that if we added both conditions in one policy, they will be combined with a logical AND operator, which is not desirable in our scenario.
• Line 20 defines the helper function has_string. Due to how Rego deals with negation, you almost always use helper functions when you need to work with arrays.

"Already working in production with Kubernetes? Want to know more about kubernetes application patterns?
👇👇

Applying The OPA Policy

In these labs, we’re using the kube-mgmt sidecar container to deploy OPA policies. To do so, you only need to create a ConfigMap object that contains the policy file. Notice that there is another way of deploying OPA policies if you’ve done the OPA integration with Kubernetes through the OPA Gatekeeper project. This method creates custom objects that contain the policy and the parameters. However, this is beyond the scope of this article. To create a new ConfigMap, make sure you are in the opa namespace and run the following command:

kubectl create configmap  enforce-image-tag --from-file=enforce-image-tag.rego

It’s always best practice to ensure that you don’t have any syntax errors in your code. While you’re highly encouraged to use the Rego Playground tool to test and debug your code before deployment, it’s also a good idea to ensure that the OPA engine has accepted the policy syntax without issues. This can be done by examining the status of the ConfigMap:

kubectl get cm enforce-image-tag -o json | jq '.metadata.annotations'
{
  "openpolicyagent.org/policy-status": "{\"status\":\"ok\"}"
}

We’re using the JSON output option of the kubectl command so that we can pipe the result to a tool like jq to extract the relevant parts. In our case, we need to see the status.

Exercising The Policy

Once the ConfigMap is created, it’s automatically picked up by OPA. You don’t need to trigger it or restart the container. Now let’s create a Pod that does not use a tag in its images and see what happens (when creating pods, make sure you don’t place them in the opa or the kube-system namespace as both of them are exempted from opa validations!):

$ kubectl apply -f - <<EOT
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
spec:
  containers:
  - name: nginx
    image: nginx
EOT
Error from server (Pod nginx could not be created because it uses images that are tagged latest or images with no tags): error when creating "STDIN": admission webhook "validating-webhook.openpolicyagent.org" denied the request: Pod nginx could not be created because it uses images that are tagged latest or images with no tags

Upon attempting to create a Pod that has an nginx container, OPA denied our request since we intentionally ignored adding tags to the image. Let’s see what happens if we do add a tag, but it’s “latest”:

$ kubectl apply -f - <<EOT
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
spec:
  containers:
  - name: nginx
    image: nginx:latest
EOT
Error from server (Pod nginx could not be created because it uses images that are tagged latest or images with no tags): error when creating "STDIN": admission webhook "validating-webhook.openpolicyagent.org" denied the request: Pod nginx could not be created because it uses images that are tagged latest or images with no tags

Nice! We see that no users will be allowed to create any pods unless they define a specific image tag. Let’s make sure that Pods with the legitimate image tags are allowed:

$ kubectl apply -f - <<EOT
apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
spec:
  containers:
  - name: nginx
    image: nginx:1.17.10
EOT
pod/nginx created

Notice that this applies not only to Pod definitions, but also to any controller that creates Pods (Deployments, DaemonSets, StatefulSets, etc.). However, when a controller that creates Pods violates OPA policy, you will not see the error message on the prompt as we did. That’s because the controller creation process did not violate the policy - a child process that creates the Pods did. In such a case, you need to query the status of the controller to find out why Pods were denied. For example, kubectl -n default describe mydeployment -o json

TL;DR

• A best practice in using Kubernetes is to always use image tags when building and pushing your images.
• Be mindful that keeping the image tag defaulted to “latest” causes ambiguity and leads to the cluster not pulling the new image, thinking that it already has the latest version.
• OPA can help you enforce a policy that will deny Pod creation if any of the images in the Pod uses the “:latest” tag, or does not specify a tag at all.