Weaveworks 2022.03 release featuring Magalix PaC | Learn more
Balance innovation and agility with security and compliance
risks using a 3-step process across all cloud infrastructure.
Step up business agility without compromising
security or compliance
Everything you need to become a Kubernetes expert.
Always for free!
Everything you need to know about Magalix
culture and much more
Welcome to a new article in our OPA series. This article is part of a series. To follow along, you should at least know what Open Policy Agent (OPA) is and how you can use Rego to describe policies. If all this seems foreign to you, we’ve got you covered. In the first few articles of this series, we cover what OPA is, how Rego language is used, and also how to integrate the agent with Kubernetes using the OPA Gatekeeper project as well as through kube-mgmt sidecar containers. So, shall we get started?
Let’s examine the following very common scenario: you have an application named xyz-app. The developers are constantly updating it, adding features, solving bugs, etc. Every two weeks or so, your CI/CD pipeline pushes the latest image to your Docker registry. By default, Docker registries tag the most recent version of an image with “latest”. If you pull the image without specifying a specific tag (naked image), you will automatically get the latest one. While this workflow sounds fine at first, it’s very risky. How do you know (and ensure) that your Kubernetes cluster is getting the newer image version? This simple diagram demonstrates what happens:
As per the diagram, the workflow is as follows:
Of course, one possible solution is to set imagePullPolicy to Always to force Kubernetes not to use the cached image and contact the container registry. But, then you need to kill the container so that the Deployment recreates it, and consequently pulls the latest image. Do we need to go that far just to deploy our newest code changes?
From the above undesirable scenario, it’s always strongly recommended that you use the appropriate image tag when building and pushing the container image. CI/CD tools make this very easy as they already provide unique values like the Git commit ID through environment variables. Following this best practice, we can re-depict our workflow as follows:
Now, it’s very straightforward to apply the actual latest image to your cluster by just patching the deployment with the changed image tag. The Deployment automatically pulls the newer image and, through rolling-updates, introduces the new application version with zero downtime.
However, this best practice is only valid if it’s applied. So, how do we ensure (and enforce) that no user or service will create a Pod (no matter which parent controller is used) that uses a naked or latest-tagged image? This is where OPA shines.
It’s highly recommended that you state your policy in plain English before starting to write code. In our case: “Any Pod that uses a container image with no tags at all or with the latest tag should not be created.”
Our enforce-image-tag.rego file may look as follows:
package kubernetes.admission
requested_images = {img | img := input.request.object.spec.containers[_].image}
deny[msg] {
# We are intersted in Pod requests only
input.request.kind.kind == "Pod"
# Combine the results of both "ensure" policies with a logical OR
ensure
# If the evaulation result is true, deny the request and send this message to the requestor
msg := sprintf("Pod %v could not be created because it uses images that are tagged latest or images with no tags",[input.request.object.metadata.name])
}
ensure {
# Does the image tag is latest? this should violate the policy
has_string(":latest",requested_images)
}
ensure {
# OR Is this a naked image? this should also violate the policy
not has_string(":",requested_images)
}
has_string(str,arr){
contains(arr[_],str)
}
We added comments whenever we could so that the code is self-explanatory. However, there are some points that require discussion:
In these labs, we’re using the kube-mgmt sidecar container to deploy OPA policies. To do so, you only need to create a ConfigMap object that contains the policy file. Notice that there is another way of deploying OPA policies if you’ve done the OPA integration with Kubernetes through the OPA Gatekeeper project. This method creates custom objects that contain the policy and the parameters. However, this is beyond the scope of this article. To create a new ConfigMap, make sure you are in the opa namespace and run the following command:
kubectl create configmap enforce-image-tag --from-file=enforce-image-tag.rego
It’s always best practice to ensure that you don’t have any syntax errors in your code. While you’re highly encouraged to use the Rego Playground tool to test and debug your code before deployment, it’s also a good idea to ensure that the OPA engine has accepted the policy syntax without issues. This can be done by examining the status of the ConfigMap:
kubectl get cm enforce-image-tag -o json | jq '.metadata.annotations'
{
"openpolicyagent.org/policy-status": "{\"status\":\"ok\"}"
}
We’re using the JSON output option of the kubectl command so that we can pipe the result to a tool like jq to extract the relevant parts. In our case, we need to see the status.
Once the ConfigMap is created, it’s automatically picked up by OPA. You don’t need to trigger it or restart the container. Now let’s create a Pod that does not use a tag in its images and see what happens (when creating pods, make sure you don’t place them in the opa or the kube-system namespace as both of them are exempted from opa validations!):
$ kubectl apply -f - <<EOT
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: default
spec:
containers:
- name: nginx
image: nginx
EOT
Error from server (Pod nginx could not be created because it uses images that are tagged latest or images with no tags): error when creating "STDIN": admission webhook "validating-webhook.openpolicyagent.org" denied the request: Pod nginx could not be created because it uses images that are tagged latest or images with no tags
Upon attempting to create a Pod that has an nginx container, OPA denied our request since we intentionally ignored adding tags to the image. Let’s see what happens if we do add a tag, but it’s “latest”:
$ kubectl apply -f - <<EOT
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: default
spec:
containers:
- name: nginx
image: nginx:latest
EOT
Error from server (Pod nginx could not be created because it uses images that are tagged latest or images with no tags): error when creating "STDIN": admission webhook "validating-webhook.openpolicyagent.org" denied the request: Pod nginx could not be created because it uses images that are tagged latest or images with no tags
Nice! We see that no users will be allowed to create any pods unless they define a specific image tag. Let’s make sure that Pods with the legitimate image tags are allowed:
$ kubectl apply -f - <<EOT
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: default
spec:
containers:
- name: nginx
image: nginx:1.17.10
EOT
pod/nginx created
Notice that this applies not only to Pod definitions, but also to any controller that creates Pods (Deployments, DaemonSets, StatefulSets, etc.). However, when a controller that creates Pods violates OPA policy, you will not see the error message on the prompt as we did. That’s because the controller creation process did not violate the policy - a child process that creates the Pods did. In such a case, you need to query the status of the controller to find out why Pods were denied. For example, kubectl -n default describe mydeployment -o json
Self-service developer platform is all about creating a frictionless development process, boosting developer velocity, and increasing developer autonomy. Learn more about self-service platforms and why it’s important.
Explore how you can get started with GitOps using Weave GitOps products: Weave GitOps Core and Weave GitOps Enterprise. Read more.
More and more businesses are adopting GitOps. Learn about the 5 reasons why GitOps is important for businesses.
Implement the proper governance and operational excellence in your Kubernetes clusters.
Comments and Responses