Weaveworks 2022.03 release featuring Magalix PaC | Learn more
Weaveworks 2022.03 release featuring Magalix PaC | Learn more
mins read
This article is part of our Open Policy Agent (OPA) series, and assumes that you are familiar with Kubernetes and OPA. If you haven’t already done so, or if you need a refresher, please have a look at the previous articles published in this series.
Today we are going to use OPA to validate our Kubernetes Network Policies. In a nutshell, a network policy in Kubernetes enables you to enforce restrictions on pod intercommunication. For example, you can require that for a pod to be able to connect to the database pods, it must have the app=web label. Such practices help decrease the attack vector in your cluster. However, a policy is only as good as its implementation. If you have a well-crafted network that lives in its YAML file and was not applied to the cluster, then it’s useless. Similarly, if important aspects were missed when creating the policy, then this poses a risk as well. OPA can help you alleviate those risks. This article provides two hands-on labs explaining the process.
In this situation, your application pods contain proprietary code that needs increased protection. As part of your security plan, you need to ensure that no pods are allowed to access your application, except the frontend ones.
You create a network policy that enforces this restriction which may look like this:
backapiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: app-inbound-policy
namespace: default
spec:
podSelector:
matchLabels:
app: prop
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 80
Let’s set up a quick lab to ensure that our policy is indeed in place. We create a deployment that creates our protected pods. For simplicity, we’ll assume that nginx is the image used by our protected app. The deployment file may look as follows:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: prop-deployment
namespace: default
spec:
selector:
matchLabels:
app: prop
replicas: 2
template:
metadata:
labels:
app: prop
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: prop-svc
namespace: default
spec:
selector:
app: prop
ports:
- protocol: TCP
port: 8080
targetPort: 80
Apply the above definition and ensure that you have two pods running. Now let’s try to connect to the protected pods using a permitted pod. The following definition creates a pod with the allowed label that uses the alpine image:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: client-deployment
namespace: default
spec:
selector:
matchLabels:
app: frontend
replicas: 1
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: alpine
image: alpine
command:
- sh
- -c
- sleep 100000
To prove that the pod created by this deployment can access our protected pods, let’s open a shell session to the container and establish an HTTP connection to the pod:
$ kubectl exec -it client-deployment-7666b46645-27psl -- sh
/ # apk add curl
/ # curl prop-svc:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
So, we were able to get HTML output, which means that the connection was successful. Now, let’s create another deployment that uses different labels for the client (you can equally change the labels of the existing pods). The deployment file, for the client pod that should not be allowed access to our protected pods, should like this:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: illegal-client-deployment
namespace: default
spec:
selector:
matchLabels:
app: backend
replicas: 1
template:
metadata:
labels:
app: backend
spec:
containers:
- name: alpine
image: alpine
command:
- sh
- -c
- sleep 100000
Opening a shell session to the container and trying to connect to our target pods:
kubectl exec -it illegal-client-deployment-55b694c9df-2rznp -- sh
/ # apk add curl
/ # curl --connect-timeout 10 prop-svc:8080
curl: (28) Connection timed out after 10001 milliseconds
We used the curl’s --connect-timeout command-line option to show that the connection was not established even after ten seconds have passed. The network policy is doing what it is supposed to.
So far, we’ve used the network security policy to protect our pods by limiting the connection sources. Now, what if this policy is deleted by mistake, or another replica of this cluster (perhaps in another region?) is deployed, but the deployment procedure missed creating the network policy? To avoid those risks, we can deploy an OPA policy that denies creating the app=prop-labelled pods if this network security policy was not in place. As usual, we start by writing the policy in Rego. It should look as follows:
package kubernetes.admission
import data.kubernetes.networkpolicies
# Deny with a message
deny[msg]{
input.request.kind.kind == "Pod"
pod_label_value := {v["app"] | v := input.request.object.metadata.labels} # true
contains_label(pod_label_value,"prop")
np_label_value := {v["app"] | v := networkpolicies[_].spec.podSelector.matchLabels}
not contains_label(np_label_value,"prop")
msg:= sprintf("The Pod: %v could not be created because it is missing an associated Network Security Policy.",[input.request.object.metadata.name])
}
contains_label(arr,val){
arr[_] == val
}
"object": {
"kind": "Pod",
"apiVersion": "v1",
"metadata": {
"uid": "17e42367-d738-410c-af6b-f02d9a8766eb",
"creationTimestamp": "2020-05-08T21:33:57Z",
"labels": {
"app": "prop",
"pod-template-hash": "9c9cf7d47"
},
We are interested in pods that host our proprietary application. They are labeled app=prop. So, we use a Rego syntax that is very similar to Python’s list comprehension. It can be read as follows:
Before we go ahead and deploy our OPA policy we need to instruct our kube-mgmt sidecar container to obtain the list of Network Policy objects defined in the cluster so that we can import it in the policy. This task is as simple as modifying the OPA deployment (please refer to our previous article on how to integrate Kubernetes with OPA using kube-mgmt) so that the kubemgmt container definition looks as follows:
- name: kube-mgmt
image: openpolicyagent/kube-mgmt:0.8
args:
- "--replicate-cluster=networking.k8s.io/v1/networkpolicies"
The syntax of the object you want to replicate is simple:
The API type + / + the object name in a small case and in the plural. For more information, you may want to refer to kube-mgmt documentation. Wait until the pods are terminated and recreated after you modify the deployment, before proceeding.
Deploying OPA policies in Kubernetes is as easy as creating a ConfigMap in the opa namespace. Let’s do that:
kubectl create configmap ensure-nap-existence --from-file=ensure-nap-exists.rego
It’s always a good practice to check that your policy was acquired by OPA with no syntax errors. You can do this by checking the status of the ConfigMap. I prefer having the output in JSON and using jq to parse it. But you can use whatever method you like:
kubectl get cm ensure-nap-existence -o json | jq '.metadata.annotations'
{
"openpolicyagent.org/policy-status": "{\"status\":\"ok\"}"
The status is OK. Now, let’s exercise our policy.
To test policy execution, we delete the Network Policy that we already defined and the Deployment. Then we recreate the Deployment without recreating the Network Policy:
$ kubectl delete deployments prop-deployment -n default
$ kubectl delete networkpolicies app-inbound-policy -n default
$ kubectl apply -f prop-deployment.yaml
deployment.apps/prop-deployment created
service/prop-svc unchanged
$ kubectl get pod -n default
NAME READY STATUS RESTARTS AGE
client-deployment-7666b46645-27psl 1/1 Running 0 18h
illegal-client-deployment-55b694c9df-2rznp 1/1 Running 0 18h
As you can see from the output, we do not have any prop pods. Let’s probe the Deployment status messages to learn why the pods were not created:
$ kubectl get deployments prop-deployment -n default -o json | jq '.status.conditions[].message'
"Created new replica set \"prop-deployment-9c9cf7d47\""
"Deployment does not have minimum availability."
"admission webhook \"validating-webhook.openpolicyagent.org\" denied the request: The Pod: prop-deployment-9c9cf7d47-qd74j could not be created because it is missing an associated Network Security Policy."
So, the last message states the reason: we do not have a Network Policy that protects those pods and they will not be deployed unless one is created.
Let’s have another test in which we do create a Network Policy, but it does not watch the pods labeled app=prop. Our definition file for this object may look as follows:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: app-inbound-policy2
namespace: default
spec:
podSelector:
matchLabels:
app: backend
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 80
This policy is concerned with restricting access to app=backend pods to be only coming from the app=frontend labels. Clearly, this policy has nothing to do with our app=prop pods. Let’s see what happens when we apply this policy, and delete and recreate our Deployment.
$ kubectl apply -f nap2.yaml
networkpolicy.networking.k8s.io/app-inbound-policy2 created
$ kubectl delete deployments prop-deployment -n default
deployment.apps "prop-deployment" deleted
$ kubectl apply -f prop-deployment.yaml
deployment.apps/prop-deployment created
service/prop-svc unchanged
$ kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
client-deployment-7666b46645-27psl 1/1 Running 0 18h
illegal-client-deployment-55b694c9df-2rznp 1/1 Running 0 18h
$ kubectl get deployments prop-deployment -n default -o json | jq '.status.conditions[].message'
"Created new replica set \"prop-deployment-9c9cf7d47\""
"Deployment does not have minimum availability."
"admission webhook \"validating-webhook.openpolicyagent.org\" denied the request: The Pod: prop-deployment-9c9cf7d47-jkv6v could not be created because it is missing an associated Network Security Policy."
We get the same result. Finally, let’s create the Network Policy that controls app=prop pods (defined earlier in the article), delete and recreate the Deployment, and observe what happens:
$ kubectl apply -f network-policy.yaml
networkpolicy.networking.k8s.io/app-inbound-policy created
$ kubectl delete deployments prop-deployment -n default
deployment.apps "prop-deployment" deleted
$ kubectl apply -f prop-deployment.yaml
deployment.apps/prop-deployment created
service/prop-svc unchanged
$ kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
client-deployment-7666b46645-27psl 1/1 Running 0 18h
illegal-client-deployment-55b694c9df-2rznp 1/1 Running 0 18h
prop-deployment-9c9cf7d47-2dt96 1/1 Running 0 4s
prop-deployment-9c9cf7d47-lvlmr 1/1 Running 0 4s
The deployment was created successfully. We have two prop pods running. OPA allowed this action because there is a Network Policy in place that the pods in question.
👇👇
In the previous example, we addressed the risk of having pods hosting a security-sensitive application getting deployed with no Network Policy. However, if you take a close look at the OPA policy we demonstrated, you’ll see that it only ensures the existence of a Network Policy that scopes app=prop-labeled pods. But take a look at the following Network Access policy:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: risky-policy
namespace: default
spec:
podSelector:
matchLabels:
app: prop
ingress:
- from:
- podSelector:
matchLabels:
tier: test
ports:
- protocol: TCP
port: 80
Now, although this object would satisfy our OPA policy restriction, since it matches app: prop labels, it does not fulfill our purpose - to allow connections to the proprietary-app pods only from pods labeled app=frontend. Instead, it allows connections from pods labeled tier=test. To address this risk, we create a second OPA policy that ensures that the Network Policy object serves our ultimate requirement.
Our policy file should look as follows:
package kubernetes.admission
# Deny with a message
deny[msg]{
ensure
count(input.request.object.spec.ingress)
msg:= sprintf("The Network Policy: %v could not be created because it violates the proprietary app security policy.",[input.request.object.metadata.name])
}
ensure {
# When the requested object is a NetworkPolicy
input.request.kind.kind == "NetworkPolicy"
# and it is controlling our protected pods (those labelled app=prop)
input.request.object.spec.podSelector.matchLabels["app"] == "prop"
# get the pod label values when they key is "app" from the list of labels that this policy controlls ingress connections to
values := {v["app"] | v:= input.request.object.spec.ingress[_].from[_].podSelector.matchLabels}
# if we do not have "app=frontend" as the allowed value, this policy is violated
not exists(values,"frontend") # false because we have frontend
}
# A hepler function to test the eixstence of the label value
exists(arr,elem) {
arr[_] == elem
}
ensure {
# Ensure that we only have one policy in the ingress
1 != count(input.request.object.spec.ingress) # false because we have just one ingress
}
ensure {
# Ensure that we only have one policy in the ingress
1 != count([from | from := input.request.object.spec.ingress[_].from]) # should be false because we have more than one from
}
We added comments where we could to make the code self explanatory. However, there are still some points that need clarification:
Line 9 calls another policy (not a function) called ensure. The reason we are using another policy to write our conditions and not have everything in the deny policy is that we need to combine multiple conditions with the logical OR instead of AND. To explain this better, consider the following restrictions that we need all of them applied:
As you can see, we need the policy if any of the above are True, and not if all of them are True. To achieve that in Rego, we create multiple policies with the same name and make the call in the main policy (deny).
Again, deploying the OPA policy is as simple as creating the ConfigMap in the opa namespace:
$ kubectl create configmap enforce-correct-nap --from-file=enforce-correct-nap.rego
configmap/enforce-correct-nap created
We won’t add the policy validation step for brevity, but it’s recommended that you always validate your policies as described earlier.
To confirm that our OPA policy is effective, let’s try to deploy the risky Network Policy that we demonstrated earlier:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: risky-policy
namespace: default
spec:
podSelector:
matchLabels:
app: prop
ingress:
- from:
- podSelector:
matchLabels:
tier: test
ports:
- protocol: TCP
port: 80
kubectl apply -f nap2.yaml
Error from server (The Network Policy: risky-policy could not be created because it violates the proprietary app security policy.): error when creating "nap2.yaml": admission webhook "validating-webhook.openpolicyagent.org" denied the request: The Network Policy: risky-policy could not be created because it violates the proprietary app security policy.
So, we are not allowed to create a Network Policy that scopes our prop pods and accepts incoming connection from anywhere except app=frontend.
Finally, delete and recreate our original Network Policy to ensure that we are allowed to deploy it. For a refresher, it looked as follows:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: app-inbound-policy
namespace: default
spec:
podSelector:
matchLabels:
app: prop
ingress:
- from:
- podSelector:
matchLabels:
app: frontend
ports:
- protocol: TCP
port: 80
$ kubectl apply -f network-policy.yaml
networkpolicy.networking.k8s.io/app-inbound-policy created
With the two policies in place, we cannot create pods with app=prop label unless we have the Network Policy that protects them also in place. Additionally, we cannot create that Network Policy unless it contains the required protection rules.
Empower developers to delivery secure and compliant software with trusted application delivery and policy as code. Learn more.
3 mins read
Automate your deployments with continuous application delivery and GitOps. Read this blog to learn more.
3 mins read
This article explains the differences between hybrid and multi-cloud model and how GitOps is an effective way of managing these approaches. Learn more.
4 mins read
Implement the proper governance and operational excellence in your Kubernetes clusters.
“We are on a mission to add intelligence to applications and infrastructure so you can focus on what truly requires human intelligence.”
Comments and Responses