How To Integrate OPA Into Your Kubernetes Cluster Using Kube-mgmt

Earlier in this series, we explained what OPA is, then we demonstrated how easy it is to integrate OPA with your Kubernetes cluster through the OPA Gatekeeper project. In this article, we explore another means of OPA-Kubernetes integration, but this time without using OPA Gateway. Despite being lengthy, this procedure will give you more control over the process and will also teach you the inner workings of how the integration is done. In this article, we’ll cover how to deploy OPA from scratch, and apply a sample policy that enforces using an Ingress hostname from a whitelist. For this lab, we’re using Minikube.

Part 1: Install OPA

Step 1: Ensure That You Have The Prerequisites

1. Kubernetes version 1.13 or higher.
2. The ValidatingAdmissionWebhook admission controller must be enabled. You can enable it, and other recommended admission controllers, when the API starts by following this guide.
3. Since we’re using minikube, we’ll need to ensure that the ingress addon is enabled: minikube addons enable ingress
4. OPA expects to load policies from ConfigMaps in the opa namespace. Let’s create this namespace now: kubectl create namespace opa
5. Change the context to the opa namespace kubectl config set-context

Step 2: Create The Necessary TLS Certificate

To secure the communication between the API server and OPA, we’ll need to configure TLS:

1. Create a certificate authority and key:

   • openssl genrsa -out ca.key 2048
     

   • openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"
     

2. Generate the key and certificate for OPA:

   cat >server.conf <<EOF
   [req]
   req_extensions = v3_req
   distinguished_name = req_distinguished_name
   [req_distinguished_name]
   [ v3_req ]
   basicConstraints = CA:FALSE
   keyUsage = nonRepudiation, digitalSignature, keyEncipherment
   extendedKeyUsage = clientAuth, serverAuth
   EOF

   $ openssl genrsa -out server.key 2048
   $ openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf
   $ openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf
   

3. Create a Kubernetes TLS Secret to store our OPA credentials:

   kubectl create secret tls opa-server --cert=server.crt --key=server.key
   

Step 3: Deploy The Admission Controller

The adminssion-controller-yaml file should look as follows:

# Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt
# replicate resources into OPA so they can be used in policies.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: opa-viewer
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts:opa
  apiGroup: rbac.authorization.k8s.io
---
# Define role for OPA/kube-mgmt to update configmaps with policy status.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: configmap-modifier
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["update", "patch"]
---
# Grant OPA/kube-mgmt role defined above.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: opa
  name: opa-configmap-modifier
roleRef:
  kind: Role
  name: configmap-modifier
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
  name: system:serviceaccounts:opa
  apiGroup: rbac.authorization.k8s.io
---
kind: Service
apiVersion: v1
metadata:
  name: opa
  namespace: opa
spec:
  selector:
    app: opa
  ports:
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  namespace: opa
  name: opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
      name: opa
    spec:
      containers:
        # WARNING: OPA is NOT running with an authorization policy configured. This
        # means that clients can read and write policies in OPA. If you are
        # deploying OPA in an insecure environment, be sure to configure
        # authentication and authorization on the daemon. See the Security page for
        # details: https://www.openpolicyagent.org/docs/security.html.
        - name: opa
          image: openpolicyagent/opa:latest
          args:
            - "run"
            - "--server"
            - "--tls-cert-file=/certs/tls.crt"
            - "--tls-private-key-file=/certs/tls.key"
            - "--addr=0.0.0.0:443"
            - "--addr=http://127.0.0.1:8181"
            - "--log-format=json-pretty"
            - "--set=decision_logs.console=true"
          volumeMounts:
            - readOnly: true
              mountPath: /certs
              name: opa-server
          readinessProbe:
            httpGet:
              path: /health?plugins&bundle
              scheme: HTTPS
              port: 443
            initialDelaySeconds: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health
              scheme: HTTPS
              port: 443
            initialDelaySeconds: 3
            periodSeconds: 5
        - name: kube-mgmt
          image: openpolicyagent/kube-mgmt:0.8
          args:
            - "--replicate-cluster=v1/namespaces"
            - "--replicate=extensions/v1beta1/ingresses"
      volumes:
        - name: opa-server
          secret:
            secretName: opa-server
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: opa-default-system-main
  namespace: opa
data:
  main: |
    package system

    import data.kubernetes.admission

    main = {
      "apiVersion": "admission.k8s.io/v1beta1",
      "kind": "AdmissionReview",
      "response": response,
    }

    default response = {"allowed": true}

    response = {
        "allowed": false,
        "status": {
            "reason": reason,
        },
    } {
        reason = concat(", ", admission.deny)
        reason != ""
    }

The file creates the necessary RBAC components, a Deployment, a ConfigMap, and a Service. There are two points that we need to emphasize in this definition file:

1. The Service name (opa) must match the CN (or the common name) that we chose for the certificate. Otherwise, TLS communication will fail.
2. In the Deployment, we have the kube-mgmt sidecar container loaded with the following command-line arguments:
   • --replicate-cluster=v1/namespaces.
   • --replicate=extensions/v1beta1/ingresses.
   Those arguments allow the sidecar container to replicate the namespace, ingress Kubernetes objects, and load them into the OPA engine. This lab is focused on enforcing policies that involve those objects, however, in other use cases, you may want to modify the kube-mgmt command to include other objects.

Now, apply the definition:

kubectl apply -f admission-controller.yaml

Already working in production with Kubernetes? Want to know more about kubernetes application patterns?

👇👇

Step 4: Deploy The Admission Webhook

For the admission controller to work, we need an admission webhook that receives the admission HTTP callbacks and executes them. Now, let’s create our webhook configuration file:

cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1beta1
metadata:
  name: opa-validating-webhook
webhooks:
  - name: validating-webhook.openpolicyagent.org
    namespaceSelector:
      matchExpressions:
      - key: openpolicyagent.org/webhook
        operator: NotIn
        values:
        - ignore
    rules:
      - operations: ["CREATE", "UPDATE"]
        apiGroups: ["*"]
        apiVersions: ["*"]
        resources: ["*"]
    clientConfig:
      caBundle: $(cat ca.crt | base64 | tr -d '\n')
      service:
        namespace: opa
        name: opa
EOF

This webhook configuration has the following properties:

1. It won’t listen to actions coming from any namespace that has the openpolicyagent.org/webhook=ignore label. That’s necessary so that OPA doesn’t intercept requests in the kube-system namespace, or its own namespace (will add labels to them later).
2. It will listen for the CREATE and UPDATE actions on all resources.
3. It uses a base64 representation of the CA certificate, that we created earlier, to be able to communicate with OPA.

Now, before we apply the configuration, let’s label the kube-system and opa namespaces so that they’re not within the webhook scope:

kubectl label ns kube-system openpolicyagent.org/webhook=ignore
kubectl label ns opa openpolicyagent.org/webhook=ignore

And apply the following configuration to register OPA as an admission controller.

kubectl apply -f webhook-configuration.yaml

Part 2: Deploy A Sample Policy

Step 1: Define The Policy In Rego

OPA uses the Rego language to describe policies. For our lab, we’ll use the example mentioned in the official documentation. Our ingress-whitelist.rego should look as follows:

package kubernetes.admission

operations = {"CREATE", "UPDATE"}

deny[msg] {
	input.request.kind.kind == "Ingress"
	operations[input.request.operation]
	host := input.request.object.spec.rules[_].host
	not fqdn_matches_any(host, valid_ingress_hosts)
	msg := sprintf("invalid ingress host %q", [host])
}

valid_ingress_hosts = {host |
	whitelist :=input.request.namespace.metadata.annotations["ingress-whitelist"]
	hosts := split(whitelist, ",")
	host := hosts[_]
}

fqdn_matches_any(str, patterns) {
	fqdn_matches(str, patterns[_])
}

fqdn_matches(str, pattern) {
	pattern_parts := split(pattern, ".")
	pattern_parts[0] == "*"
	str_parts := split(str, ".")
	n_pattern_parts := count(pattern_parts)
	n_str_parts := count(str_parts)
	suffix := trim(pattern, "*.")
	endswith(str, suffix)
}

fqdn_matches(str, pattern) {
    not contains(pattern, "*")
    str == pattern
}

If you’re new to Rego, the code may seem cryptic at first, but Rego makes it really easy to define policies. Let’s spend a few moments highlighting how this policy enforces using Ingress namespace from the whitelist:

• Line 1: The package is used the same way that it’s used in other languages.
• Line 3: We define a data set that contains two items: CREATE and UPDATE
• Line 5: This is the policy itself - it starts with deny followed by the policy body. If the combination of the statements in the body evaluates to true, the policy is violated, the action is blocked, and the message is returned to the user with the reason why the action was blocked.
• Line 6: The input object is special. Any JSON message sent to OPA starts with the input object at the root. We’re traversing the JSON object until we reach the resource in question and it must be “Ingress” for the policy to be applied.
• Line 7: We need to apply the policy to create or update the resource. In Rego, we can do that through the shorthand operations[input.requset.operation] where the code inside the square brackets extracts the operation specified in the request. The statement is true if it matches one of the items defined in the operations set in line number 3.
• Line 8: To extract the host(s) that the Ingress object will have, we iterate through the rules array of the JSON object. Again, Rego provides the _ character to loop through the array and return all the items to the host variable.
• Line 9: Now that we have our host, we need to ensure that it’s not one of the whitelisted hosts. Remember, the policy is violated only if it evaluates to true. In order to check the existence of the host in the valid-hosts list, we use the fqdn_matches_any function that is defined in line 19.
• Line 10: Defines the message that should be returned to the user explaining why the Ingress object could not be created.
• Lines 13-17: This part extracts the whitelisted hostnames from the annotations part of the Ingress namespace. The hostnames are added in a comma-separated list and the split built-in function is used to convert the hosts to a list. Finally, the _ is used to iterate through all the extracted hosts. The result is piped to the host variable through the | character. If you’ve programmed in Python before, this is very similar to list comprehensions.
• Line 19: The function simply accepts a string and searches for it in a list of patterns, which is the second argument. To do that, it uses another function, fqdn_matches defined in line 23 and also in line 33. In Rego, you can define multiple functions with the same signature as long as all of them produce the same output. When you call a function that’s defined more than once, all the occurrences of the function are called. More on this in the official docs.
• Lines 23-31: The first fqdn_matches definition.
  • First, it extracts the hostname from the pattern by splitting the pattern to tokens by the dot (.) so *.example.com becomes *, example, and com.
  • Next, it ensures that the first token of the pattern is an asterisk. The splitting operation is done on the str part (which would be the fqdn name when the function is called).
  • It counts the number of tokens in the pattern and the input string.
  • If extracts the suffix from the pattern by removing the *. Part.
  • Finally, it evaluates whether the input string input ends in the suffix or not. So, if the allowed pattern is *.mydomain.com and the string is www.example.com, the policy is violated since www is not part of mydomain.com.
• Lines 33-36: The second validation function. This function is used to validate patterns that do not use wildcards. For example, when the pattern is written as mycompany.mydomain.com
  • First, we need to ensure that the supplied pattern does not contain a wildcard. Otherwise, the statement will evaluate to false and the function will not continue.
  • It the pattern refers to a specific domain, then we just need to ensure that the fqdn matches this pattern. In other words, if the pattern is mycompany.mydomain.com then the fqdn of the host must also be mycompany.mydomain.com.

The reason we have two functions with the same name and signature is because of a limitation in the Rego language that prevents functions from producing more than one output value. So, to make more than one validation at the same time with different logic, you must use multiple functions with the same name.

Step 2: Apply The Policy

In a real-world case, you should thoroughly test your Rego code before applying it to the cluster. There are several ways to perform quality assurance on Rego code including Unit Testing. You can (and should) also use the Rego Playground to try the code and spot any errors using sample data.

To apply the policy to the cluster, you need to create a ConfigMap with the file contents in the opa namespace:

kubectl create configmap ingress-whitelist --from-file=ingress-whitelist.rego

Step 3: Ensure That The Policy Is Working As Expected

Next, let’s create two namespaces: one for the QA environment and the other for production. Notice that both of them contain the ingress-whitelist annotation, holding a list of the domain patterns that the Ingress hostname should strictly be part of.

1. qa-namespace.yaml:

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    ingress-whitelist: "*.qa.acmecorp.com,*.internal.acmecorp.com"
  name: qa

2. production-namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    ingress-whitelist: "*.acmecorp.com"
  name: production

Apply both files to the cluster:

kubectl apply -f qa-namespace.yaml -f production-namespace.yaml

Next, let’s create an Ingress that uses one of the allowed domains:

kubectl apply -f - <<EOT                                                                                                
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-ok
  namespace: production
spec:
  rules:
  - host: signin.acmecorp.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80
EOT

Ensure that the Ingress was created:

kubectl get ing -n production
NAME         CLASS    HOSTS                 ADDRESS          PORTS   AGE
ingress-ok   <none>   signin.acmecorp.com   192.168.99.101   80      54s

Now, let’s try to create an Ingress that should not be allowed:

$ kubectl apply -f - <<EOT
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-bad
  namespace: qa
spec:
  rules:
  - host: acmecorp.com
    http:
      paths:
      - backend:
          serviceName: nginx
          servicePort: 80
EOT
Error from server (invalid ingress host "acmecorp.com"): error when creating "STDIN": admission webhook "validating-webhook.openpolicyagent.org" denied the request: invalid ingress host "acmecorp.com"

As you can see from the output, the API server refused to create the Ingress object since the hostname we tried to use for the QA Ingress is reserved for production.

Troubleshooting

Your policy is not working as expected, or at all? Most likely you have some kind of error in your Rego code. You can always check the status of the policy by examining the properties of the ConfigMap that contains the policy:

kubectl get cm ingress-whitelist -o yaml

Observe the metadata.annotations.openpolicyagent.org/policy-status. For example, the following snippet shows a syntax error in the Rego code of the ConfigMap:

metadata:
  annotations:
    openpolicyagent.org/policy-status: '{"status":"error","error":{"code":"invalid_parameter","message":"error(s)
      occurred while compiling module(s)","errors":[{"code":"rego_unsafe_var_error","message":"var
      namespaces is unsafe","location":{"file":"opa/ingress-whitelist/ingress-whitelist.rego","row":14,"col":2}},{"code":"rego_unsafe_var_error","message":"var
      _ is unsafe","location":{"file":"opa/ingress-whitelist/ingress-whitelist.rego","row":16,"col":2}}]}}'

TL;DR

• OPA is a general-purpose, platform-agnostic policy enforcement tool.
• Kubernetes is among the many technologies that OPA can integrate with.
• You can integrate OPA into Kubernetes using the OPA Gatekeeper project. This was covered in our previous article.
• You can also deploy OPA by manually creating the admission controller, the admission webhook, and a few other necessary components that we covered in this article.
• Once OPA is deployed, you can use it to enforce different types of policies that differ from what RBAC does. For example, you can deny a request that was authenticated and authorized but contains a property that you disallow.
• In this article, we explored a use case where you can use OPA to deny creating an Ingress object (although the user has the required privileges to create it) when it contains a hostname that you don’t allow.
• OPA is super-powerful - to fully utilize its potential, you need to understand the Rego language. The language reference is an excellent place to start and you should try and exercise different policies on your own.
• The Rego Playground is a very helpful learning and debugging tool for the Rego language.