Weaveworks 2022.03 release featuring Magalix PaC | Learn more
Balance innovation and agility with security and compliance
risks using a 3-step process across all cloud infrastructure.
Step up business agility without compromising
security or compliance
Everything you need to become a Kubernetes expert.
Always for free!
Everything you need to know about Magalix
culture and much more
Earlier in this series, we explained what OPA is, then we demonstrated how easy it is to integrate OPA with your Kubernetes cluster through the OPA Gatekeeper project. In this article, we explore another means of OPA-Kubernetes integration, but this time without using OPA Gateway. Despite being lengthy, this procedure will give you more control over the process and will also teach you the inner workings of how the integration is done. In this article, we’ll cover how to deploy OPA from scratch, and apply a sample policy that enforces using an Ingress hostname from a whitelist. For this lab, we’re using Minikube.
To secure the communication between the API server and OPA, we’ll need to configure TLS:
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -days 100000 -out ca.crt -subj "/CN=admission_ca"
cat >server.conf <<EOF
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
EOF
$ openssl genrsa -out server.key 2048
$ openssl req -new -key server.key -out server.csr -subj "/CN=opa.opa.svc" -config server.conf
$ openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 100000 -extensions v3_req -extfile server.conf
Create a Kubernetes TLS Secret to store our OPA credentials:
kubectl create secret tls opa-server --cert=server.crt --key=server.key
The adminssion-controller-yaml file should look as follows:
# Grant OPA/kube-mgmt read-only access to resources. This lets kube-mgmt
# replicate resources into OPA so they can be used in policies.
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: opa-viewer
roleRef:
kind: ClusterRole
name: view
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
name: system:serviceaccounts:opa
apiGroup: rbac.authorization.k8s.io
---
# Define role for OPA/kube-mgmt to update configmaps with policy status.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: opa
name: configmap-modifier
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["update", "patch"]
---
# Grant OPA/kube-mgmt role defined above.
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: opa
name: opa-configmap-modifier
roleRef:
kind: Role
name: configmap-modifier
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: Group
name: system:serviceaccounts:opa
apiGroup: rbac.authorization.k8s.io
---
kind: Service
apiVersion: v1
metadata:
name: opa
namespace: opa
spec:
selector:
app: opa
ports:
- name: https
protocol: TCP
port: 443
targetPort: 443
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: opa
namespace: opa
name: opa
spec:
replicas: 1
selector:
matchLabels:
app: opa
template:
metadata:
labels:
app: opa
name: opa
spec:
containers:
# WARNING: OPA is NOT running with an authorization policy configured. This
# means that clients can read and write policies in OPA. If you are
# deploying OPA in an insecure environment, be sure to configure
# authentication and authorization on the daemon. See the Security page for
# details: https://www.openpolicyagent.org/docs/security.html.
- name: opa
image: openpolicyagent/opa:latest
args:
- "run"
- "--server"
- "--tls-cert-file=/certs/tls.crt"
- "--tls-private-key-file=/certs/tls.key"
- "--addr=0.0.0.0:443"
- "--addr=http://127.0.0.1:8181"
- "--log-format=json-pretty"
- "--set=decision_logs.console=true"
volumeMounts:
- readOnly: true
mountPath: /certs
name: opa-server
readinessProbe:
httpGet:
path: /health?plugins&bundle
scheme: HTTPS
port: 443
initialDelaySeconds: 3
periodSeconds: 5
livenessProbe:
httpGet:
path: /health
scheme: HTTPS
port: 443
initialDelaySeconds: 3
periodSeconds: 5
- name: kube-mgmt
image: openpolicyagent/kube-mgmt:0.8
args:
- "--replicate-cluster=v1/namespaces"
- "--replicate=extensions/v1beta1/ingresses"
volumes:
- name: opa-server
secret:
secretName: opa-server
---
kind: ConfigMap
apiVersion: v1
metadata:
name: opa-default-system-main
namespace: opa
data:
main: |
package system
import data.kubernetes.admission
main = {
"apiVersion": "admission.k8s.io/v1beta1",
"kind": "AdmissionReview",
"response": response,
}
default response = {"allowed": true}
response = {
"allowed": false,
"status": {
"reason": reason,
},
} {
reason = concat(", ", admission.deny)
reason != ""
}
The file creates the necessary RBAC components, a Deployment, a ConfigMap, and a Service. There are two points that we need to emphasize in this definition file:
Now, apply the definition:
kubectl apply -f admission-controller.yaml
👇👇
For the admission controller to work, we need an admission webhook that receives the admission HTTP callbacks and executes them. Now, let’s create our webhook configuration file:
cat > webhook-configuration.yaml <<EOF
kind: ValidatingWebhookConfiguration
apiVersion: admissionregistration.k8s.io/v1beta1
metadata:
name: opa-validating-webhook
webhooks:
- name: validating-webhook.openpolicyagent.org
namespaceSelector:
matchExpressions:
- key: openpolicyagent.org/webhook
operator: NotIn
values:
- ignore
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*"]
clientConfig:
caBundle: $(cat ca.crt | base64 | tr -d '\n')
service:
namespace: opa
name: opa
EOF
This webhook configuration has the following properties:
Now, before we apply the configuration, let’s label the kube-system and opa namespaces so that they’re not within the webhook scope:
kubectl label ns kube-system openpolicyagent.org/webhook=ignore
kubectl label ns opa openpolicyagent.org/webhook=ignore
And apply the following configuration to register OPA as an admission controller.
kubectl apply -f webhook-configuration.yaml
OPA uses the Rego language to describe policies. For our lab, we’ll use the example mentioned in the official documentation. Our ingress-whitelist.rego should look as follows:
package kubernetes.admission
operations = {"CREATE", "UPDATE"}
deny[msg] {
input.request.kind.kind == "Ingress"
operations[input.request.operation]
host := input.request.object.spec.rules[_].host
not fqdn_matches_any(host, valid_ingress_hosts)
msg := sprintf("invalid ingress host %q", [host])
}
valid_ingress_hosts = {host |
whitelist :=input.request.namespace.metadata.annotations["ingress-whitelist"]
hosts := split(whitelist, ",")
host := hosts[_]
}
fqdn_matches_any(str, patterns) {
fqdn_matches(str, patterns[_])
}
fqdn_matches(str, pattern) {
pattern_parts := split(pattern, ".")
pattern_parts[0] == "*"
str_parts := split(str, ".")
n_pattern_parts := count(pattern_parts)
n_str_parts := count(str_parts)
suffix := trim(pattern, "*.")
endswith(str, suffix)
}
fqdn_matches(str, pattern) {
not contains(pattern, "*")
str == pattern
}
If you’re new to Rego, the code may seem cryptic at first, but Rego makes it really easy to define policies. Let’s spend a few moments highlighting how this policy enforces using Ingress namespace from the whitelist:
The reason we have two functions with the same name and signature is because of a limitation in the Rego language that prevents functions from producing more than one output value. So, to make more than one validation at the same time with different logic, you must use multiple functions with the same name.
In a real-world case, you should thoroughly test your Rego code before applying it to the cluster. There are several ways to perform quality assurance on Rego code including Unit Testing. You can (and should) also use the Rego Playground to try the code and spot any errors using sample data.
To apply the policy to the cluster, you need to create a ConfigMap with the file contents in the opa namespace:
kubectl create configmap ingress-whitelist --from-file=ingress-whitelist.rego
Next, let’s create two namespaces: one for the QA environment and the other for production. Notice that both of them contain the ingress-whitelist annotation, holding a list of the domain patterns that the Ingress hostname should strictly be part of.
apiVersion: v1
kind: Namespace
metadata:
annotations:
ingress-whitelist: "*.qa.acmecorp.com,*.internal.acmecorp.com"
name: qa
apiVersion: v1
kind: Namespace
metadata:
annotations:
ingress-whitelist: "*.acmecorp.com"
name: production
Apply both files to the cluster:
kubectl apply -f qa-namespace.yaml -f production-namespace.yaml
Next, let’s create an Ingress that uses one of the allowed domains:
kubectl apply -f - <<EOT
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-ok
namespace: production
spec:
rules:
- host: signin.acmecorp.com
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
EOT
Ensure that the Ingress was created:
kubectl get ing -n production
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-ok <none> signin.acmecorp.com 192.168.99.101 80 54s
Now, let’s try to create an Ingress that should not be allowed:
$ kubectl apply -f - <<EOT
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-bad
namespace: qa
spec:
rules:
- host: acmecorp.com
http:
paths:
- backend:
serviceName: nginx
servicePort: 80
EOT
Error from server (invalid ingress host "acmecorp.com"): error when creating "STDIN": admission webhook "validating-webhook.openpolicyagent.org" denied the request: invalid ingress host "acmecorp.com"
As you can see from the output, the API server refused to create the Ingress object since the hostname we tried to use for the QA Ingress is reserved for production.
Your policy is not working as expected, or at all? Most likely you have some kind of error in your Rego code. You can always check the status of the policy by examining the properties of the ConfigMap that contains the policy:
kubectl get cm ingress-whitelist -o yaml
Observe the metadata.annotations.openpolicyagent.org/policy-status. For example, the following snippet shows a syntax error in the Rego code of the ConfigMap:
metadata:
annotations:
openpolicyagent.org/policy-status: '{"status":"error","error":{"code":"invalid_parameter","message":"error(s)
occurred while compiling module(s)","errors":[{"code":"rego_unsafe_var_error","message":"var
namespaces is unsafe","location":{"file":"opa/ingress-whitelist/ingress-whitelist.rego","row":14,"col":2}},{"code":"rego_unsafe_var_error","message":"var
_ is unsafe","location":{"file":"opa/ingress-whitelist/ingress-whitelist.rego","row":16,"col":2}}]}}'
Self-service developer platform is all about creating a frictionless development process, boosting developer velocity, and increasing developer autonomy. Learn more about self-service platforms and why it’s important.
Explore how you can get started with GitOps using Weave GitOps products: Weave GitOps Core and Weave GitOps Enterprise. Read more.
More and more businesses are adopting GitOps. Learn about the 5 reasons why GitOps is important for businesses.
Implement the proper governance and operational excellence in your Kubernetes clusters.
Comments and Responses