Weaveworks 2022.03 release featuring Magalix PaC | Learn more
Balance innovation and agility with security and compliance
risks using a 3-step process across all cloud infrastructure.
Step up business agility without compromising
security or compliance
Everything you need to become a Kubernetes expert.
Always for free!
Everything you need to know about Magalix
culture and much more
At the core of Kubernetes itself, there is a large set of controllers. A controller ensures that a specific resource is (and remains) at the desired state dictated by a declared definition. If a resource deviates from the desired state, the controller is triggered to do the necessary actions to get the resource state back to where it should be. But, how do controllers “know” that a change happened? For example, when you scale up a deployment, you actually send a request to the API server with the new desired configuration. The API server in return publishes the change to all its event subscribers (any component that listens for changes in the API server). Thus, the Deployment controller creates one or more Pod to conform to the new definition. A new Pod creation is, in itself, a new change that the API server also broadcasts to the event listeners. So, if there are any actions that should get triggered on new Pod creation, they kick in automatically. Notice that Kubernetes uses the declarative programming methodology, not the imperative one. This means that the API server only publishes the new definition. It does not instruct the controller or any event listener about how they should act. The implementation is left to the controller.
While native Kubernetes controllers like Deployments, StatefulSets, Services, Job, etc. are enough on their own to handle most application needs, sometimes you want to implement your own custom controller. Kubernetes allows you to extend and build upon the existing functionality without having to break or change Kubernetes’s source code. In this article, we discuss how we can do this and the best practices.
From the origin of Kubernetes, it was thought of as controllers as the way developers can extend Kubernetes functionality by providing new behavior. Because of the many phases that extended-controllers has passed through, we can roughly classify them into two main categories:
The above classification is only used to differentiate between different concepts that you need to understand in each model. But in the end, the concept stays the same; we are extending Kubernetes by creating a new controller. In this article, we are interested in the first type; custom controllers.
Although Golang has established itself as a robust system-programming language, it’s not mandatory to be used when developing custom controllers. The reason why it is very common in this domain is that Kubernetes itself was written in Go. Additionally, Go naturally has a complete client library for working with Kubernetes. However, there’s nothing that can stop you from writing a controller in Python, Ruby, or even as a bash script. All that you need is a program that can send various HTTP calls to the API server.
As mentioned, controllers continuously watch for any changes that occur to the resources they manage. If a change occurs, the controller executes the necessary actions to bring back the resource as close as possible to the desired state. A resource is created through a definition. For example, the following is the definition of a Pod:
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
app: myapp
spec:
containers:
- name: myapp-container
image: busybox
command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600']
If we deploy the above definition, a Pod controller will be assigned to monitor the resource and act upon any changes when they happen. Technically, a controller can monitor and watch any field in the resource definition. However, metadata, annotations, and ConfigMaps are the most suitable fields for detecting configuration changes. Let’s have a brief discussion on how each of them can be used:
Labels: they are most suitable for storing key-value data. Common usage scenarios are when you need to select objects. For example, production pods can have app=prod label. When you need to select production pods, you need to pass app: label in the selector labels. However, labels have some limitations like accepting alphanumeric characters only. If you need to store more data than just a key-value pair, then annotations may be a better option.
Annotations: annotations serve the same purpose as labels: they add metadata to the resource. However, annotations are not indexed in the Kubernetes database. This means that they cannot be used for selection operations. Annotations do not have the limitations that labels impose. Hence, you can add non-identifying data to the resource through an annotation. Common usage scenarios include options that the resource uses. For example, the application URL.
ConfigMaps: if the configuration data that you want the resource to use does not fit into a label or an annotation, you should use a ConfigMap. The controller can watch the ConfigMap for any changes and update the resource accordingly.
Our application depends on an external configuration file to work. It’s a Python API server that displays a message when it receives an HTTP request. The application uses a configuration file to determine the message it should display. The configuration file is stored in a ConfigMap and mounted to the application through a volume. When we need to change the application message, we change the ConfigMap. The problem is that we also have to restart all the Pods using this ConfigMap to reflect the changes. The application is not designed to hot-reload when it detects a configuration change. We need to automate this process.
We can create a custom controller that does the following:
The full workflow can be depicted in the following diagram (the red color represents the set of changes that occur as a result of a modified ConfigMap).
There are a number of components that need to be built for this lab to work. Let’s start with our application.
The application uses Python Flask to respond to HTTP requests. The structure looks as follows:
app/main.py
from flask import Flask
app = Flask(__name__)
app.config.from_pyfile('/config/config.cfg')
@app.route("/")
def hello():
return app.config['MSG']
if __name__ == "__main__":
# Only for debugging while developing
app.run(host='0.0.0.0', debug=True, port=80)
The script loads the configuration file, /config/config.cfg, and uses it to display the message whenever the / endpoint is hit. The configuration file looks as follows:
config.cfg:
MSG="Welcome to Kubernetes"
We need to place this application in a container, so our Dockerfile looks like this:
FROM tiangolo/uwsgi-nginx-flask:python3.7
COPY ./app /app
Notice that the Dockerfile does not reference the config.cfg file. That’s because config.cfg will be mounted to the application pod through a volume backed by a ConfigMap.
Let’s build and push this image:
$ docker build -t magalixcorp/flask:cuscon -f Dockerfile-app . --no-cache
Sending build context to Docker daemon 9.728kB
Step 1/2 : FROM tiangolo/uwsgi-nginx-flask:python3.7
---> 53353fb7df32
Step 2/2 : COPY ./app /app
---> ee35c3f92fcd
Successfully built ee35c3f92fcd
Successfully tagged magalixcorp/flask:cuscon
$ docker push magalixcorp/flask:cuscon
The push refers to repository [docker.io/magalixcorp/flask]
-- output truncated --
cuscon: digest: sha256:a88b1401dde496c9a6d10dda190fc97222f71da629a60710f47c6e015de2f08e size: 5964
We cannot test our application yet, it still needs the configuration. Let’s create the ConfigMap and Deployment controller.
The definition for our ConfigMap file looks as follows:
apiVersion: v1
kind: ConfigMap
metadata:
name: "frontend-config"
annotations:
magalix.com/podDeleteMatch: "app=frontend"
data:
config.cfg:
MSG="Welcome to Kubernetes"
Notice that we added an annotation to the definition at line 6. As mentioned, an annotation acts as a label in placing arbitrary metadata to a resource. In this definition, we are using it to add a label that defines which Pods are using this ConfigMap. Later on, when we create the Pods, we need to add app=frontend label to them if we want them to be restarted when the ConfigMap’s data changes. Let’s apply this definition:
$ kubectl apply -f configmap.yml
configmap/frontend-config created
Our deployment definition looks as follows:
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
labels:
app: frontend
spec:
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: app
image: magalixcorp/flask:cuscon
volumeMounts:
- name: config-vol
mountPath: /config
volumes:
- name: config-vol
configMap:
name: frontend-config
A few things to note about this deployment before we apply it:
Let’s apply this deployment and test that the application is working properly:
$ kubectl apply -f deployment.yml
deployment.apps/frontend created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
frontend-5989794999-wlgqh 1/1 Running 0 4s
$ kubectl exec -it frontend-5989794999-wlgqh bash
root@frontend-5989794999-wlgqh:/app# curl localhost && echo
Welcome to Kubernetes
root@frontend-5989794999-wlgqh:/app#
In a real-world scenario, we’d create a service and/or Ingress for this deployment. However, we needed to keep things simple to concentrate on the main concept.
In the second part of this lab, we create the custom controller that would delete the Pods when their ConfigMap changes. Notice that restarting the Pods is done by the Deployment and not the custom controller.
We need to simply create a program/script that:
The above procedure can be implemented in all modern programming languages. Even a shell script can do the job. Most similar implementations would use Go as their language choice. But we thought Python is easier to understand even if you’ve never programmed in it before.
Since our Custom Controller is just a program that needs to run indefinitely, we can deploy it in a container and manage it through a Deployment controller. Before we delve into the code, we need to pause a little and explain the architecture we’ll be using.
As mentioned, we’ll launch the controller code through a Pod managed by a Deployment controller. Only one replica is deployed since we don’t need several controllers managing the same resource.
To access the Kubernetes API using raw HTTP calls (no kubectl) you have a number of options:
In our lab, we use a sidecar container
You should use the first option if you need full control over how the API call is made and handle authentication yourself. But it involves much coding than the second option. For more information about accessing the Kubernetes API, please refer to https://kubernetes.io/docs/tasks/adm on which we run the kubectl proxy. Later on, the application container can connect to the API by accessing the kubectl proxy on localhost. This pattern is referred to as the Ambassador pattern, which was covered in another article.
So, now that we have a broad idea of how our solution would look like, let’s jump into code.
Our program’s code looks as follows:
import requests
import os
import json
import logging
import sys
log = logging.getLogger(__name__)
out_hdlr = logging.StreamHandler(sys.stdout)
out_hdlr.setFormatter(logging.Formatter('%(asctime)s %(message)s'))
out_hdlr.setLevel(logging.INFO)
log.addHandler(out_hdlr)
log.setLevel(logging.INFO)
base_url = "http://127.0.0.1:8001"
namespace = os.getenv("res_namespace", "default")
def kill_pods(label):
url = "{}/api/v1/namespaces/{}/pods?labelSelector={}".format(
base_url, namespace, label)
r = requests.get(url)
response = r.json()
pods = [p['metadata']['name'] for p in response['items']]
for p in pods:
url = "{}/api/v1/namespaces/{}/pods/{}".format(base_url, namespace, p)
r = requests.delete(url)
if r.status_code == 200:
log.info("{} was deleted successfully".format(p))
else:
log.error("Could not delete {}".format(p))
def event_loop():
log.info("Starting the service")
url = '{}/api/v1/namespaces/{}/configmaps?watch=true"'.format(
base_url, namespace)
r = requests.get(url, stream=True)
for line in r.iter_lines():
obj = json.loads(line)
event_type = obj['type']
configmap_name = obj["object"]["metadata"]["name"]
if "annotations" in obj["object"]["metadata"]:
if "magalix.com/podDeleteMatch" in obj["object"]["metadata"]['annotations']:
label = obj["object"]["metadata"]["annotations"]["magalix.com/podDeleteMatch"]
if event_type == "MODIFIED":
log.info("Modification detected")
kill_pods(label)
event_loop()
Let’s have a quick discussion about what this code exactly does:
As discussed, we’ll need two containers for our custom controller; the application container and the Ambassador container. The Dockerfile for the application container should look like this:
FROM python:latest
RUN pip install requests
COPY main.py /main.py
ENTRYPOINT [ "python", "/main.py" ]
Let’s build and push this image:
$ docker build -t magalixcorp/controller-app:cuscon -f Dockerfile-custom-controller .
$ docker push magalixcorp/controller-app:cuscon
The deployment definition should look like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: cuscon
labels:
app: cuscon
spec:
selector:
matchLabels:
app: cuscon
template:
metadata:
labels:
app: cuscon
spec:
containers:
- name: proxycontainer
image: lachlanevenson/k8s-kubectl
command: ["kubectl","proxy","--port=8001"]
- name: app
image: magalixcorp/controller-app:cuscon
env:
- name: res_namespace
valueFrom:
fieldRef:
fieldPath: metadata.namespace
A few notes to make about the definition before we apply it:
Let’s apply this Deployment:
kubectl apply -f cuscon-deployment.yml
Currently, we should be having two deployments, each with one Pod. To confirm that our custom controller is working, we need to make and apply a change to the ConfigMap. Change the message in the configmap.yml file to “Welcome to Custom Controllers”. The file should look like this:
apiVersion: v1
kind: ConfigMap
metadata:
name: "frontend-config"
annotations:
magalix.com/podDeleteMatch: "app=frontend"
data:
config.cfg:
MSG="Welcome to Custom Controllers!"
Apply the modified configmap and watch the Flask App Pods:
$ kubectl apply -f configmap.yml
configmap/frontend-config configured
$ kubectl get pods --watch
NAME READY STATUS RESTARTS AGE
cuscon-68cdd77d5c-s45wj 2/2 Running 0 25m
frontend-5989794999-zktqw 1/1 Running 0 6s
NAME READY STATUS RESTARTS AGE
frontend-5989794999-zktqw 1/1 Terminating 0 43s
frontend-5989794999-gg294 0/1 Pending 0 1s
frontend-5989794999-gg294 0/1 Pending 0 1s
frontend-5989794999-gg294 0/1 ContainerCreating 0 1s
frontend-5989794999-gg294 1/1 Running 0 3s
frontend-5989794999-zktqw 0/1 Terminating 0 46s
frontend-5989794999-zktqw 0/1 Terminating 0 47s
frontend-5989794999-zktqw 0/1 Terminating 0 47s
As you can see, when we made a change to the ConfigMap, the custom controller automatically detected that change and acted upon it by deleting all the Pods with matching labels. The Flask App deployment automatically started a new Pod to replace the deleted ones. The new Pod is using the updated ConfigMap.
We can also view the controller’s logs to see what happened:
$ kubectl logs cuscon-68cdd77d5c-s45wj -c app
2019-09-21 14:19:22,939 Modification detected
2019-09-21 14:19:23,003 frontend-5989794999-zktqw was deleted successfully
Finally, let’s see what message the Flask APP will respond with now:
$ kubectl exec -it frontend-5989794999-gg294 -- bash
root@frontend-5989794999-gg294:/app# curl localhost && echo
Welcome to Custom Controllers!
Self-service developer platform is all about creating a frictionless development process, boosting developer velocity, and increasing developer autonomy. Learn more about self-service platforms and why it’s important.
Explore how you can get started with GitOps using Weave GitOps products: Weave GitOps Core and Weave GitOps Enterprise. Read more.
More and more businesses are adopting GitOps. Learn about the 5 reasons why GitOps is important for businesses.
Implement the proper governance and operational excellence in your Kubernetes clusters.
Comments and Responses