Extending the Kubernetes Controller

Overview

At the core of Kubernetes itself, there is a large set of controllers. A controller ensures that a specific resource is (and remains) at the desired state dictated by a declared definition. If a resource deviates from the desired state, the controller is triggered to do the necessary actions to get the resource state back to where it should be. But, how do controllers “know” that a change happened? For example, when you scale up a deployment, you actually send a request to the API server with the new desired configuration. The API server in return publishes the change to all its event subscribers (any component that listens for changes in the API server). Thus, the Deployment controller creates one or more Pod to conform to the new definition. A new Pod creation is, in itself, a new change that the API server also broadcasts to the event listeners. So, if there are any actions that should get triggered on new Pod creation, they kick in automatically. Notice that Kubernetes uses the declarative programming methodology, not the imperative one. This means that the API server only publishes the new definition. It does not instruct the controller or any event listener about how they should act. The implementation is left to the controller.

While native Kubernetes controllers like Deployments, StatefulSets, Services, Job, etc. are enough on their own to handle most application needs, sometimes you want to implement your own custom controller. Kubernetes allows you to extend and build upon the existing functionality without having to break or change Kubernetes’s source code. In this article, we discuss how we can do this and the best practices.

Custom Controller or Operator?

From the origin of Kubernetes, it was thought of as controllers as the way developers can extend Kubernetes functionality by providing new behavior. Because of the many phases that extended-controllers has passed through, we can roughly classify them into two main categories:

• Custom Controllers: those are controllers that act upon the standard Kubernetes resources. They are used to enhance the platform and add new features.
• Operator: at their heart, they are custom controllers. However, instead of using the standard K8s resources, they act upon custom resource definitions (CRDs). Those are resources that were created specifically for the operator. Together, an operator and its CRD can handle complex business logic that a native or an extended controller cannot handle.

The above classification is only used to differentiate between different concepts that you need to understand in each model. But in the end, the concept stays the same; we are extending Kubernetes by creating a new controller. In this article, we are interested in the first type; custom controllers.

Do I Have To Write My Custom Controller in Go?

Although Golang has established itself as a robust system-programming language, it’s not mandatory to be used when developing custom controllers. The reason why it is very common in this domain is that Kubernetes itself was written in Go. Additionally, Go naturally has a complete client library for working with Kubernetes. However, there’s nothing that can stop you from writing a controller in Python, Ruby, or even as a bash script. All that you need is a program that can send various HTTP calls to the API server.

How Does a Custom Controller Watch The Resource Definition Changes?

As mentioned, controllers continuously watch for any changes that occur to the resources they manage. If a change occurs, the controller executes the necessary actions to bring back the resource as close as possible to the desired state. A resource is created through a definition. For example, the following is the definition of a Pod:

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec:
  containers:
  - name: myapp-container
    image: busybox
    command: ['sh', '-c', 'echo Hello Kubernetes! && sleep 3600']

If we deploy the above definition, a Pod controller will be assigned to monitor the resource and act upon any changes when they happen. Technically, a controller can monitor and watch any field in the resource definition. However, metadata, annotations, and ConfigMaps are the most suitable fields for detecting configuration changes. Let’s have a brief discussion on how each of them can be used:

Labels: they are most suitable for storing key-value data. Common usage scenarios are when you need to select objects. For example, production pods can have app=prod label. When you need to select production pods, you need to pass app: label in the selector labels. However, labels have some limitations like accepting alphanumeric characters only. If you need to store more data than just a key-value pair, then annotations may be a better option.

Annotations: annotations serve the same purpose as labels: they add metadata to the resource. However, annotations are not indexed in the Kubernetes database. This means that they cannot be used for selection operations. Annotations do not have the limitations that labels impose. Hence, you can add non-identifying data to the resource through an annotation. Common usage scenarios include options that the resource uses. For example, the application URL.

ConfigMaps: if the configuration data that you want the resource to use does not fit into a label or an annotation, you should use a ConfigMap. The controller can watch the ConfigMap for any changes and update the resource accordingly.

LAB: Automatically Restart Pods When Their ConfigMap Changes

The Problem

Our application depends on an external configuration file to work. It’s a Python API server that displays a message when it receives an HTTP request. The application uses a configuration file to determine the message it should display. The configuration file is stored in a ConfigMap and mounted to the application through a volume. When we need to change the application message, we change the ConfigMap. The problem is that we also have to restart all the Pods using this ConfigMap to reflect the changes. The application is not designed to hot-reload when it detects a configuration change. We need to automate this process.

The Approach

We can create a custom controller that does the following:

• Continuously watch the API server for changes.
• When it detects a change in our ConfigMap it searches for all the Pods that have app=frontend label and deletes them.
• Because our application is deployed through a Deployment controller, all deleted Pods are automatically started again.

The full workflow can be depicted in the following diagram (the red color represents the set of changes that occur as a result of a modified ConfigMap).

There are a number of components that need to be built for this lab to work. Let’s start with our application.

Application Files

The application uses Python Flask to respond to HTTP requests. The structure looks as follows:

app/main.py

from flask import Flask
app = Flask(__name__)
app.config.from_pyfile('/config/config.cfg')
@app.route("/")
def hello():
    return app.config['MSG']

if __name__ == "__main__":
    # Only for debugging while developing
    app.run(host='0.0.0.0', debug=True, port=80)

The script loads the configuration file, /config/config.cfg, and uses it to display the message whenever the / endpoint is hit. The configuration file looks as follows:

config.cfg:

MSG="Welcome to Kubernetes"

We need to place this application in a container, so our Dockerfile looks like this:

FROM tiangolo/uwsgi-nginx-flask:python3.7
COPY ./app /app

Notice that the Dockerfile does not reference the config.cfg file. That’s because config.cfg will be mounted to the application pod through a volume backed by a ConfigMap.

Let’s build and push this image:

$ docker build -t magalixcorp/flask:cuscon -f Dockerfile-app . --no-cache                                                                                                   
Sending build context to Docker daemon  9.728kB
Step 1/2 : FROM tiangolo/uwsgi-nginx-flask:python3.7
 ---> 53353fb7df32
Step 2/2 : COPY ./app /app
 ---> ee35c3f92fcd
Successfully built ee35c3f92fcd
Successfully tagged magalixcorp/flask:cuscon
$ docker push magalixcorp/flask:cuscon                                                                                                                                      
The push refers to repository [docker.io/magalixcorp/flask]
-- output truncated --
cuscon: digest: sha256:a88b1401dde496c9a6d10dda190fc97222f71da629a60710f47c6e015de2f08e size: 5964

We cannot test our application yet, it still needs the configuration. Let’s create the ConfigMap and Deployment controller.

The ConfigMap

The definition for our ConfigMap file looks as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "frontend-config"
  annotations:
    magalix.com/podDeleteMatch: "app=frontend"
data:
  config.cfg: 
    MSG="Welcome to Kubernetes"

Notice that we added an annotation to the definition at line 6. As mentioned, an annotation acts as a label in placing arbitrary metadata to a resource. In this definition, we are using it to add a label that defines which Pods are using this ConfigMap. Later on, when we create the Pods, we need to add app=frontend label to them if we want them to be restarted when the ConfigMap’s data changes. Let’s apply this definition:

$ kubectl apply -f configmap.yml                                                                                                                                            
configmap/frontend-config created

The Deployment

Our deployment definition looks as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
  labels:
    app: frontend
spec:
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: app
        image: magalixcorp/flask:cuscon
        volumeMounts:
        - name: config-vol
          mountPath: /config
      volumes:
      - name: config-vol
        configMap:
          name: frontend-config

A few things to note about this deployment before we apply it:

• It uses the app=frontend label for its Pods.
• It uses the ConfigMap we created earlier to mount the configuration file to the Pod through a volume.

Let’s apply this deployment and test that the application is working properly:

$ kubectl apply -f deployment.yml                                                                                                                                           
deployment.apps/frontend created
$ kubectl get pods                                                                                                                                                          
NAME                        READY   STATUS    RESTARTS   AGE
frontend-5989794999-wlgqh   1/1     Running   0          4s
$ kubectl exec -it frontend-5989794999-wlgqh bash                                                                                                                           
root@frontend-5989794999-wlgqh:/app# curl localhost && echo 
Welcome to Kubernetes
root@frontend-5989794999-wlgqh:/app#

In a real-world scenario, we’d create a service and/or Ingress for this deployment. However, we needed to keep things simple to concentrate on the main concept.

The Custom Controller

In the second part of this lab, we create the custom controller that would delete the Pods when their ConfigMap changes. Notice that restarting the Pods is done by the Deployment and not the custom controller.

We need to simply create a program/script that:

• Continuously watch the API for changes in the ConfigMap.
• When a change is detected, it enumerates all the Pods with the label that matches the one in the ConfigMap annotation and deletes them.

The above procedure can be implemented in all modern programming languages. Even a shell script can do the job. Most similar implementations would use Go as their language choice. But we thought Python is easier to understand even if you’ve never programmed in it before.

Since our Custom Controller is just a program that needs to run indefinitely, we can deploy it in a container and manage it through a Deployment controller. Before we delve into the code, we need to pause a little and explain the architecture we’ll be using.

The Architecture

As mentioned, we’ll launch the controller code through a Pod managed by a Deployment controller. Only one replica is deployed since we don’t need several controllers managing the same resource.

To access the Kubernetes API using raw HTTP calls (no kubectl) you have a number of options:

• Use native client libraries. Currently, Kubernetes officially supports Go, Python, .Net, JavaScript, Java, and Haskell. There is also a number of community-supported libraries in other languages. For more information, please refer to https://kubernetes.io/docs/reference/using-api/client-libraries/
• Use kubectl to create a proxy server. The proxy server listens on a port of your choice and acts as a reverse proxy to the cluster. It relays your API calls to the cluster and it takes care of determining the API it should connect to, the authentication tokens and other required aspects. inister-cluster/access-cluster-api/.

In our lab, we use a sidecar container

You should use the first option if you need full control over how the API call is made and handle authentication yourself. But it involves much coding than the second option. For more information about accessing the Kubernetes API, please refer to https://kubernetes.io/docs/tasks/adm on which we run the kubectl proxy. Later on, the application container can connect to the API by accessing the kubectl proxy on localhost. This pattern is referred to as the Ambassador pattern, which was covered in another article.

So, now that we have a broad idea of how our solution would look like, let’s jump into code.

The Code

Our program’s code looks as follows:

import requests
import os
import json
import logging
import sys

log = logging.getLogger(__name__)
out_hdlr = logging.StreamHandler(sys.stdout)
out_hdlr.setFormatter(logging.Formatter('%(asctime)s %(message)s'))
out_hdlr.setLevel(logging.INFO)
log.addHandler(out_hdlr)
log.setLevel(logging.INFO)


base_url = "http://127.0.0.1:8001"


namespace = os.getenv("res_namespace", "default")


def kill_pods(label):
    url = "{}/api/v1/namespaces/{}/pods?labelSelector={}".format(
        base_url, namespace, label)
    r = requests.get(url)
    response = r.json()
    pods = [p['metadata']['name'] for p in response['items']]
    for p in pods:
        url = "{}/api/v1/namespaces/{}/pods/{}".format(base_url, namespace, p)
        r = requests.delete(url)
        if r.status_code == 200:
            log.info("{} was deleted successfully".format(p))
        else:
            log.error("Could not delete {}".format(p))


def event_loop():
    log.info("Starting the service")
    url = '{}/api/v1/namespaces/{}/configmaps?watch=true"'.format(
        base_url, namespace)
    r = requests.get(url, stream=True)
    for line in r.iter_lines():
        obj = json.loads(line)
        event_type = obj['type']
        configmap_name = obj["object"]["metadata"]["name"]
        if "annotations" in obj["object"]["metadata"]:
            if "magalix.com/podDeleteMatch" in obj["object"]["metadata"]['annotations']:
                label = obj["object"]["metadata"]["annotations"]["magalix.com/podDeleteMatch"]
        if event_type == "MODIFIED":
            log.info("Modification detected")
            kill_pods(label)


event_loop()

Let’s have a quick discussion about what this code exactly does:

• Lines 1-5: we import the necessary libraries.
• Lines 7-12: the necessary logic for implementing logging. We need to log application messages to STDOUT so that we can later check them using kubectl logs.
• Line 15: the base URL of the API server. As mentioned in the architecture, we use a sidecar container to act as a reverse proxy to the cluster. Sidecar containers live in the same Pod as the application and can be accessed through the loopback interface.
• Line 18: we need to acquire the namespace name. We use an environment variable that the Deployment shall possess at runtime. if for any reason the namespace name was not available, we default to “default”.
• Lines 21-33: this is the function that is used to list and kill the Pods that match our label. The label is passed from the event_loop function (line 50), which watches for changes in the ConfigMap. The implementation of the kill_pods function is:
• Issue an HTTP call to get the list of Pods matching the label (lines 22-25).
• The response contains all the information about the Pods. We only need their names, so we extract their metadata.name from the response (line 26).
• Once we have a list containing the list of Pods, we issue an HTTP DELETE request to kill each one of them (lines 27-29)
• We need to confirm that the process was successful. So, we examine the return code of the HTTP request. If it is not 200, then we report an error.
• Lines 36-50: this is the event_loop function, the main application logic. It continuously monitors the API server for changes and acts upon the changes that we are interested in. Let’s see how it does this:
• It initiates an HTTP GET request to the API endpoint that watches ConfigMaps. It uses the stream=True to ensure that the connection stays open indefinitely (lines 38-40).
• Once any changes happen to any ConfigMap, this endpoint sends a JSON object which contains the details of what happened. The response includes the ConfigMap name, the operation that occurred (ADDED for newly created objects, MODIFIED for changed objects), and the rest of the ConfigMap’s information including the annotations.
• The function examines the response. If the ConfigMap annotation matches magalix.com/podDeleteMatch, it extracts the value of this annotation (app=frontend in our case).
• Now that it has the label, it calls the kill_pods function, passing in the label as an argument.

The Controller’s Application Container

As discussed, we’ll need two containers for our custom controller; the application container and the Ambassador container. The Dockerfile for the application container should look like this:

FROM python:latest
RUN pip install requests
COPY main.py /main.py
ENTRYPOINT [ "python", "/main.py" ]

Let’s build and push this image:

$ docker build -t magalixcorp/controller-app:cuscon -f Dockerfile-custom-controller .
$ docker push magalixcorp/controller-app:cuscon

The Custom Controller Deployment

The deployment definition should look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cuscon
  labels:
    app: cuscon
spec:
  selector:
    matchLabels:
      app: cuscon
  template:
    metadata:
      labels:
        app: cuscon
    spec:
      containers:
      - name: proxycontainer
        image: lachlanevenson/k8s-kubectl
        command: ["kubectl","proxy","--port=8001"]
      - name: app
        image: magalixcorp/controller-app:cuscon
        env:
          - name: res_namespace
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

A few notes to make about the definition before we apply it:

• For the kubectl proxy (the Ambassador container), we use a ready-made image that has kubectl already installed. We only need to run the command kubectl proxy --port=8001 as the entry point.
• In the application container, we use the image we built earlier.
• Since our application is designed to expect the namespace name through an environment variable, we pass it through the env stanza. The namespace name can be brought from the downward api. For more information about this pattern, refer to our Reflection Pattern article.

Let’s apply this Deployment:

kubectl apply -f cuscon-deployment.yml

Testing Our Work

Currently, we should be having two deployments, each with one Pod. To confirm that our custom controller is working, we need to make and apply a change to the ConfigMap. Change the message in the configmap.yml file to “Welcome to Custom Controllers”. The file should look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: "frontend-config"
  annotations:
    magalix.com/podDeleteMatch: "app=frontend"
data:
  config.cfg: 
    MSG="Welcome to Custom Controllers!"

Apply the modified configmap and watch the Flask App Pods:

$ kubectl apply -f configmap.yml                                                                                                                                            
configmap/frontend-config configured
$ kubectl get pods --watch                                                                                                                                                  
NAME                        READY   STATUS    RESTARTS   AGE
cuscon-68cdd77d5c-s45wj     2/2     Running   0          25m
frontend-5989794999-zktqw   1/1     Running   0          6s
NAME                        READY   STATUS        RESTARTS   AGE
frontend-5989794999-zktqw   1/1     Terminating   0          43s
frontend-5989794999-gg294   0/1     Pending       0          1s
frontend-5989794999-gg294   0/1     Pending       0          1s
frontend-5989794999-gg294   0/1     ContainerCreating   0          1s
frontend-5989794999-gg294   1/1     Running             0          3s
frontend-5989794999-zktqw   0/1     Terminating         0          46s
frontend-5989794999-zktqw   0/1     Terminating         0          47s
frontend-5989794999-zktqw   0/1     Terminating         0          47s

As you can see, when we made a change to the ConfigMap, the custom controller automatically detected that change and acted upon it by deleting all the Pods with matching labels. The Flask App deployment automatically started a new Pod to replace the deleted ones. The new Pod is using the updated ConfigMap.

We can also view the controller’s logs to see what happened:

$ kubectl logs cuscon-68cdd77d5c-s45wj -c app                                                                                                                               
2019-09-21 14:19:22,939 Modification detected
2019-09-21 14:19:23,003 frontend-5989794999-zktqw was deleted successfully

Finally, let’s see what message the Flask APP will respond with now:

$ kubectl exec -it frontend-5989794999-gg294 -- bash                                                                                                                        
root@frontend-5989794999-gg294:/app# curl localhost && echo
Welcome to Custom Controllers!

TL;DR

• Containers are where the real Kubernetes work happens. Internally, Kubernetes uses controllers to reconcile the state to be as close as possible to the desired one.
• Kubernetes uses the declarative programming model rather than the imperative one. In the declarative model, you only dictate the overall state that you need. The system is responsible for conforming to the desired state. The implementation is not visible nor enforced by the user.
• At its simplest form, a controller is nothing more than a program that continuously monitors the Kubernetes API for changes. When an event is detected, the program examines the event details to determine whether it should trigger an action in response. All actions can be performed through HTTP calls to the API server.
• When building a custom controller, you can either use the native client libraries (the officially supported or the community ones) or you can just use a simple script that calls the API server through kubectl proxy.
• Using this pattern, you have virtually endless possibilities of how you can implement custom application logic and automate repetitive stuff.