Kubernetes Patterns: Unchangeable Configuration Pattern

When there is a need to inject external data to a Kubernetes Pod, we use environment variables, ConfigMaps, and Secrets. Each method has its own use-case scenario and best practice. However, you sometimes require that no changes should be made to the application configuration once the application is deployed. Any configuration change should be through a new version update. In such a scenario, you need the configuration to be tightly coupled with the application. At the same time, you need this configuration to be pluggable so that it can be used with multiple Pods as needed. In a containerized application, you can implement this pattern by devoting a container just for holding the configuration data. If you’re using Docker, this is as simple as using a volume and linking it to the application container In Kubernetes, and since there are no linked volumes, we can use init containers with a little tweaking. In this article, we discuss both approaches.

The Example Application

For the labs in this article, we’ll use a simple Python Flask application that creates a “Hello World!” API endpoint. The API uses a configuration file to determine the message that it should reply with, to the HTTP request. The application files are listed below:

app/main.py:

from flask import Flask
app = Flask(__name__)
@app.route("/")
def hello():
    app.config.from_pyfile('/config/config.cfg')
    return app.config['MSG']

if __name__ == "__main__":
    # Only for debugging while developing
    app.run(host='0.0.0.0', debug=True, port=80)

config.cfg:

MSG="Welcome to Flask"

The Docker Case: Using Linked Volumes

Docker allows you to expose a volume from one container and link it to another. This allows the volume to be shared between both containers. The following illustration depicts this scenario:

First, we need to create the “config” container. That is the one that has the sole purpose of holding the configuration file. The Dockerfile for our config container may look as follows:

FROM scratch
COPY config.cfg /config/config.cfg
VOLUME /config

Notice that we are using the scratch image. The scratch image is special as it does not contain any filesystem or libraries. We don’t need anything from this image other than holding our volume. Let’s build this image:

docker build -t magalixcorp/appconfig -f Dockerfile-config .

We don’t even need to run a container from this image; by just creating the container we are able to use its volume:

docker create --name appconfig magalixcorp/appconfig :

The colon “:” at the end of the command is a shell builtin. In our example, it does nothing other than giving the docker to create and subcommand an executable to run. Remember, we didn’t specify an ENTRYPOINT or CMD in our Dockerfile.

Moving to our application, the Dockerfile may look as follows:

FROM tiangolo/uwsgi-nginx-flask:python3.7
COPY ./app /app

Let’s build our application image:

docker build -t magalixcorp/app -f Dockerfile-app .

To run our application, we need to provide the config.cfg file, which is currently part of the appconfig image. With a simple Docker command, we can link appconfig’s volume with the application container as follows:

docker run -d -p 8080:80 --volumes-from appconfig magalixcorp/app

Let’s confirm that the application is functioning correctly:

$ curl localhost:8080                                                                                                                                            
Welcome to Flask

If we need to change “Welcome to Flask” to something else, this requires building a new version of the appconfig image.

The Kubernetes Case: Using Init Containers

By default, containers in the same Kubernetes Pod share the same volume. However, a given container cannot share its own data with another container even if both are living on the same Pod (at least not as of the time of this writing). One possible approach to this limitation is to use a sidecar container. A sidecar container (a container sharing the same Pod with the main container) would be responsible for availing the data to the application container. You need to implement some mechanism to ensure that the application container does not start unless the sidecar finishes its work. Additionally, the sidecar container remains running after its job is done, consuming resources unnecessarily. A better approach is to use an Init container. Init containers are guaranteed to start before any other containers on the same Pod. The application container does not start unless the init container exits with no errors.

The following illustration demonstrates what we are going to build:

Since we are not linking volumes, we cannot use the scratch image with the init container. We need an image that has at least the cp command available. So, we can use busybox: a minimal Linux image that has basic tools.

Our config container’s Dockerfile is slightly different than our previous example. It may look as follows:

FROM busybox
ADD config.cfg /src/config.cfg
ENTRYPOINT [ "sh","-c","cp /src/config.cfg $1","--" ]

We copy the local configuration file to a temp directory inside the image (/src). Then we instruct the container to copy this file to whichever location specified through command line arguments. Adding any further arguments will cause the copy command to file, so we add the double dash “--” to prevent accepting any more arguments after the destination directory.

The application Dockerfile will not change. Let’s have a look at the Pod definition that makes use of both containers:

kind: Pod
apiVersion: v1
metadata:
  name: myflaskapp
spec:
  initContainers:
  - image: magalixcorp/appconfig
    name: appconfig
    args:
    - "/config"
    volumeMounts:
    - mountPath: "/config"
      name: config-dir
  containers:
  - image: magalixcorp/app
    name: app
    volumeMounts:
    - mountPath: "/config"
      name: config-dir
  volumes:
  - name: config-dir
    emptyDir: {}

The definition uses the init Containers to avail the configuration file to the application. Notice that we must pass the destination directory as an argument to the init container (lines 9 and 10). Otherwise, the container will fail and the application container will never start.

Both the application container and the init container use an emptyDir volume mounted at /config. When this Pod starts, the Init container copies the configuration file from the image to the shared volume. Later on, the application container launches and uses the configuration file at /config.

Complicated? Maybe, But There’s Some Value in it

The whole point of using separate images/containers for holding your configuration is immutability. Once the configuration is deployed and used by the application, there is no way to change it except by creating a new image, with a new tag. If you used ConfigMaps in the preceding example, there’s a very good chance that someone would change and apply the definition file or, worse, use kubectl edit.

Depending on the criticality of your application and the importance of enforcing a well-defined configuration history, you may opt for the unchangeable configuration pattern. Let’s have a quick look at the advantages of this pattern:

• You certainly have different configuration settings for different environments (dev, test, stage, prod, etc.) You can have a separate “configuration image” for each environment. Deploying your Pod to a different environment is as easy as changing the configuration image accordingly. Since the configuration data is immutable by nature, there’s very little chance for errors.
• Unlike ConfigMaps, having the configuration a traditional Docker image allows for testing even outside the cluster. For example, developers don’t need to set up a local K8s cluster to try different configuration settings.
• You are not bound to ConfigMaps limits. For example, you cannot use the same ConfigMap in multiple namespaces, you have to create a separate one for each namespace. On the other hand, a config container can be dropped in any Pod’s init containers stanza to avail its data to the rest of the Pod containers.

TL;DR

• The Unchangeable Configuration Pattern entails “sealing” your configuration settings by making them immutable. Once the configuration is deployed, it cannot be changed unless a new version gets created.
• One way of following this pattern is by building a Docker image (configuration image) that adds the configuration data as an image layer. Later on, containers can expose this data through volumes.
• In Kubernetes, you can use configuration images to deploy init containers. The init container copies the data from the image to a volume. This volume is shared among all the containers living in the same Pod.
• The Unchangeable Configuration Pattern adds a layer of complexity. But it can prove useful in environments with untraditional configuration requirements, or where you need to maintain strict configuration immutability.
• Notice that this pattern does not approach the security and confidentiality of the configuration data. Hence, you should pay careful attention if you intend to store sensitive information on a configuration image. A Secret may be a better option here.