14-days FREE Trial

 

Right-size Kubernetes cluster, boost app performance and lower cloud infrastructure cost in 5 minutes or less

 

GET STARTED

  Blog

Monitoring of Kubernetes Clusters To Manage Large Scale Projects

What Is Monitoring?

Today, the massive use of technology in large-scale ventures is very popular for businesses. Monitoring systems are responsible for monitoring a company's equipment (hardware, networks, communications, operating systems, or applications), to monitor its operation and performance, and identify potential errors. A successful monitoring system is capable of monitoring devices, infrastructure, software, facilities, and even processes in industry. We often use control systems to track device resources, such as usage and frequency of CPUs, or the amount of free RAMs. These are often used to display free space on one or more hard drives, the temperature of the CPU and other critical parts, and networking details. It’s very useful to have a good monitoring device to identify and avoid failures.

Types Of Monitoring Tools With Kubernetes


1. Prometheus

Prometheus is one of the most common surveillance tools used by Kubernetes. It’s a part of the Native Computing Base Cloud. This project, initially created by SoundCloud and subsequently donated to CNCF, is inspired by Google Borg Monitor.

As a time series Prometheus stores all of its info. This data can be queried through the language of the PromQL database and visualized through an integrated browser of text. Since Prometheus is not a dashboard, Grafana relies on it for data visualization.

Version 1.0 of this tool was published in 2016, and is now one of Kubernetes’ most commonly used monitoring devices. Other Kubernetes ecosystem resources, including Istio, include an embedded Prometheus adapter that exposes metrics produced.

Prometheus can be built as a single binary that can be run on your host or as a Docker server directly. With the Prometheus Controller, running Prometheus on Kubernetes is easily accomplished.

Monitor Kuberentes Cluster To Manage Large Scale Projects

2. Container Advisor (CAdvisor)

CAdvisor is an agent for the use of container tools and output analysis - it’s built into the Kubelet binary. CAdvistor auto-discovers all computer containers and collects memory, network use, file system and CPU statistics. CAdvisor supports native Docker containers. It does not function on the pod level but on each node. Nonetheless, be advised: CAdvisor is a simple yet restricted device, so if you're looking to store metrics for long-term usage or conduct complex monitoring behavior, cAdvisor won't suit your requirements:

3. Kubernetes Dashboard

Kubernetes Dashboard is a web-based, Kubernetes cluster UI add-on. It has several features allowing users to build and manage workloads, as well as discovery, load balancing, setup, storage, and monitoring. It's helpful to small clusters, and people who are just starting to learn Kubernetes.

This tool offers different views aggregated over all nodes for CPU and memory usage metrics. It can also be used to track workload health status (Pods, Deployments, Replica Sets, Cron jobs, etc.). It’s very quick and easy to install the Kubernetes Dashboard, which can be done using ready-to-use YAML files.

4. Kubewatch

Kubewatch is a Kubernetes watchman that publishes updates of events in a Slack channel and lets you determine the resources to be monitored. It is written in Golang and uses a client library from Kubernetes to communicate with a server from the Kubernetes API.

You can pick the resources you need to monitor: daemon sets, servers, pods, replica sets, replication controllers, services, secrets, and configuration maps using a simple YAML file.

LAB: Creating Prometheus On Kubernetes Cluster To Monitor Your Nodes

There are many ways to install Prometheus in your Kubernetes cluster:

  • Directly as a single binary running on your hosts, which is fine for learning, testing and developing purposes but not appropriate for a containerized deployment.
  • As a Docker container which has several orchestration options: Raw Docker containers, Kubernetes Deployments / StatefulSets, the Helm Kubernetes package manager, Kubernetes operators, etc.

Prometheus Web Interface (Port 9090 By Default).

The better option to deploy the Prometheus server inside a container:

docker run -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml 
 prom/prometheus

Note that you can easily adapt this Docker container into a proper Kubernetes Deployment object that will mount the configuration from a ConfigMap, expose a service, and deploy multiple replicas, etc.

And then you can apply this yaml file below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-deployment
  labels:
    app: prometheus
    purpose: example
spec:
  replicas: 2
  selector:
    matchLabels:
      app: prometheus
      purpose: example
  template:
    metadata:
      labels:
        app: prometheus
        purpose: example
    spec:
      containers:
      - name: prometheus-example
        image: prom/prometheus
        volumeMounts:
          - name: config-volume
            mountPath: /etc/prometheus/prometheus.yml
            subPath: prometheus.yml
        ports:
        - containerPort: 9090
      volumes:
        - name: config-volume
          configMap:
           name: prometheus-example-cm

Next, add this service file below (to the above yaml file):

---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-example-service
spec:
  selector:
    app: prometheus
    purpose: example
  ports:
  - name: promui
    protocol: TCP
    port: 9090
    targetPort: 9090

And we can run the below command :

kubectl create configmap prometheus-example-cm --from-fileprometheus.yml

If you don’t want to configure a LoadBalancer, then you can specify the type NodePort for your service.

After a few seconds, you should see the pods of Prometheus running in your cluster as below:

$ kubectl get pods
NAME                                     READY     STATUS    RESTARTS   AGE
prometheus-deployment-68c5f4d474-cn5cb   1/1       Running   0          3h
prometheus-deployment-68c5f4d474-ldk9p   1/1       Running   0          3h

There are several configuration tweaks that you can implement at this point, such as configuring pod Antiaffinity to force the Prometheus server pods to be located in different nodes.

Monitoring The Kubernetes Nodes With Prometheus:

The Kubernetes nodes or hosts will need to be monitored and we have plenty of tools to monitor a Linux host. In this guide, we’re going to use the Prometheus Node-exporter:

  • It’s hosted by the Prometheus project itself
  • It will be automatically deployed when we use the Prometheus operator in the next chapters
  • Can be deployed as a DaemonSet, so it will automatically scale if you add or remove nodes from your cluster.

You have several options to deploy this service, for example, using the DaemonSet in this repo for minikube:

kubectl create ns monitoring 
kubectl create -f https://raw.githubusercontent.com/bakins/minikube-prometheus-demo/master/node-exporter-daemonset.yml

Or Using Helm / Tiller:

If you want to use Helm 3, remember to create the RBAC roles and service accounts for the tiller component before proceeding.

helm init --service-account tiller
helm install --name node-exporter stable/prometheus-node-exporter

Once the chart is installed and running, you can display the service that you need:

kubectl get svc 
NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE
node-exporter-prometheus-node-exporter   ClusterIP   10.101.57.207            9100/TCP                                    17m

Once you add the config, you can start collecting and displaying the node metrics as seen in the below graph:

Monitor Kuberentes Cluster To Manage Large Scale Projects

Download Kubernetes Application Patterns E-Book

TL;DR

  • To sum up the monitoring process, all of your Kubernetes data can easily be put together into one view. Available in cluster explorer Kubernetes, logs provide a near-instant search with complete contextual log information.
  • Additionally, you can connect the log messages with program, network, Kubernetes, and event data while configuring logs in context.
Mohamed Tawfik

May 20, 2020