<img src="https://ws.zoominfo.com/pixel/JHVDdRXH2uangmUMQBZd" width="1" height="1" style="display: none;">

Monitoring of Kubernetes Clusters To Manage Large Scale Projects

Kubernetes Prometheus K8s write-for-cloud-native Large Scale
Monitoring of Kubernetes Clusters To Manage Large Scale Projects
Kubernetes Prometheus K8s write-for-cloud-native Large Scale

What Is Monitoring?

Today, the massive use of technology in large-scale ventures is very popular for businesses. Monitoring systems are responsible for monitoring a company's equipment (hardware, networks, communications, operating systems, or applications), to monitor its operation and performance, and identify potential errors. A successful monitoring system is capable of monitoring devices, infrastructure, software, facilities, and even processes in industry. We often use control systems to track device resources, such as usage and frequency of CPUs, or the amount of free RAMs. These are often used to display free space on one or more hard drives, the temperature of the CPU and other critical parts, and networking details. It’s very useful to have a good monitoring device to identify and avoid failures.

Types Of Monitoring Tools With Kubernetes

1. Prometheus

Prometheus is one of the most common surveillance tools used by Kubernetes. It’s a part of the Native Computing Base Cloud. This project, initially created by SoundCloud and subsequently donated to CNCF, is inspired by Google Borg Monitor.

As a time series Prometheus stores all of its info. This data can be queried through the language of the PromQL database and visualized through an integrated browser of text. Since Prometheus is not a dashboard, Grafana relies on it for data visualization.

Version 1.0 of this tool was published in 2016, and is now one of Kubernetes’ most commonly used monitoring devices. Other Kubernetes ecosystem resources, including Istio, include an embedded Prometheus adapter that exposes metrics produced.

Prometheus can be built as a single binary that can be run on your host or as a Docker server directly. With the Prometheus Controller, running Prometheus on Kubernetes is easily accomplished.

Monitor Kuberentes Cluster To Manage Large Scale Projects

2. Container Advisor (CAdvisor)

CAdvisor is an agent for the use of container tools and output analysis - it’s built into the Kubelet binary. CAdvistor auto-discovers all computer containers and collects memory, network use, file system and CPU statistics. CAdvisor supports native Docker containers. It does not function on the pod level but on each node. Nonetheless, be advised: CAdvisor is a simple yet restricted device, so if you're looking to store metrics for long-term usage or conduct complex monitoring behavior, cAdvisor won't suit your requirements:

3. Kubernetes Dashboard

Kubernetes Dashboard is a web-based, Kubernetes cluster UI add-on. It has several features allowing users to build and manage workloads, as well as discovery, load balancing, setup, storage, and monitoring. It's helpful to small clusters, and people who are just starting to learn Kubernetes.

This tool offers different views aggregated over all nodes for CPU and memory usage metrics. It can also be used to track workload health status (Pods, Deployments, Replica Sets, Cron jobs, etc.). It’s very quick and easy to install the Kubernetes Dashboard, which can be done using ready-to-use YAML files.

4. Kubewatch

Kubewatch is a Kubernetes watchman that publishes updates of events in a Slack channel and lets you determine the resources to be monitored. It is written in Golang and uses a client library from Kubernetes to communicate with a server from the Kubernetes API.

You can pick the resources you need to monitor: daemon sets, servers, pods, replica sets, replication controllers, services, secrets, and configuration maps using a simple YAML file.

LAB: Creating Prometheus On Kubernetes Cluster To Monitor Your Nodes

There are many ways to install Prometheus in your Kubernetes cluster:

  • Directly as a single binary running on your hosts, which is fine for learning, testing and developing purposes but not appropriate for a containerized deployment.
  • As a Docker container which has several orchestration options: Raw Docker containers, Kubernetes Deployments / StatefulSets, the Helm Kubernetes package manager, Kubernetes operators, etc.

Prometheus Web Interface (Port 9090 By Default).

The better option to deploy the Prometheus server inside a container:

docker run -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml 

Note that you can easily adapt this Docker container into a proper Kubernetes Deployment object that will mount the configuration from a ConfigMap, expose a service, and deploy multiple replicas, etc.

And then you can apply this yaml file below:

apiVersion: apps/v1
kind: Deployment
  name: prometheus-deployment
    app: prometheus
    purpose: example
  replicas: 2
      app: prometheus
      purpose: example
        app: prometheus
        purpose: example
      - name: prometheus-example
        image: prom/prometheus
          - name: config-volume
            mountPath: /etc/prometheus/prometheus.yml
            subPath: prometheus.yml
        - containerPort: 9090
        - name: config-volume
           name: prometheus-example-cm

Next, add this service file below (to the above yaml file):

kind: Service
apiVersion: v1
  name: prometheus-example-service
    app: prometheus
    purpose: example
  - name: promui
    protocol: TCP
    port: 9090
    targetPort: 9090

And we can run the below command :

kubectl create configmap prometheus-example-cm --from-fileprometheus.yml

If you don’t want to configure a LoadBalancer, then you can specify the type NodePort for your service.

After a few seconds, you should see the pods of Prometheus running in your cluster as below:

$ kubectl get pods
NAME                                     READY     STATUS    RESTARTS   AGE
prometheus-deployment-68c5f4d474-cn5cb   1/1       Running   0          3h
prometheus-deployment-68c5f4d474-ldk9p   1/1       Running   0          3h

There are several configuration tweaks that you can implement at this point, such as configuring pod Antiaffinity to force the Prometheus server pods to be located in different nodes.

Monitoring The Kubernetes Nodes With Prometheus:

The Kubernetes nodes or hosts will need to be monitored and we have plenty of tools to monitor a Linux host. In this guide, we’re going to use the Prometheus Node-exporter:

  • It’s hosted by the Prometheus project itself
  • It will be automatically deployed when we use the Prometheus operator in the next chapters
  • Can be deployed as a DaemonSet, so it will automatically scale if you add or remove nodes from your cluster.

You have several options to deploy this service, for example, using the DaemonSet in this repo for minikube:

kubectl create ns monitoring 
kubectl create -f https://raw.githubusercontent.com/bakins/minikube-prometheus-demo/master/node-exporter-daemonset.yml

Or Using Helm / Tiller:

If you want to use Helm 3, remember to create the RBAC roles and service accounts for the tiller component before proceeding.

helm init --service-account tiller
helm install --name node-exporter stable/prometheus-node-exporter

Once the chart is installed and running, you can display the service that you need:

kubectl get svc 
NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE
node-exporter-prometheus-node-exporter   ClusterIP            9100/TCP                                    17m

Once you add the config, you can start collecting and displaying the node metrics as seen in the below graph:

Monitor Kuberentes Cluster To Manage Large Scale Projects

Download Kubernetes Application Patterns E-Book


  • To sum up the monitoring process, all of your Kubernetes data can easily be put together into one view. Available in cluster explorer Kubernetes, logs provide a near-instant search with complete contextual log information.
  • Additionally, you can connect the log messages with program, network, Kubernetes, and event data while configuring logs in context.

Comments and Responses

Related Articles

Team Productivity: Resource Management

Since the introduction of containers, the method of building and running applications in an organization has

Read more
Capacity Management for Teams on Kubernetes: Setting Pod CPU and Memory Limits

Capacity management is a complex, ever-moving target, for teams on any infrastructure, whether on-prem,

Read more
Kubernetes cost saving K8s
Kubernetes Cost Optimization with Magalix

Is our Spending Getting Worse? I woke up one day to see this email from our CEO in my mailbox. I knew this

Read more

start your 14-day free trial today!

Automate your Kubernetes cluster optimization in minutes.

Get started View Pricing
No Card Required