14-days FREE Trial

 

Right-size Kubernetes cluster, boost app performance and lower cloud infrastructure cost in 5 minutes or less

 

GET STARTED

  Blog

Kubernetes Vulnerable to DoS Attacks, You Need to Patch NOW!

 

 

 

Kubernetes Vulnerable to DoS Attacks, you Need to Patch NOW!

 

The recently-discovered vulnerabilities allow an attacker to launch a DoS (Denial of Service) attack against the machines running Kubernetes, bringing them to their knees. Fortunately, Kubernetes development team has already addressed this issue and provided the necessary patches to mitigate the threat.

Kubernetes 1.0 was released in mid-2015. Written in Go, the technology has quickly found its way to the top. Despite the existence of other similar technologies like Docker Swarm, Apache Mesos, Nomad, Rancher and others, Kubernetes remains the most widely used with a good margin.

The security flaw was discovered in the net/http library of the Go programming language in which Kubernetes is written. Since the weak point lies in the core language itself, it affects all versions and all components of Kubernetes. The threat was revealed by Kubernetes Product Security Committee's Micah Hausler on the announcement list for Kubernetes security issues.

In an unpatched version of Kubernetes, any process that listens for HTTP or HTTPS connections is vulnerable to a DoS attack.

Discovery and Attack Vector

On August 13, Netflix released a high-severity security advisory in which it disclosed that third-party HTTP/2 implementations could suffer from a DoS attack. A DoS is an information-security term that describes a type of offensive in which the victim service is overloaded with so many fake requests. As a result, the service is no longer capable of responding to legitimate requests. In extreme cases, the attack leads to depleting the system resources of the entire machine (CPU, memory, network bandwidth, disk I/O,etc.), causing even more damage.

The discovered vulnerability in HTTP/2 allows the attacker to abuse the Transport Layer of the protocol, asking the server to do something that generates an output response and refusing to read that response. If the server does not handle its queues efficiently, this may lead to excessive use of memory and CPU, eventually causing the other legitimate connections to be denied and, ultimately, the application to crash.

Netflix announced eight vulnerabilities in their advisory that affect HTTP/2 implementations; two of them affect the Go net/http library: CVE-2019-9512 "Ping Flood" and CVE-2019-9514 "Reset Flood." Any application written in Go that uses the net/http package to listen for HTTP/2 requests is vulnerable to DoS attack, including Kubernetes. The following is a brief overview of how the offense can happen through the discovered vulnerabilities:

CVE-2019-9512 "Ping Flood": the attacker hammers the HTTP/2 listener with a continuous flow of ping requests. The recipient - to respond to each request - starts queuing the responses one after the other. That queue could grow, allocating more memory and CPU until the application crashes.

CVE-2019-9514 "Reset Flood": this attack has a similar theme as the first, except that it exploits the RST_STREAM frame of HTTP/2. RST_STREAM is simply a frame type that - when sent from a peer - signals to the other peer that the connection needs to be canceled. So, a DoS attack can be crafted by opening several streams to the server and sending invalid data through them. Having received invalid data, the server sends RST_STREAM frames to the attacker to cancel the "invalid" connection. With lots of RST_STREAM responses, they start to queue. As the queue gets more massive, more and more CPU and memory get allocated to the application until it eventually crashes.

The Fix

Kubernetes has released the required patches to mitigate the issues as mentioned above. The new versions were built using the patched versions of Go so that the required fixed are applied to the net/http library. The patched releases are listed here:

  • Kubernetes v1.15.3 - go1.12.9
  • Kubernetes v1.14.6 - go1.12.9
  • Kubernetes v1.13.10 - go1.11.13

If you need assistance on how you can upgrade your cluster to mitigate the issue, please refer to this document: https://kubernetes.io/docs/tasks/administer-cluster/cluster-management/#upgrading-a-cluster

If you are using other non-Kubernetes products that are written in Go, it is highly recommended that you obtain their patched versions. Finally, if you are a Go programmer, make sure you use the latest patched version of Go to avoid the discovered vulnerabilities affecting the applications you're developing.

Mohamed Ahmed

Aug 21, 2019