<img src="https://ws.zoominfo.com/pixel/JHVDdRXH2uangmUMQBZd" width="1" height="1" style="display: none;">

Capacity Management for Teams on Kubernetes: Setting Pod CPU and Memory Limits

Kubernetes Capacity Management
Capacity Management for Teams on Kubernetes: Setting Pod CPU and Memory Limits
Kubernetes Capacity Management

Capacity management is a complex, ever-moving target, for teams on any infrastructure, whether on-prem, cloud-native, or hybrid cloud.

It can be especially difficult for teams running Kubernetes clusters. In this post, you can learn how to get started right by starting small and focusing on step 3 from the Magalix Ultimate Guide to Capacity Management eBook: setting pods’ CPU limits.

Setting the Right Pod CPU limits: Some Heuristics

When setting CPU limits in your pods, it’s important to use the right statistics to properly budget your resources.

For example, a common method of budgeting is using the average utilization of the CPU, which doesn’t work. Using the maximum or percentiles without considering metric resolution causes a lot of resource churn.

Setting the proper CPU limits makes your system predictable and performant. Not setting limits allows CPU-hungry containers to slow down all other containers on the same worker node.

On the other hand, setting low CPU limit values can degrade the performance of your applications. Setting a low value for the pod’s CPU limit exposes your apps to throttling, which can lead to significant performance issues. If you’re running infrastructure for a mobile game, a slow gaming experience can be a huge burden for user acquisition and engagement. For things like healthcare or finance, it can be a death sentence for your business, and can be even worse for your end-users (patients and clients).

At This Point, Let’s Stop and Review Pod CPU Limit Recommendations:

  • If you frequently update your pods’ CPU resources, use the 95th percentile of 1-minute resolution as a guideline to set the CPU limit of your pods.
  • Keep an eye out for CPU throttling and see if it goes up with your CPU utilization. You may need to use a higher resolution metric, i.e. 10-second data points, to set the right CPU limit value for pods.
  • If you frequently update your pods’ CPU resources, use the 95th percentile of 1-minute resolution as a guideline to set the CPU limit of your pods.
  • Keep an eye out for CPU throttling and see if it goes up with your CPU utilization. You may need to use a higher resolution metric, i.e. 10-second data points, to set the right CPU limit value for pods.
Capacity Management for Teams on Kubernetes: Setting Pod CPU and Memory Limits

Get a deeper dive on metrics in our webinar on Capacity Management - How to select the right metrics

Rewatch it Here


The Next Step: Setting Pod Memory Limits

Lastly, to get a quick start on capacity management on Kubernetes, your team will need a quick word on why setting pod’s memory limit is critical for the availability of your cluster and containers.

Many engineers shy away from setting pod memory limits because the OS will kill the container when that container exceeds the memory limit, and they think this situation is bad enough that they’d like to avoid it by not setting limits.

However, not setting the pod’s memory limit leads to a much bigger blast radius when the same container or other containers reach the limit of the running node’s available memory. Multiple pods will be OOMkilled and scheduled to another node, or the process of memory hijacking will be repeated with a separate set of containers.

Not Setting Pod’s Memory Limits is Much Like Having a Chaos Monkey in Your Cluster.

Setting low memory values also puts pods at risk of frequent crashes. Setting high memory values puts a lot of pressure on your infrastructure budget since you will have a lot of idle capacity. Neither is ideal.

Quick Tips for Setting Pod Memory Limits for Your Team:

  • Monitor memory usage at a relatively high resolution, i.e. 1 or 10-second data points for extended periods of time.
  • Always use the maximum value as your guideline.
  • Make sure that maximum value of high-resolution metrics, i.e. 10-seconds, are saved as you downsample your metrics. Use that maximum in your long term planning.
  • Re-evaluate your limit values with every major release.
  • Monitor memory usage at a relatively high resolution, i.e. 1 or 10-second data points for extended periods of time.
  • Always use the maximum value as your guideline.
  • Make sure that maximum value of high-resolution metrics, i.e. 10-seconds, are saved as you downsample your metrics. Use that maximum in your long term planning.
  • Re-evaluate your limit values with every major release.

Note: Get the recommended memory limit values automatically with KubeOptimizer. You can see the collected metrics and recommended values in a single screen! You can also apply the suggested value with a single click, and look like a super hero 😉

TL;DR

At this point, let’s stop and review pod CPU limit recommendations:

  • If you frequently update your pods’ CPU resources, use the 95th percentile of 1-minute resolution as a guideline to set the CPU limit of your pods.
  • Keep an eye out for CPU throttling and see if it goes up with your CPU utilization. You may need to use a higher resolution metric, i.e. 10-second data points, to set the right CPU limit value for pods.

Quick tips for setting pod memory limits for your team:

  • Monitor memory usage at a relatively high resolution, i.e. 1 or 10-second data points for extended periods of time. 
  • Always use the maximum value as your guideline. 
  • Make sure that maximum value of high-resolution metrics, i.e. 10-seconds, are saved as you downsample your metrics. Use that maximum in your long term planning.
  • Re-evaluate your limit values with every major release. 

Comments and Responses

Related Articles

Team Productivity: Resource Management

Since the introduction of containers, the method of building and running applications in an organization has

Read more
Capacity Management for Teams on Kubernetes: Setting Pod CPU and Memory Limits

Capacity management is a complex, ever-moving target, for teams on any infrastructure, whether on-prem,

Read more
Kubernetes cost saving K8s
Kubernetes Cost Optimization with Magalix

Is our Spending Getting Worse? I woke up one day to see this email from our CEO in my mailbox. I knew this

Read more

start your 14-day free trial today!

Automate your Kubernetes cluster optimization in minutes.

Get started View Pricing
No Card Required