Originally published at https://medium.com/@betz.mark/understanding-resource-limits-in-kubernetes-memory-6b41e9a955f9
When I started working with kubernetes at scale I began encountering something that didn’t happen when I was just running experiments on it: occasionally a pod would get stuck in pending status because no node had sufficient cpu or ram available to run it. You can’t add cpu or ram to a node, so how do you un-stick the pod? The simplest fix is to add another node, and I admit resorting to this answer more than once. Eventually it became clear that this strategy fails to leverage one of kubernetes greatest strengths: its ability to efficiently utilize compute resources. The real problem in many of these cases was not that the nodes were too small, but that we had not accurately specified resource limits for the pods.
Resource limits are the operating parameters that you provide to kubernetes that tell it two critical things about your workload: what resources it requires to run properly; and the maximum resources it is allowed to consume. The first is a critical input to the scheduler that enables it to choose the right node on which to run the pod. The second is important to the kubelet, the daemon on each node that is responsible for pod health. While most readers of these posts probably have at least a basic familiarity with the concept of resources and limits, there is a lot of interesting detail under the hood. In this two-part series I’m going to first look closely at memory limits, and then follow up with a second post on cpu limits.
