Understanding resource limits in kubernetes: memory

Originally published at https://medium.com/@betz.mark/understanding-resource-limits-in-kubernetes-memory-6b41e9a955f9

When I started working with kubernetes at scale I began encountering something that didn’t happen when I was just running experiments on it: occasionally a pod would get stuck in pending status because no node had sufficient cpu or ram available to run it. You can’t add cpu or ram to a node, so how do you un-stick the pod? The simplest fix is to add another node, and I admit resorting to this answer more than once. Eventually it became clear that this strategy fails to leverage one of kubernetes greatest strengths: its ability to efficiently utilize compute resources. The real problem in many of these cases was not that the nodes were too small, but that we had not accurately specified resource limits for the pods.

Resource limits are the operating parameters that you provide to kubernetes that tell it two critical things about your workload: what resources it requires to run properly; and the maximum resources it is allowed to consume. The first is a critical input to the scheduler that enables it to choose the right node on which to run the pod. The second is important to the kubelet, the daemon on each node that is responsible for pod health. While most readers of these posts probably have at least a basic familiarity with the concept of resources and limits, there is a lot of interesting detail under the hood. In this two-part series I’m going to first look closely at memory limits, and then follow up with a second post on cpu limits.

Continue reading

Understanding resource limits in kubernetes: cpu time

Originally published at https://medium.com/@betz.mark/understanding-resource-limits-in-kubernetes-cpu-time-9eff74d3161b

In the first post of this two-part series on resource limits in kubernetes I discussed how the ResourceRequirements object was used to set memory limits on containers in a pod, and how those limits were implemented by the container runtime and linux control groups. I also talked about the difference between requests, used to inform the scheduler of a pod’s requirements at schedule time, and limits, used to assist the kernel in enforcing usage constraints when the host system is under memory pressure. In this post I want to continue by looking in detail at cpu time requests and limits. Having read the first post is not a prerequisite to getting value from this one, but I encourage you to read them both at some point to get a complete picture of the controls available to engineers and cluster administrators.

CPU limits

As I mentioned in the first post cpu limits are more complicated than memory limits, for reasons that will become clear below. The good news is that cpu limits are controlled by the same cgroups mechanism that we just looked at, so all the same ideas and tools for introspection apply, and we can just focus on the differences. Let’s start by adding cpu limits back into the example resources object that we looked at last time:

Continue reading

Is kubernetes too complicated?

Originally published at https://medium.com/@betz.mark/is-kubernetes-too-complicated-1ffd8a3a6fb4

post by Caleb Doxsey a week or so back generated a fair bit of discussion when it was shared on hacker news. In it he talked about deploying a small project on kubernetes and demonstrated some of the techniques he used. The resulting comment thread highlighted a debate that is currently common in certain software communities, i.e. that kubernetes is either the awesome-est sauce ever and you should use it for everything, or it’s a massive pile of overkill for all but the largest organizations. The perspective of the commenter, either pro or con, is often likely to be further extended to containers themselves. Docker is awesome… docker is a crutch for people who can’t even chroot, etc.

In this post I want to talk about why the truth is, unsurprisingly, somewhere in the middle, but more than that I want to explain why I think containers and orchestration are a fundamental shift in the level of abstraction at which we interact with compute resources. Many of the most significant changes in the way we practice software development, deployment and operations over the last 50 years have been changes in the level of abstraction of the interface between us and the things we’re working with. Abstractions are incredibly potent and useful tools for grappling with (seemingly) unbounded complexity, but when good new ones come along they are not always welcomed with open arms. I think there are some fundamental reasons for that.

Continue reading