A Linux sysadmin's introduction to cgroups

https://www.redhat.com/sysadmin/cgroups-part-one

343 69 6 years ago

This is getting a little long in the tooth but I created a few videos and did some tests around cgroups back in 2013-2014. Both these help explain the underpinnings of Docker and I guess Kubernetes now too. Turtles all the way up. I'm just mentioning this since these are still some of my most popular videos and still the core tech under the hood.

A personal note here too. cgroups were invented at Google in the early 2000's. If you're using search today, gmail, docs, maps, etc. You're using cgroups. It sounds simple but this tech really powered a whole wave of innovation/startups/projects that almost everyone interacts with on a daily basis. Either through touching a Google service or interacting with anyone using Docker or Kubernetes (running on cgroups/namespaces). Pretty impressive.

#14 - Introduction to Linux Control Groups (Cgroups)

https://sysadmincasts.com/episodes/14-introduction-to-linux-...

#24 - Introduction to Containers on Linux using LXC

https://sysadmincasts.com/episodes/24-introduction-to-contai...

LeCow• 6 years ago

Dude, those videos are awesome!

Thank you.

caymanjim• 6 years ago

Kubernetes is just another layer on top of Docker. It can use other container execution environments, but Kubernetes itself doesn't manage the container runtime; it just orchestrates containers at a higher level. If you're running Kubernetes, you're almost always still running Docker.

WestCoastJustin5• 6 years ago

Yeah, funny how it all worked out. cgroups/namespaces powered simple LXC containers. Docker comes along and makes a nice workflow and package management system (wrapping cgroups/namespaces). Kubernetes comes along and makes a nice workflow/cluster management layer (wrapping Docker). Cloud providers come along and make a nice Kubernetes management layer (wrapping Kubernetes). Pretty crazy to see the evolution over the past few years.

This tech has completely changed the sysadmin landscape/job descriptions and sort of threw tons of gas the whole devops movement.

Disclaimer: I worked at both Docker & Google. Although not on this tech specifically. Opinions are my own here.

kbenson• 6 years ago

If you're running Kubernetes, you're almost always still running Docker.

Red Hat is trying to base all their stuff on Podman, which is Docker rewritten to by a script instead of a client/server (but can be renamed and used as a drop-in Docker replacement with the same command API). If they have their way, that statement will not longer be quite as correct.

antonvs• 6 years ago

If you're running Kubernetes, you're almost always still running Docker.

I'm not sure that's true - a lot of Kubernetes system use containerd these days.

tonetheman• 6 years ago

Man your site is GREAT. Super clear videos. Good stuff.

contingencies3• 6 years ago

cgroups were invented at Google in the early 2000's.

In my understanding, most of the initial commits (at least for LXC, which was the initial userspace) came from IBM, who funded it with interest in kernel-level resource balkanization for their largest mainframes. Google's kubernetes only appeared post-facto after docker, which itself was basically an lxc wrapper for a long time.

Source: I corresponded with two of the authors ~2009, was an early lxc user, and provided some security fixes for early docker.

WestCoastJustin5• 6 years ago

I don't believe so. "Engineers at Google (primarily Paul Menage and Rohit Seth) started the work on this feature in 2006 under the name "process containers"." [1] Here's the kernel patch from Google that was merged into the Linux Kernel from 2006 [2]. Google used this tech in their Borg cluster manager that way pre-dates Docker/Kubernetes/etc. I'm sure other folks jumped in an helped out after the fact but this was created at Google.

[1] https://en.wikipedia.org/wiki/Cgroups

[2] https://lwn.net/Articles/199643/

FedericoRazzoli2• 6 years ago

Fantastic videos! Thanks for the links!

Yes, Kubernetes uses cgroup too.

Systemd units are also based on cgroups.

tofflos• 6 years ago

These videos were absolutely amazing. Thank you.

obiwanpallav1• 6 years ago

The 4th part[1] of this tutorial series also add a sysadmincasts.com video in the `Sources` section

[1]https://www.redhat.com/sysadmin/cgroups-part-four

jeffbee4• 6 years ago

These things should strive for accuracy and this article is not accurate. The cgroups facility does not control "the number of CPU shares per process." Although you can put such a thing into effect with control groups, it's more accurate to say that a control group limits the resources of a set of tasks. Those tasks may be from one or several processes, and it's also the case that a single process can divide its own tasks into several cgroups.

waynesonfire• 6 years ago

Hey webster, whats a task?

stuff4ben• 6 years ago

While I agree that we should strive for accuracy, you're really splitting hairs there. Plus the author puts a disclaimer in there "NOTE: This is a gross simplification and is NOT technically accurate should you be looking to get involved with the underlying cgroup code. The above explanation is meant for clarity of understanding."

jeffbee4• 6 years ago

But the explanation does not contribute to clarity of understanding. This has nothing to do with hacking on cgroups itself. As a user of cgroups you should know that one group may control multiple processes, and one process may be controlled by multiple groups! There certainly is not a 1:1 correspondence.

kortilla• 6 years ago

It’s not an internal detail, it’s fundamental to understanding it from a user perspective.

flexops• 6 years ago

Found an interesting ongoing blogpost series that also tries to explain all the other low level kernel mechanism that make up Docker and other container technologies on Linux. They haven't reach the topic of control groups though.

https://www.schutzwerk.com/en/43/posts/linux_container_intro...

FedericoRazzoli2• 6 years ago

Still work in progress and the topics are not too in-depth... but it's interesting, thanks.

ape4• 6 years ago

systemd uses cgroups so nearly every Linux box is using them.

craigsmansion• 6 years ago

Ah yes, "cgroups", according to notable no-nonsense kernel hacker Al Viro:

"it's not just badly written kernel/cgroup.c - the interfaces on both sides (userland and the rest of kernel) are seriously misdesigned. As far as I'm concerned, configuring it out solves my problem nicely."

That was in 2011, so things might have improved. What remains however is that cgroups was added to the kernel, by Googlers, for easier maintenance, but with an implicit understanding that no sane person would actually make use of it to do something important.

... enter SystemD.

jeffbee4• 6 years ago

The whole google is built on cgroups v1, which seems to stand as an existence proof that these interfaces are not useless.

yjftsjthsd-h2• 6 years ago

"bad design" and "useful" can be overlapping groups

PaulDavisThe1st• 6 years ago

so when someone is using gmail on a macOS system, how are they using cgroups in any way? how is "the whole google" built on this? perhaps you means "google's web-facing servers using cgroups extensively" ?

menage3• 6 years ago

It's true that a better userspace API for cgroups might have involved something more like ioctl(), rather than a filesystem with text-based control files. But people had been trying to get such APIs into the kernel for years with no success. Which is why we tried to piggyback on something existing.

Cpusets, which had already been accepted into the kernel a year or two previously, provided the basic userspace API for cgroups. It was just expanded to support control files for more resource types. (And multiple resource hierarchies, although that's gone in cgroups v2 I think).

dianhuji1• 6 years ago

so great info

grantseltzer• 6 years ago

I wrote a blog post on cgroups a couple years ago that's still accurate and goes further into depth and gives workable examples of using them both inside and outside of containers: https://www.grant.pizza/blog/understanding-cgroups/

aschatten• 6 years ago

This is Part I from a series. Part II: https://www.redhat.com/sysadmin/cgroups-part-two Part III: https://www.redhat.com/sysadmin/cgroups-part-three Part IV: https://www.redhat.com/sysadmin/cgroups-part-four

ldng• 6 years ago

Someone managed to apply part three ? Systemd gets in the way for me.

Edit: well, at least part four, finally, properly document how to cope with systemd. And guess what, it is different than 90% of documentation on cgroups on the internet.

meatspaceping• 6 years ago

Conspiracy theory. our kubernetes clusters as a service are not really running on VMs.

musicale3• 6 years ago

I like the cgroups approach because it seems to logically build on the established idea of process groups.

I also tend to think that processes in cgroups are a sweet spot of lightweight containerization that can do quite a bit.

A Linux sysadmin's introduction to cgroups

Related Stories