Efficient service isolation on Alpine with VRFs

https://ariadne.space/2021/09/13/efficient-service-isolation-on-alpine-with-vrfs/

by zdw • 5 years ago

33 10 5 years ago

zamadatix2• 5 years ago

Alpine can do that, as far as I know, no other distribution can easily do out of the box yet: service isolation using the base networking stack itself instead of netfilter.

Most distros come with VRF support compiled in their default kernel and ship a modern enough version of iputils for this to work. Actually looking closer at the blog I remember this name, it's the person that enabled CGROUP_BPF to make this work on Alpine! https://gitlab.alpinelinux.org/alpine/aports/-/issues/9856. Big thanks as that saved some minor headache of dealing with a custom kernel while using Alpine on the VM partition of some switches in a production deployment.

I'll admit to being lazy like most and just using network namespaces out of habit as it's a catch-all fallback you don't have to think about. In the above where I actually used VRFs I matched them to the VRFs I had in the rest of the network and used namespaces to make multiple service contexts just as logical separation more than it had to be segmented as such. That being said most of the time I think VRFs are what people want for containers it just took too many years to land in the kernel and everyone became familiar with macvtap and namespaces first.

One thing I haven't looked into is I know VRFs are far more lightweight but I have never actually measured how lightweight. For half of the resource utilization story while doing some testing recently I created ~3000 network namespaces on a small box and it took just under a gigabyte of RAM. I didn't do any throughput testing though as mostly I just needed things to generate and respond to ARPs on a bunch of mapped network tunnels.

Anyways fantastic article and great choice on talking about VRFs instead of a typical firewall post!

sargun• 5 years ago

Great article from Ariadne.

Just a comment:

Two other frequently cited options are to use seccomp and network namespaces for this task. Seccomp is a natural solution to consider, but again has significant overhead, since all syscalls must be audited by the attached seccomp handler. Although seccomp handlers are eBPF programs and may be JIT compiled, the performance cost isn’t zero.

Seccomp gained the ability to fast-path syscalls in 5.12 (I think), and ones that will always end in success. The other thing is that seccomp filters are written in cBPF (although compiled to eBPF under the hood).

e12e3• 5 years ago

Alpine presently uses OpenRC as its service manager, although we plan to switch to s6-rc in the future.

Oh, that's exciting - I always liked the daemon-tools family of service managers.

hda111• 5 years ago

This makes Alpine interesting for containers. Docker makes it so hard to run systemd.

grawlinson• 5 years ago

Can’t you just install systemd in a Docker container? Can you expand on your comment? I’m interested in what you mean by it.

skratlo• 5 years ago

Can anyone share a use case or two, for this or in general for this kind of service isolation?

e12e3• 5 years ago

I'd love to see some examples, too. Would it be useful for a two-stage email setup for example, where something like postfix or opensmtpd was exposed to the internet, and a richer mail service (with more attack surface) was confined to a vfr, and only the "outer" service listen on smtp to the internet, and forward via LMTP to the "inner" service for example?

zamadatix2• 5 years ago

In general if you do it with a namespace you could probably have done it with a lightweight VRF instead. If VRFs hadn't taken nearly a decade to land in the kernel you'd probably see these much more commonly used.

Another use case from the original network reason VRFs were made would be conflicting IP space or hanging a single physical device connected to multiple disparate local networks.

oneplane• 5 years ago

I'm curious if the performance and simplicity of this isn't the same as creating a virtual interface and bridging it to your real interface and then simply setting all the rules on that virtual interface. Then your service just binds to that virtual interface and everything simply looks the same as any other virtual network.

Efficient service isolation on Alpine with VRFs

Related Stories