Systemd service sandboxing and security hardening (2020)

https://www.ctrl.blog/entry/systemd-service-hardening.html

by capableweb • 5 years ago

293 78 5 years ago

max002• 5 years ago

Great article :) thank you!

kenniskrag• 5 years ago

if you want to test these settings I can recommend `sudo systemd-run -p "DynamicUser=yes" -p "ProtectSystem=yes" -p "ProtectHome=yes" --shell` but be in a readable directory like /tmp or you receive an error.

5e92cb50239222b8• 5 years ago

This is a very handy command in day-to-day work, actually. For example, I use to limit the total amount of memory available to an application, including page cache:

  $ systemd-run --user --scope --property=MemoryHigh=1G qbittorrent

It works just as you'd expect — if qbittorrent's working set goes above 1024 MiB, it pushes the least recently used page out of the page cache. Doesn't really have any effects on upload or download speeds, while helping to keep more useful data in memory.

Many isolation flags are not available in `systemd-run --user`, though, so if you'd like to have some protection you either have to combine `sudo systemd-run` with `su -c`, or wrap the command in firejail.

https://github.com/netblue30/firejail/

wmanley• 5 years ago

I have a bash alias for `make` and `ninja` to do something similar. Just having all the spawned processes in a cgroup helps with system interactivity while building. This works because the kernel will then schedule the whole build as a single unit against the other work on the system, rather than scheduling each process that the build spawns against every other process that I'm running.

t0astbread2• 5 years ago

Interesting, a few months ago I tried using systemd-run to implement unprivileged memory limits for a process and I'm pretty sure it didn't work with the user manager. Is this a recent addition? (I'm not sure what version of systemd I had at the time.)

pram• 5 years ago

Ooh, is this a good way to sandbox execs like ImageMagick or stuff like that?

5e92cb50239222b8• 5 years ago

Use firejail, it's a "one click" solution with prepackaged profiles.

https://github.com/netblue30/firejail/

It uses the same kernel knobs as systemd does, but is more user-friendly and has more features.

I use it for every application that handles data received from other machines: books, images, documents, whatever.

the84724• 5 years ago

Alas, no whitelisting option. A service should start in an empty filesystem root without network access - and if we had something as convenient as pledge() also without any allowed syscalls - and then you could only add what is needed.

firejail does this a bit better but it also started out with a blacklist approach and it's more geared towards desktop application use, not system services.

Icathian• 5 years ago

One of my favorite podcasts, Risky Business[0] regularly plugs Airlock[1]. They seem like they might be the one out front, at least as a paid service.

[0] https://risky.biz/netcasts/risky-business/ [1] https://www.airlockdigital.com/

5e92cb50239222b8• 5 years ago

What's the problem with firejail? Start with an empty profile, blacklist everything, and whitelist only the stuff you need. It works just fine for server applications, and unlike systemd isolation flags you can setup a proper separate firewall with the `netfilter` option.

the84724• 5 years ago

blacklist everything,

That isn't a whitelist approach.

Seirdy• 5 years ago

Firejail has had multiple sandbox escape vulns in the past. Firejail is an SUID executable in which sandbox escapes can lead to privilege escalation. In contrast, Systemd allows you to run services as unprivileged users, and even create users on demand.

Systemd also supports firewalling: it supports IP address allow/deny policies, ports, etc. For more advanced firewall policies you're probably better off using an actual firewall daemon like firewalld or ufw.

Someone2• 5 years ago

pledge is excellent, but it protects programmers against writing security bugs that have large impact, it doesn’t protect you against the software they write. It’s those programmers who restrict what their tools can do, and who decide when to throw the switch to enable those restrictions.

If you trust those programmers, it’s indeed way more convenient than other tools, if only because it removes the need for configuring things twice. For example, instead of configuring your web server to serve files from /foo/bar/ _and_ telling SELinux that your web server is allowed to read from /foo/bar, you only configure the web server, and it will tell the OS “I shouldn’t read from anything but /foo/bar, starting … now”.

You’ll have to trust the web server to do that, though.

the84724• 5 years ago

That's what it is intended for. But pledge has nice properties beyond that which are also useful for external sandboxing. Such as defining easy to understand syscall groups maintained by the kernel as new syscalls are introduced. If linux had that we could for example grant stdio+rpath and not worry about the kernel introducing preadv3 and programs compiled with that getting broken or suboptimal performance when isolated and it would automatically apply to equivalent io_uring implementations block equivalent SQEs too.

nwmcsween2• 5 years ago

The problem of using all-encompassing filters to secure applications is they are crap. Take for example something using Bernstein chaining where permissions are properly separated for each process, with pledge you can restrict access, with global filters it still allows the highest privilege.

5e92cb50239222b8• 5 years ago

This is a pretty lax policy IMHO, you can go much farther. These days I usually start with this, it's much more strict:

https://news.ycombinator.com/item?id=29976096

Or simply follow whatever `systemd-analyze security` recommends, just make sure you run it on a system with recent systemd.

westurner2• 5 years ago

Which distro has the best out-of-the-box output for:?

  systemd-analyze security

Is there a tool like `audit2allow` for systemd units? selinux/python/audit2allow/audit2allow: https://github.com/SELinuxProject/selinux/blob/master/python...

https://stopdisablingselinux.com/

5e92cb50239222b8• 5 years ago

Which distro has the best out-of-the-box output

I haven't seen any difference between distributions with the same systemd version. Anything with a recent one should do fine. More recent than RHEL8, mind you (which is on systemd 239): for example, a syscall allow/deny analysis is buggy there and asks you to enable some protections, and then disable them. The same unit is analyzed correctly on my desktop with v250 (I use the popular rolling release distribution).

I haven't seen anything like audit2allow. It's probably not especially necessary because of the difference in philosophies: SELinux is deny by default, while in systemd you're playing whack-a-mole anyway, and are expected to add directives one by one until the application stops working. Unit logs usually make it obvious if something was denied.

goodpoint2• 5 years ago

Debian does a lot of sandboxing.

d2wa• 5 years ago

This is a getting started/101 introduction; it also talks about and recommends systemd-analyze security. There’s a link to part two at the bottom of the article that goes deeper into things.

DyslexicAtheist• 5 years ago

any system that starts security by blacklisting instead of whitelisting tends to be doomed by upcoming changes.

HowardStark• 5 years ago

Is there any advice for working with older systemd versions? Right off the bat, systemd 237 is out because there is no security feature for that version of systemd-analyze.

5e92cb50239222b8• 5 years ago

Use the same config you'd use for the latest systemd version. It will ignore flags it doesn't know (and warn you in unit logs).

bloopernova• 5 years ago

Not meant to be a snarky comment, but a serious question: how does this differ from SELinux?

staticassertion2• 5 years ago

It's vastly simpler, for one thing. SELinux is basically a weird DSL/ programming language for describing system interactions whereas systemd is providing a very basic interface for common restrictions.

I would pretty much never ask a human being to write SELinux policies unless that was explicitly part of their job whereas I can pretty much point any developer to what systemd is providing and they'll be able to work with it.

tyingq• 5 years ago

It seems to be using mostly the linux capabilities: https://man7.org/linux/man-pages/man7/capabilities.7.html

So the overlap choice seems to be more around SELinux versus Capabilities. Where SELinux is more fine-grained and tunable, but more complicated also.

aseipp• 5 years ago

It's not just Linux capabilities; on their own Linux capabilities actually suck majorly and are very limited (AKA "crapabilities"). But systemd also makes extensive usage of cgroups and namespacing facilities to back it up e.g. preventing runaway memory/CPU quotas and stopping applications from accessing paths they shouldn't, restricting network access, stuff like that. Some of this overlaps with SELinux (e.g. restricting file access) but the mechanism is fairly different.

The overlap/comparison between capabilities, systemds features, and selinux features isn't really well defined in any meaningful way IMO. It's really like 5 different features being used in various ways.

chasil2• 5 years ago

SELinux is designed as "mandatory access control," meaning that it is not normally disabled.

The normal filesystem permissions of read/write/execute for user/group/other are among those known as "discretionary access controls," meaning that they can be relaxed.

The systemd unit security options are discretionary, at the control of the administrator.

t0astbread2• 5 years ago

Is SELinux not also in the administrator's control?

candiddevmike4• 5 years ago

These days, systemd is better/easier to sandbox _services_ than SELinux. SELinux/AppArmor is still the best way to protect individual GUI and user apps (anything not ran from systemd basically).

mbakke• 5 years ago

I don't have much experience with SELinux, but at least in my org the base policy is to run anything started interactively by the user (or root) in unconfined_t, i.e. with protections disabled.

That is, the same command that gets denied by SELinux through systemd will run fine (and unprotected) when started from a shell.

Do you write your own policies for individual end-user programs?

p_l3• 5 years ago

Easier, maybe. Better, nope. The breadth and detail available just don't compare, and not in the way where systemd can even touch the scope available to SElinux

kaba0• 5 years ago

Why not use both? They are not complementary.

cpuguy83• 5 years ago

They are completely different things, and where available should be used together.

SELinux is a policy system where policy is enforced via labels.

Labels are applied to processes which classify what the process is.

Labels are applied to files which define the what classification of process can access the file.

The application of labels happens automatically based on policy. Such policy would include the location of the file or the label of the parent process.

As an example, the default policy for httpd would prevent httpd from accessing /etc/passwd even though the process is running as (or can be) the root user. I believe you could also do interesting things like prevent httpd from opening a socket on a non-standard port if you wanted to.

SELinux is very powerful but complicated. Ideally you use this with distro packages which should have policies already configured for you.

Critically it is not one vs the other. Use both if you have it.

loudtieblahblah• 5 years ago

Does your box still touch local dns before connecting to VPN? No?

Then anything with systemd and security can stuff it.

getcrunk• 5 years ago

What?

ape43• 5 years ago

Apache needs to start as `root` but then drops to an non-privileged user. systemd's `User=<user>` can't really express that. Perhaps an option that says a unit needs to be root until the first fork when it has to be a specified user. `ForkUser=apache`

candiddevmike4• 5 years ago

It only needs to root to bind to privileged ports I believe. You should be able to use a non-root user and give it CAP_NET_BIND_SERVICE:

[Service]

AmbientCapabilities=CAP_NET_BIND_SERVICE

ape43• 5 years ago

Cool! But then I suppose the forked processes could then bind to a low numbered port - something they can't do now. So Apache would have to make sure to revoke that capability when forking.

5e92cb50239222b8• 5 years ago

I don't know about httpd specifically, but many applications want root only to be able to bind to a privileged port (like :80). This can be circumvented in one of a few ways:

1. add this to .service

  AmbientCapabilities=CAP_NET_BIND_SERVICE

2. or listen on :8080 and use NAT:

  iptables -t nat -I OUTPUT -p tcp -o lo --dport 80 -j REDIRECT --to-ports 8080

3. or make the port unprivileged

  sysctl -w net.ipv4.ip_unprivileged_port_start=80

It may work for httpd too, I haven't tested it.

Un1corn• 5 years ago

The correct Systemd solution would be to create a socket unit but your solutions works without modifying the service code

marcosdumay• 5 years ago

Apache also uses the start user to read stuff like TLS private keys, that its normal user does not have access to.

_0w8t2• 5 years ago

Some applications like nginx or php-fpm want to run worker processes under a different user account to isolate them from the main process. In those cases user namespaces are the best option. Too bad PrivateUsers= in systemd unit just does not cover this case.

nwmcsween2• 5 years ago

These are workarounds and not good practices

staticassertion2• 5 years ago

This is one of the main problems with "whole program sandboxes". Many times a program only needs permissions right at the start and then never again. From the outside though there's no way to signal "OK, I'm done, lock me down" for most sandboxing systems.

One approach that may work with systemd is to have two processes. One would be a broker, running as root. It would grab a port, for example. The other process would be spawned by the broker as a limited service and inherit that port from the parent, with no permissions of its own to open it, only to inherit.

IDK how to express that in systemd-land though. At that point you might be better off just writing the code to sandbox things yourself.

eliaspro2• 5 years ago

Many applications don't need to bind the port themselves but will happily accept one passed to them during process invocation.

This allows to let systemd to manage ports using socket units which will also stay up and buffer requests when restarting a service, allow service activation on demand/incoming requests or per connection service instances, e.g. for better isolation of sshd's per connection/user.

pquerna• 5 years ago

hello.

i added support for httpd to support systemd socket activation in 2013: https://svn.apache.org/viewvc?view=revision&revision=1511033

httpd can start as non-root, assuming other configurations like the access / error logs are writable by the non-root user.

a-dub• 5 years ago

can you limit outbound network access to specified masks/ports/devices on a per-service level?

egberts1• 5 years ago

Whose gonna write THE holy-grail of analyzer of many executables to determine what Linux capabilities, cgroups, and syscalls are just being referenced?

Caveat: it has to dig into ALL the linked libraries as well.

Systemd service sandboxing and security hardening (2020)

Related Stories