Mastering systemd: Securing and sandboxing applications and services

https://www.redhat.com/sysadmin/mastering-systemd

by kureikain • 6 years ago

190 71 6 years ago

This is cool. But how do I use these sandboxing if I want to run a desktop GUI application?

jchw2• 6 years ago

I think the effective answer is you don’t; this is for services. However, sandboxes for this specific purpose exists too: Flatpak, for example. If you want to sandbox programs that are part of the system installation, Firejail can provide pretty useful sandboxing. SELinux and AppArmor also can be configured directly, and some distros will come with default configuration for system apps. No one solution will ever be a panacea for either security or use cases, but together they can help build a setup with defense-in-depth.

Koshkin2• 6 years ago

this is for services

"applications and services"

tapoxi• 6 years ago

Probably Flatpak (https://flatpak.org/) which uses bubblewrap to sandbox the application.

reddotX• 6 years ago

you use snapcraft https://snapcraft.io/

ATsch• 6 years ago

Not to start a flamewar, but I'd like to point out why I think snap is not only inferior to flatpak technically but actually a threat to the linux desktop:

Snap is very deliberately centralized, with a single hard-coded repo URL. The server is also closed-source. This is because snap's somewhat transparent primary goal is to give Canonical central control of app installation across all linux distributions. The plan from there will include taking cuts of sales revenue and publishing fees. The pieces for this (like DRM) are falling into place.

Flatpak, because it isn't born out of such business a model, supports an arbitrary number of user-defined repos, which are trivial to host because they are static folders and can be installed with a single-click via a `flatpakrepo` url.

This is on top of other advantages such as upstream support from the likes of GNOME, support for sharing code between apps using "frameworks", supporting themes, using namespaces instead of modified AppArmor, p2p support, a better permission system, etc.

inshadows• 6 years ago

Snapcraft is about Canonical control over devices, the sandbox is just a mean to an end. Snapcraft leaves you with no control over which code runs on your device.

pengaru• 6 years ago

It takes some fussing about but you can do it in more or less the obvious way. I regularly use firefox in a systemd-nspawn container, the only annoying thing at this time is needing root to start the container but there's already a plan for how to fix that and I believe someone has also posted a PR for supporting starting "machines" (in the systemd-machined sense) as a regular user.

WhatIsDukkha2• 6 years ago

For this job you probably want to use Firejail -

https://github.com/netblue30/firejail

It's actually more mature for desktop sandboxing then the other solutions because its been effective for years and has a lot of community work on sandboxing -

https://github.com/chiraag-nataraj/firejail-profiles

If you want to use a prepackaged application vs sandboxing a new app or something in your distribution you should go with -

https://flathub.org/home

Not every app is well sandboxed with flatpak but this is the goal and they are making very good progress. Also a great way to track a larger project that your distribution doesn't keep up to date as you'd like.

just to note -- flatpak/flathub use bubblewrap under the covers which is very promising but I don't see the community profiles like firejail yet.

eeZah7Ux2• 6 years ago

Use firejail. It comes with hundreds of ready-made profiles for userspace applications.

Avoid snap, flatpack and so on as most of them are not really security-oriented.

dbmueller• 6 years ago

(sorry for the piggyback)

Something I'd like to see is a simple way to define “containers” on my desktop that would allow me to run sandboxed versions of my standard apps in bundles.

The plan would be something like the following.

You have a simple gui that would allow you to create new containers, for which you could define what it has access to (specific folders, internet, sound, etc). You could then add apps to your container, and they would only be able to play with each other in the container with the restrictions given. I think that would work with a simple app essentially based on bubblewrap. For example:

* A "torrents" container, where the only apps would be firefox, deluge and vlc, and access to no folder in my home directory, but the container would have its own home directory.

* An "admin" container, with only firefox and thunderbird and libreoffice, say, and access to my ~/Downloads and ~/Documents folders.

You should be able to run the admin.firefox and torrents.firefox side by side, since they'd have different profiles. By default, each container would have its own "virtual filesystem" with no accesss to anything outside (modulo what's really needed), and only by toggling "links" would it be able to access your actual fs tree. The GUI would be easy enough for computer "illiterate" people to work with it. And the GUI would be smart enough to create desktop files with each new application I add to a container, with customized icons.

I don't expect it would be too complicated (essentially bookkeeping on top of bubblewrap). If anyone is interested, I'd happily discuss it more!

WhatIsDukkha2• 6 years ago

Look into firejail, see my sibling comment.

It does have a gui similar in spirit to what you are asking for but take a look and see if you can pitch in and help.

m23khan• 6 years ago

Nice - wondering if these are applicable to centOS as well given that centOS is touted as the freeware version of RHEL.

akeck• 6 years ago

I expect that's the case. For the most part, CentOS has feature parity with RHEL.

nightfly2• 6 years ago

CentOS is RHEL without the RHEL branding.

papercrane• 6 years ago

Yes, these mostly look to be features in mainline systemd and should work in any distro using systemd (assuming that version of systemd has these options.)

_0w8t2• 6 years ago

I use those features in Debian 10. Even Debian 9 supports most of them.

_0w8t2• 6 years ago

This sandboxing for services provides similar isolation as various container runtimes. Plus due to integration with systemd things like live update without dropping a single connection is possible to implement with straightforward application code.

thu21116• 6 years ago

If I understand Docker correctly, it's not actually intended to be a sandbox and wasn't designed as such (e.g. the daemon runs as root, or at least used to). It's not clear to me what the threat model for running untrusted Docker images is, or how you'd know what the expected set of permissions were except by reading a README.

Whereas this feature is explicitly a sandboxing feature, and the needed permissions are enumerated by the service file.

colechristensen2• 6 years ago

Cgroups limit the impact anything inside the container can do to anything outside the container.

It doesn't matter that the daemon runs as root, it starts processes in an a way that prevents them from interacting with other daemons, filesystems, etc. resources.

You don't quite understand docker correctly :)

speedgoose• 6 years ago

A Docker container cannot contact the daemon as far as I know. Unless you bind it, but then you know about the risks.

hiram1123• 6 years ago

Not that it's exactly relevant to this article, but on RHEL 8, at least, Docker isn't supported, and instead they use their own container runtime called Podman along with Buildah for building them.

Podman does not run as root, and thus neither do the containers.

I tested it out on my development backup laptop; I usually use Docker-CE on my main MBP. Podman and Buildah were able to deal with all my individual containers, but their replacement for Docker-Compose failed on all my compose environments, and the errors were not helpful. I ended up installing an unsupported version of Docker-CE, and everything worked fine.

* Podman https://podman.io/

* Buildah: https://github.com/containers/buildah

* Podman-Compose: https://github.com/containers/podman-compose

dirtydroog• 6 years ago

Are you suggesting that a system's services should be run in individual docker containers?

nightfly2• 6 years ago

No, they're saying that systemd provides many of the same benefits that running things in Docker provides.

MertsA2• 6 years ago

No, he's saying that isolation via systemd is basically the same kind of isolation that you get with runtimes like docker. It's just Linux namespaces, which conventional wisdom is to assume that it's a relatively modest security boundary at best. The key takeaway here is that you can get this added security with minimal to no performance impact in a way that's simple and straightforward for the sysadmin to configure.

eeZah7Ux2• 6 years ago

It's way better than container runtimes. It's proper security sandboxing and it comes by default on most Linux distributions.

arianvanp• 6 years ago

it's the same kernel features. So no it's identical.

thu21116• 6 years ago

I have to admit, having recently spent some time learning it and updating my Linux skills I really don't understand why systemd seems to have provoked so much hate and controversy in the Linux world. So far I really like it. It makes a Linux server feel more like something designed rather than incrementally patched together out of hacky shell scripts written 20 years ago.

The unit config files are small and simple. Every line makes sense. Things are coherent - learning how to start a service at bootup means you've partly learned how to configure the new equivalent of cron jobs. The same configuration works at the system and per-user level. You can quickly locate logs. I didn't quite like the command line interface at first e.g. why "systemctl" and not "service", plus I always forget if it's the service name in the second or first position. But the basic functionality is all there.

I realise Docker is all the rage at the moment but I've found it kind of flaky and complicated. For my own purposes systemd feels about the right level of abstraction. It's not picky about where software comes from, it just manages it. You can unzip a tarball and make it run isolated, depend on other services, and a whole lot of other neat things. If you've learned it on one distro you learned it for the rest, unlike SysV init. And it seems to constantly get more useful features that are all pretty easy to configure as well.

traverseda• 6 years ago

having recently spent some time learning it and updating my Linux skills

It has gotten better over time. If you're just updating it now you're probably not having nearly as rough a time of it as you would back when it was first being pushed. The documentation was pretty terrible and things would break randomly a fair bit. I've had multiple occasions where my networking just stopped working due to systemd components.

One thing I still find annoying is how "enterprise-ey" it is. There's no "put all your unit files here" easy to look at folder, you need to first use some systemd specific command to find out exactly which of the many many sub folders the unit file lives in. There's also a tendency to split configuration across many different files, which I guess is how we always did it but somehow seems more annoying when coupled with the whole "you don't get to know where the unit file is" system. Like one service could have 3 or 4 different unit files if it's complicated, if it relies on socket activation or mount points or times or anything like that.

One of my biggest (practical) complaints these days is simply that everything is spread out all over the filesystem.

thu21116• 6 years ago

Yeah that's true. I don't really understand yet why timers and sockets are in separate files to services.

_gjrn• 6 years ago

You can read up on https://suckless.org/sucks/systemd/

It is a good primer into why systemd provoke(d|s) so much hate and controversy.

qqssccfftt• 6 years ago

What PID 1 Should Do
When your system boots up the kernel is executing a given binary in its known namespace. To see what are the only tasks the application running as pid 1 has to do, see sinit. Just wait for child process to reap and run some other init scripts.

Very objective and balanced article!

Koshkin2• 6 years ago

sinit looks like another extreme: systemd does (almost) everything, sinit does (almost) nothing. I think a good design is somewhere in between.

guenthert• 6 years ago

You didn't have to debug a system which failed to boot.

benchaney• 6 years ago

This. While systemd has a lot of issues that are obvious from a distance, everyone who hates it has had it break something because of a really stupid design decision it made.

castillar763• 6 years ago

Purely my perspective here, as a 25-year user of Linux and BSD (for context). On the one hand, I very much agree with you that Systemd brings a lot to the table. The files are much easier to work with, the service ordering and integration is logical and works well (to the extent I've beaten on it), and I can't deny that a faster boot sequence is helpful for things that boot often like container images. It's been a lot easier writing systemd config files for my new services than it was to write init.d boot scripts, for sure, and the integration of systemctl is really nice: one command does all the service things from info to disabling.

The flip-side for me, the one that continues to get under my skin, is the approach of the systemd project. It's the habit systemd has of simply subsuming all other system functions into itself (DHCP client? Sure! DNS client? Mine now! Logging? We handle that now. Firewall control? ALL SHALL BE ASSIMILATED.). If the systemd versions of those functions were both obvious in presence and easy to replace with more fully functional replacements when I wanted, I'd feel better about it, but I keep running across cases where system functions that have worked that way since Linus was in knee-pants have suddenly been replaced behind the scenes with a systemd module whose configuration files and knobs aren't obvious or well-documented, it's difficult-to-impossible to uninstall, and trying to disable and replace it leads to cascading failures.

Worse, I keep seeing security issues brought up to the systemd devs and then tossed aside with "well, just don't do that" or "how is that even a problem". It's not pervasive or constant, but it's steady enough to be worrying. Obviously not every security issue raised will be top priority, but it concerns me how much of my systems are being subsumed by a project that seems to prioritize "do all the things now" over "do things securely".

Compare that to the approach taken by, say, OpenBSD, which has also been steadily replacing long-standing system bits with their own custom-developed pieces. Their approach has been "we will provide basic functionality that is iron-clad secure", while leaving you the ability to swap in something else for stuff like OpenSMTPd without breaking your system. And yes, Theo can be just as much an <unprintable> as Poettering, but I'm a lot less worried about the outputs of his work for the above reasons.

Ultimately I think systemd is a good way forward, but it needs someone else to take over the project, rein it in, and keep it focused on being good at what it does rather than trying to be all things everywhere. Or, alternately, it needs to just implement its own kernel and go off to be SystemdOS v1, which seems to be the trajectory it's on right now.

dTal• 6 years ago

Worse, I keep seeing security issues brought up to the systemd devs and then tossed aside with "well, just don't do that" or "how is that even a problem". It's not pervasive or constant, but it's steady enough to be worrying. Obviously not every security issue raised will be top priority, but it concerns me how much of my systems are being subsumed by a project that seems to prioritize "do all the things now" over "do things securely".

I would be even harsher than that. It's not just security issues that earn "don't do that" from systemd devs - it's everything that doesn't fit their narrowly imagined use cases. You don't even get "do all the things now" - you just get "do this particular thing now". Generally with no regard for POSIX. And if you want the old behaviour back, expect to boil the oceans. Exhibit A: https://news.ycombinator.com/item?id=19023885

jcelerier2• 6 years ago

The flip-side for me, the one that continues to get under my skin, is the approach of the systemd project. It's the habit systemd has of simply subsuming all other system functions into itself (DHCP client? Sure! DNS client? Mine now! Logging? We handle that now. Firewall control? ALL SHALL BE ASSIMILATED.).

well, it's in the name. systemd is a system daemon. system is... well, everything that isn't your text editor and music player mostly.

MisterTea• 6 years ago

ALL SHALL BE ASSIMILATED ...

People forget that Systemd was crafted by Lennart Poettering and Kay Sievers while working for Red Hat. Red Hat is a multibillion dollar company who sells "Linux solutions". If their solution is the standard, then they control the market. This is lifted straight from the Microsoft 101 handbook: Embrace. Extend. Extinguish (going back to DOS ain't done till 123 wont run).

Or, alternately, it needs to just implement its own kernel and go off to be SystemdOS v1, which seems to be the trajectory it's on right now.

SystemdOS already exists: Red Hat Enterprise Linux. That's the gold standard. All other penguins shall follow. This is the extinguish phase.

The hate for Poettering is misplaced Red Hat hate. He's just a hired gun with terrible ideas along with his sycophants. How many blog posts did those creeps write boasting of how it will help the desktop community which Red Hat doesn't give a damn about it. Red Hat had them lie about that to push it through to further their control over the Linux Server. Its enterprise grade bullshit.

hiram1123• 6 years ago

I don't really mind Systemd as a replacement for init, it's just that other unrelated systems are now requiring it explicitly that probably shouldn't (e.g. Gnome) and also Systemd is creating more and more sub projects that replace even more standard Unix functionality. Let's see:

1. Systemd-network is a whole replacement for Network Manager, which itself was bloated compared to old ifcfg files and other simple tools that manage just single aspects of the network (dhcp, wifi, etc.)

2. Systemd-resolved - replaces everything that deals with DNS. This really bothered me when it was discovered that it was ignoring certain standard configurations such that users using VPNs / encrypted DNS were leaking DNS lookups. That could get someone tossed in prison or worse in countries like Iran or China.

* Systemd-logind - replaces various tools used for logging in to both X and the console. Seems like it's a mixed bag, though I know there was a lot of complaints because it broke standard behavior with tools like nohup.

* Systemd-boot - replaces Grub and other similar tools. I hate Grub, so this doesn't really bother me.

* Systemd-locale, hostname, timedate - these replace traditional tools dealing with the obvious features their names describe. This isn't a big deal to me as it's just replacing one simple tool with another, though I'm sure the dependencies will end up requiring a lot more unrelated systemd libs, which the original tools were completely stand -alone.

Seems like more and more are coming out every year - encrypting and exporting home directories, partition managers, Cron, etc.

Even if the replacements libs are better than the originals, it seems the developers purposefully make sure that the more of these projects your distro depends on, the more you're locked into all the other Systemd projects.

This will be the end of BSDs and any other Linux distros that don't want to use any Systemd, or even a few of the smaller tools, because everything will eventually require the DBus and Systemd / Journald main libs.

I'm not a Linux expert at this level, but it certainly reminds me of a few notorious Java libraries - you import a single library to use just a few methods, and all of the sudden your app has grown by 100MB of other dependencies that have no relation to your requirements, and it breaks everything because it adds all sorts of Servlet libraries that override and break things everywhere else. Usually the only way to keep things clean is to NOT use these libs or pull out just the source for the things you want.

Interestingly, the libraries that have caused me the most trouble over the years were created by Redhat - you end up with all sorts of JBoss specific modules that, for no apparent good reason, override your logging, validation, web, etc. libs with their own versions.

thu21116• 6 years ago

Ah yes, Systemd-resolved puzzled me too. From what I could tell this is actually the fault of Golang. Historically Linux implemented DNS control using config files read by the C library. On Linux for not really good reasons the C library is perceived as an optional gateway to the kernel rather than the primary OS interface as it is on other operating systems like macOS or Windows, so statically linked programs or programs that don't use the C library ended up bypassing your configured DNS resolvers.

Arguably they shouldn't have done that and directly invoking syscalls is a bad idea, but Go does it and Linus encourages it, so to make DNS configurable again means making the DNS resolver on the network card a userspace daemon that can then apply/reapply the configuration.

I read the nohup thread linked to elsewhere in this discussion. Poettering doesn't seem like the best communicator but his point was that systemd had to choose between a security-oriented default and a compatibility-oriented default, and they chose security. I can see that the security argument here is rather theoretical (how many systems got hacked over the years because of nohup?!) and the compatibility argument is real, also Poettering's belief that distros are either security or compatibility focussed and would know about this setting is a bit off. But this isn't really a systemd specific issue: macOS and Windows have also upset devs and users a lot with security-oriented backwards incompatible changes.

na85• 6 years ago

In the early days of systemd it broke systems on the regular, and instead of humility, Lennart Poettering and his homies tended to react with ego and arrogance.

jcranmer2• 6 years ago

I really don't understand why systemd seems to have provoked so much hate and controversy in the Linux world

A few major reasons:

* It's originally written by the person who also wrote Pulseaudio, so any vitriol people have left over from Pulseaudio's configuration issues gets poured into systemd as well.

* The project takes a monolithic approach to the problem of configuring services, and many people think that's wrong. [1]

* It's change, and change means work for people who have their old setups.

[1] This criticism is usually described as "it breaks the Unix model," a formulation I don't like because it's meaningless and an appeal to authority instead of articulating the issues they have with it. Watch how people try to explain how the similarly monolithic Linux kernel doesn't break the Unix model, it can be funny sometimes.

kyuudou2• 6 years ago

explain how the similarly monolithic Linux kernel doesn't break the Unix model

It does one thing and one thing well: talk to hardware.

thu21116• 6 years ago

Right, what does monolithic approach to configuring services even mean? I seem to have hundreds of systemd services on my box, all of which appear to do a tiny part of system startup. That doesn't come across as very monolithic, if anything service overload is one of the things I dislike about systemd (but it wasn't better previously).

LockAndLol• 6 years ago

What I'd like to know is how whether it's possible run GUI applications in their own containers. From what I understand about X, if a GUI app runs in the same context as the DE, it will have access to all other windows, the clipboard, etc.

That makes me think that Xephyr is mandatory in order to run an app in a container, but I haven't found a satisfactorily easy way to do so. Would systemd be the easy solution I'm looking for?

mook• 6 years ago

Firejail, mentioned elsewhere in the thread, should do that correctly. Personally though I've been doing docker (substitute with systemd-nspawn or whatever you like) with xpra; not sure it's as secure, but it should block accidental snooping while still supporting clipboard transfers.

It appears that the relevant developers are pushing towards using Wayland for more secure remote windowing, but I do not know what state it's in.

proactivesvcs2• 6 years ago

The official documentation on these directives was of great value when I started looking into unit file hardening. There were a few minor cases where I had to, or felt the need to go elsewhere for deeper explanation, but for the most part it was readable and comprehensive.

I was able to understand the changes that I made and while carefully testing, few unexpected problems resulted.

The changes that I applied as a result of this has meant the unit files now score around 1.5 from the "systemctl-analyze security" system. Considering I approached the process with almost no knowledge of systemd, this speaks volumes on the quality of the documentation, its timeliness and practical relevance, and the fruit that can be borne of excellent documentation.

lovelettr• 6 years ago

Was that the freedesktop.org wiki?

proactivesvcs2• 6 years ago

Primarly the systemd.exec man page at https://www.freedesktop.org/software/systemd/man/systemd.exe... , as well as the accompanying unit and service file man pages for more general reference.

Mastering systemd: Securing and sandboxing applications and services

Related Stories