Jails on FreeBSD

https://ogris.de/howtos/freebsd-jails.html

by rodrigo975 • 3 years ago

143 114 3 years ago

jmclnx• 3 years ago

Very nice

I used jails on FreeBSD and nothing in Linux comes close. Yes, it is not a pointy-clicky setup like Linux likes to do. But IMHO Jails are far more secure, in a way: you get what you 'pay' for.

betaby4• 3 years ago

Please provide technical details of `Jails are far more secure`.

throw0101c3• 3 years ago

Multiple CVEs for escaping? E.g.:

* https://blog.aquasec.com/cve-2022-0185-linux-kernel-containe...

* https://securitylabs.datadoghq.com/articles/dirty-pipe-conta...

* https://snyk.io/learn/docker-security/top-5-vulnerabilities/

* https://www.container-security.site/attackers/container_brea...

The only escaping of jails that I've heard of in the last ~20 years is one not in the jails code, but tunnelling out through devfs:

* https://www.freebsd.org/security/advisories/FreeBSD-SA-14:07...

gtirloni3• 3 years ago

What makes Jails so far ahead of containers in your opinion?

doublerabbit4• 3 years ago

Not op, but for me:

You can setup Linux in a jail.

You can nest bhyve virtual machines in a jail.

You can assign individual NICs to a jail.

You can encrypt with ZFS a jail.

You can run browsers in a jail with full set of features.

irusensei• 3 years ago

I don’t know about the low level stuff but LXC is pretty much the same end result as FreeBSD jails. At least for my very basic needs that is.

johnisgood2• 3 years ago

What about `firejail` on Linux? Is it not much more secure than FreeBSD Jails which is just based on chroot?

bell-cot2• 3 years ago

FreeBSD Jails are conceptually just based on chroot. Their actual design and implementation are a few orders of magnitude more secure.

tptacek13• 3 years ago

Jails are not, in fact, far more secure than Linux containers. Arguably they're a better, more coherent design, but both are shared-kernel isolation schemes, and the high-order bit of their security is exactly the same. The best argument you'll come up with is that FreeBSD kernel LPEs are rarer than those of Linux, but now we're just back to the standard Linux vs. FreeBSD security debate, which never goes anywhere.

zie• 3 years ago

I agree with all of this, with perhaps one tiny addendum. Defaults matter, and last I checked Docker/container defaults are miserable for security. I'm guessing if you randomly inspected production container vs jails out in the world, way more container solutions would have giant holes in them compared to jails.

1over137• 3 years ago

Instruction are for FreeBSD 12, I wonder if it's still current for 13 and 14...?

BanazirGalbasi• 3 years ago

In my limited experience with BSD over the years (I tinker with it every year or so), a lot of the core experience remains the same. Additions to rc.conf should keep working, as well as configuration files for things like jails.

UI_at_80x242• 3 years ago

Not really.

Specifically the suggestion to use lo# instead of vnet.

st3fan• 3 years ago

I upvoted this because I have a lot of love for FreeBSD and I think Jails are great.

However, I really wish the FreeBSD folks would educate themselves a bit more about what is actually available on Linux and how those options compare to FreeBSD Jails.

doublerabbit4• 3 years ago

And vice versa. I really wish the Linux folk would educate themselves a bit more about what is actually available on FreeBSD.

tptacek13• 3 years ago

Why would they? What difference would it make to them?

thehigherlife• 3 years ago

Since this is gaining some traction. I'm curious if you are using jails and a jail manager, what are you're using now? I'm still using IOCAGE (Version 1.2) on freeBSD 13, but it seems like its not the best way to do it anymore? Documentation is all over the place on what is the best way to manage jails now.

UI_at_80x242• 3 years ago

I'm in the same boat.

Started with ezjail, switched to iocage, now thinking about bastille or roll-my-own.

kitsunesoba• 3 years ago

Still on ezjail myself. Set my NAS/home server up several years ago with a blog post tutorial centered around ezjail and have just carried that forward. Eventually will need to do a fresh reinstall though and would be interested to know what's considered best practice these days.

kevans91• 3 years ago

iocage as well... I've been trying to sucker someone into writing a new jail manager in flua, since we have that in base. It hasn't worked yet.

soupbowl2• 3 years ago

I use BastilleBSD, it has a simple template system which I really like and decent docs. https://github.com/BastilleBSD/bastille

Bastille also has a sister project 'rocinante' which allows you to use Bastille templates on the host. I converted my ansible scripts to bastille templates and it works a lot better for *ME*. I found I spend more time updating ansible scripts whenever I needed to use them, it costed more time then just using a setup.sh script, which rocianate basically is. https://github.com/BastilleBSD/rocinante

Another new kid on the block for jails is AppJail, it has some interesting features. I have not played with it enough to say how stable it is. https://github.com/DtxdF/AppJail

soupbowl2• 3 years ago

There are a lot of jail tools these days, it would be nice if we posted more modern FreeBSD guides. Myself I use BastilleBSD heavily for personal and small business use.

https://github.com/BastilleBSD/bastille

codetrotter3• 3 years ago

Let me chime in and say, Michael W Lucas has an awesome book about FreeBSD Jails. I bought it recently and I read the whole thing and it helped me a lot. I still had to figure some things out by myself because the book is for a slightly older version of FreeBSD. But it is an awesome book.

I use only the tools included in base system for setting up my jails. No “ezjail” or anything.

If you read his whole book you will see how it might be the correct choice to just do it yourself. Depending on what you want to do etc.

For me I am definitely much better off having set it up myself with the help of mwl’s book.

https://www.freebsdmall.com/cgi-bin/fm/bsdmjails

Buy the physical copy of the book.

PS: Use vnet interfaces for most of your jails.

chunk_waffle• 3 years ago

All of Michael W Lucas' books are great, they're so easy to digest.

mikece• 3 years ago

Is there a "best article" out there comparing BSD Jails to Docker and the strengths and weaknesses of choosing one or the other?

(I'm also curious if BSD Jails are the same thing as Solaris Zones but with a different name or if there is significant nuance making them different).

tptacek13• 3 years ago

Use whichever one you're more comfortable with and that suits the rest of your stack best. There's nothing Jails are going to allow you to do that Linux containers can't do. All the shared-kernel workload isolation schemes are comparably safe.

whalesalad• 3 years ago

https://blog.jessfraz.com/post/containers-zones-jails-vms/

paulddraper2• 3 years ago

For starters, Docker runs as a daemon.

__turbobrew__2• 3 years ago

For starters starters, Docker is a few layers up in the stack from jails. You would be better off comparing runc to jails or just cgroups+namespaces to jails.

jmillikin3• 3 years ago

Whenever an article about FreeBSD's jails gets posted here, the comments tend to drift toward an argument about jails vs Docker containers. People call them "Linux containers", but really they're talking about Docker (or its clones like Podman).

This may be me shouting into the void, but I wish there were an article directly comparing jails with namespaces, which is the Linux functionality that Docker uses. I can totally believe that FreeBSD jails provide a better / more unified / more secure experience than Docker, but to extend that into saying "FreeBSD jails are better than Linux namespaces" feels like a category error.

Questions I would like to see answered in that article:

* Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?

* Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?

* Can live processes be moved into or out of a jail?

* Can jails be used to make a process think it's running as a different user?

I feel like the answer to these questions is generally "no, that's not what jails are for", which is (1) a fine answer given the apparent goal of being a better chroot(), and (2) reinforces that jails and namespaces are addressing different problem domains.

throw0101c3• 3 years ago

Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?

Jails are basically like a VM-like/light: it appears like jail is its own system. Not quite as heavy as an actual VM under (e.g.) Qemu with virtualized hardware. A jail can have its own network stack:

* https://klarasystems.com/articles/virtualize-your-network-on...

You can then create a 'virtual patch cable' between the host and the jail and send one side of the 'cable' to the jail and do routing and stuff on the host:

* https://man.freebsd.org/cgi/man.cgi?epair

But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.

Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?

Yes:

* https://wiki.freebsd.org/JailResourceLimits

Can live processes be moved into or out of a jail?

No: jails are VM-like in functionality.

Can jails be used to make a process think it's running as a different user?

The jail, being VM-like, would have its own passwd.

the84723• 3 years ago

Calling "VM-like" is not helpful because containers have also been called that, it also doesn't explain anything. Is it running a separate kernal image? Does it do hypercalls to the host kernel? If not then I wouldn't call it VM-like.

The low-level details and fiddly knobs matter, not how it looks like from userspace when all is done.

Based on the jail(8) manpage I get the impression that jails are more like all the namespace and cgroup things rolled into a single entity. Which I guess makes it more difficult to use them incorrectly. But it also prevents other uses such as the one that jmillikin has in mind.

0x4574• 3 years ago

But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.

Technically, you don't have to. It's just because jails don't have convenient tooling around them (like docker or podman) it's easier to just boot it up like a normal system.

toast04• 3 years ago

But a (sub)process cannot be 'sent' to a jail: jails 'boot up' like a normal system does, and so you'd have your PID 1 run your regular daemon startup.

It's an option to run jails with a whole init and everything; then it's like a separate host. But you can also just run stuff in the jail, without a whole everything. I currently run two daemons in a vnet jail to get a separate network for those daemons, but chrooted to /, because they don't need a separate filesystem. At my last job, we ran a TLS termination proxy chrooted to a very limited directory, because we didn't trust OpenSSL after Heartbleed; the chroot had just the executable, ld-elf and the libraries it loaded, unix sockets to communicate with the origin server, and logfiles, most of which was chflags schg.

tptacek13• 3 years ago

Jails are not VM-like. The defining attribute of a VM is that it runs its own kernel. FreeBSD has a virtualization system: it's called bhyve.

bear8642• 3 years ago

> Can live processes be moved into or out of a jail?
No: jails are VM-like in functionality.

Could this be implemented though if moving between sufficiently similar operating systems?

Seems like might be useful to imprison a suspicious acting process or to release once seen to be safe.

inkyoto• 3 years ago

Jails are basically like a VM-like/light […]

With all due respect, they are not.

The definition of a «virtual machine» is a settled matter, and the «M» in «VM» is important and is the differentiator: it allows one to run a different operating system kernel on the same host under the auspices of a hardware or a software supervisor (somewhat less of a defining feature).

Neither jails nor cgroups possess such a property, and both restrict users to the same host operating system kernel and its version, so none of them are «VM-like/light» irregardless of the semantic interpretation of the «-like» suffix.

VM-like functionality is provided by a different OS subsystem in both, Linux and FreeBSD, kernels.

toast04• 3 years ago

* Can jails be used to run subprocesses in the normal filesystem, but with a different network environment (for example making a given command run its net traffic through TAP)?

Yes, you can run a vnet jail chrooted to /; same filesystem as the host, but a separate network system (you have to setup the network for the jail at this point; but I imagine the jail tools help with that). This is a fine use for jails. I currently run a vnet jail chrooted to / to do some crazy network stuff, but I just need a separate network, no other separation.

* Can jails be used to limit memory/cpu/IO/network for subprocesses? For threads within a process?

I haven't used it, but it looks like yes/maybe with the rctl subsystem. This allows limits on lots of things, include memory of several types, cpu of a few types, filesystem io in bytes per second and operations per second, number of threads. Valid subjects for resource limits are process, user, loginclass and jail. Based on the manual, this doesn't let you limit network as you wanted, or place limits on threads within a process like you wanted. If you run a vnet jail, you can potentially set networking limits in other ways, but only if you pass virtual interfaces (such as epair, or taptun) to the jail rather than physical interfaces. This is a reasonable use for jails, but you might not need a jail for this?

* Can live processes be moved into or out of a jail?

A process can move itself into a jail, but only if it's superuser. This is generally used administratively. Jailed processes can not be moved out of the jail, afaik; when the jail is destroyed, all processes within are killed. This is not within the design scope of jails.

* Can jails be used to make a process think it's running as a different user?

I don't think so, but I'd use something with LD_PRELOAD to override getuid/geteuid for this; but I think I must not understand the question. You can certainly have entirely different users inside jail vs on the host?

heavyset_go3• 3 years ago

Hate to nitpick, but a comparison to bare namespaces wouldn't be apt, either, as Docker et al use more features than just namespaces in their container implementations, like control groups, union/overlay/snapshot file systems, nftables, etc.

For several of those questions about jails, the answer would be the same if you were just evaluating the questions against namespaces by itself.

assimpleaspossi• 3 years ago

The article is about how to set up jails on FreeBSD. The article has nothing to do with anything you are talking or asking about.

vermaden• 3 years ago

First of all - the Namespaces and Cgroups on Linux are not about security. They are about features.

FreeBSD Jails provide both secure isolation from the host and also provide the separated network namespace and resources with rctl(8) when needed.

To have Linux containers separated and secured you need additional layer for that - like SELinux or AppArmor. Only then Docker/Podman/other/... Linux containers are isolated and secure.

Regards.

icedchai2• 3 years ago

You're telling those namespace / cgroup features have nothing to do with security? Not in any way, shape, or form? I'm a little skeptical.

user6723• 3 years ago

Underrated post.

craftkiller• 3 years ago

Can live processes be moved into or out of a jail?

That might be possible on DragonflyBSD. They have support for freezing a running process to disk and restoring the process again (potentially on another machine) [1], and they have support for jails, so I don't see why you wouldn't be able to freeze a running process on the host and restoring it in the jail or vice versa.

[1] https://www.dragonflybsd.org/features/#index9h2

alberth• 3 years ago

Off topic: I miss the days when web pages looked like this link.

High info density, consistent look, “responsive” without being responsive.

layer82• 3 years ago

In the aughts (2000s) I rented a "managed" VPS that was based on FreeBSD jails and an overlay filesystem. It was great in that you didn't have to maintain most of the software yourself (e.g. security updates) because it was maintained by the hoster on the base filesystem layer, while at the same time you were still root on your own filesystem overlay, where you could add or modify any file from the base filesystem. If you messed up somehow, you could simply revert to the base filesystem version of the respective files. Furthermore only the files on your overlay counted against your filesystem quota. Unfortunately the hoster discontinued that offering after a couple of years.

porker• 3 years ago

Verio VPS FTW! That was the first proper hosting I had.

codetrotter3• 3 years ago

Unfortunately the hoster discontinued that offering after a couple of years.

I’m planning on starting a FreeBSD based hosting service, but I am not sure if anyone actually wants what I have in mind.

The thing is that I have some couple of very specific ideas in mind.

I will offer a small amount of storage, and not general internet connectivity.

Users will have 25GB space and the idea is that they will be able to connect over Wireguard, but they cannot make outbound connections from the host.

It’ll be like a /home/user in the cloud.

So you can keep some files there and connect from wherever in the world and use the cli tools we all know and love. But idk if there is an actual market for that or not.

The selling point is that I will be focusing on the storage of that data. ZFS with redundancy and offsite backups – the works. That’s why the amount of data should be low. Only keep the most important files there.

Also I’m gonna accept payments in Bitcoin only, and people will have to sign up for many years upfront.

I think maybe the number of people that want this can be counted on one hand heh

gtirloni3• 3 years ago

One question you may want to answer is why anyone would use your solution vs something like rsync.net or a small VM in DigitalOcean, etc. You said the selling point would be storage. What would be the differentiator in that area? Would price be it?

Just trying to help with question to validate the idea. Best of luck.

layer82• 3 years ago

No outbound connections would likely be a deal-breaker for many.

jedberg• 3 years ago

Just one nit, FreeBSD had jails before March of 2000. I know this because I was using them in 1999. And in fact they were merged into CURRENT in early 1999 and STABLE in mid/late 1999. It was merged into RELEASE in March 2000.

erk__• 3 years ago

They were actually developed a year before, but the person who paid for development had a year of exclusive use.

http://freebsd.dk/sagas/jails/

gumballindie• 3 years ago

Every time I think of freebsd and how cool it was and is I wondering what caused linux to overtake it. I like linux as well, but I just don't understand the mechanics behind freebsd not gaining traction.

jbverschoor• 3 years ago

For me it was Java. I think there was something with mmap().

I ran FreeBSD servers with jails around ‘00. They worked fine.

Not for me, but in hindsight, maybe the FreeBSD daemon mascotte had something to do with it.

But around that time bea weblogic and ibm websphere also became popular, and I don’t think Java was officially supported. It was also a period where people tended to run oracle, but I’m not sure if that tab on Linux or something else. I think it was some custom Linux “unbreakable oracle”

chungy2• 3 years ago

AT&T v BSDi had a large hand in that outcome of fates. It put the legality of BSD code in all its places into doubt for many years.

Ironically, just a few years later, SCO thought "hey that was a good idea" and did it to Linux too.

Detrytus• 3 years ago

Offtopic, but isn't it funny that the system that calls itself FreeBSD has a concept of "jails"? Couldn't they come up with better name? :)

Jails on FreeBSD

Related Stories