Rsync.net Technical Notes – Q4 2021

https://www.rsync.net/resources/notes/2021-q4-rsync.net_technotes.html

by rsync • 4 years ago

82 47 4 years ago

rsync9• 4 years ago

Happy, as always, to answer any questions or discuss any comments here.

Thanks again to Allan Jude of Klara Systems and Luke Young of bored.engineer for their very interesting comments and explanations.

(these tech notes were published in March of 2022 - they are backward looking to the events of Q4 2021)

tempnow987• 4 years ago

Site loading a bit lumpily for me for a minute there.

Otherwise interesting write-up and pruning unused features I do think helps with security generally.

pokstad3• 4 years ago

Any news on adding the mkfifo command? I requested it a while back and it sounded like it was doable but I have not heard any updates.

rsync9• 4 years ago

Thanks! mkfifo was included in this new command addition/whitelisting and we're going to write up an entire section on it next quarter.

Can you email ? I actually searched for our conversation for a while and could not find it ... bottom line is that 'mkfifo' is in place and I can't wait to craft up some fascinating remote ssh examples with it ...

liberix• 4 years ago

Hi, the rsync.net website seems to be offline right now. Is there any other link where we can find the technical notes?

rsync9• 4 years ago

site never went down but was perhaps slow for a brief moment ... try again, please ...

EvanAnderson• 4 years ago

I had trouble loading it, too, but managed to get an archive.is capture: https://archive.ph/YEzR5

benlivengood• 4 years ago

Is native encryption available on the send-capable accounts?

rsync9• 4 years ago

Yes. ZFS encryption and raw send.

cjbconnor• 4 years ago

The link isn't working for me.

ireflect• 4 years ago

Not technical per se, but I'm interested to know how the "unlimited plan for life" experiment worked out. Did many folks take you up on the offer? I didn't, but I strongly considered it (if I'm being honest I just procrastinated the decision until I realized the deadline had passed).

rsync9• 4 years ago

Shhh ... :)

It was, indeed, a very limited test promotion to some small subset of customers and I am still undecided on whether it makes sense to pursue or not.

Generally speaking, I dislike "lifetime" offers and there is some bad history with those being badly abused - specifically by cloud providers. Was it Joyent that dishonored the strongspace lifetime offers ?

I forget ...

reaperducer3• 4 years ago

Generally speaking, I dislike "lifetime" offers and there is some bad history with those being badly abused

Back when Sirius Satellite Radio started, it offered "lifetime" subscriptions. People eventually learned that Sirius meant the lifetime of the radio as long as it tied to the purchaser's account, not the human being.

Many people got Sirius with their new cars. When they started trading them in three or four years later, they learned that their "lifetime" subscription not only didn't go with them to their new car, but the buyer of their old car also didn't get the old subscription.

Today, it wouldn't be as big a deal because you can stream Sirius ($10/month), or some other radio service over the internet. But this was back before even RealPlayer, and the mass adoption of cell phones.

selykg• 4 years ago

Yes, Joyent was the one that totally screwed the pooch on that.

I had a few lifetime plans with them. One was with TextDrive, before the Joyent acquisition/merger and that was their shared hosting plan back when they were doing everything on FreeBSD. Then after the merger they did a sort of pre-VPS type plan where it was using Solaris Containers. Those were really cool little boxes.

I also had one of their Strongspace lifetime plans, and that still exists as Expandrive bought them and continues offering the lifetime plans.

Edit: oh, and for how it was ended, Joyent was purchased by Samsung (I think it was) and to get the lifetime stuff off the books they basically forced everyone onto their VPS-like solution but only for ~2 years I think it was. So everything disappeared.

amock• 4 years ago

I haven't forgotten how Joyent treated the customer who had lifetime offers and I'll never do business with Joyent again.

mkelly2• 4 years ago

I got the offer and really considered it as well! It was the fear of an acquisition that convinced me not to take the offer. I was a TextDrive customer before they were acquired by Joyent, I was a LastPass customer before they were acquired by LogMeIn, and it was CrashPlan's cancellation of CrashPlan for Home that drove me to find rsync.net.

None of those other companies are rsync.net, so it's not really fair of me to consider them when making this decision -- but it made me think it's safer to just pay you guys regularly for a good service. Worst case, I spend a little more over the life of my account, but that money is going to a company I really like, so it's not much of a downside.

tbe• 4 years ago

I have two accounts in my name, one for me personally and one for work. I got the offer on behalf of my employer, and was like; what would this even mean for this account? I'm pretty sure I would take the offer had it been for my personal account.

lechner• 4 years ago

Your technical notes really illustrate that there is a science to running good backups. Thanks so much!

anderspitman4• 4 years ago

and there is your arbitrary python code execution - which is exactly what we don't want.

The state of userspace sandboxing across all operating systems I'm aware of is dismal. As a developer it should be trivial to spawn a process with a subset of the privileges of the parent process.

Linux is better than most in terms of options, but you have to understand seccomp, namespaces, cgroups, containers, capabilities, etc etc. It's too confusing. Why can't I just say "spawn this process, limit it to reading files in this directory, writing files in this directory, this socket, max 10GB disk space, max 5% CPU, max 10% memory"? I'd even settle for just being able to limit filesystem and network access.

Even if you do manage to make it work, since the APIs require SETUID executables you might actually be opening yourself up to a bigger risk.

Someone needs to pull a WireGuard and make a small, simple API for userspace sandboxing. And please somehow miraculously make it so it is likely to be implemented by Windows.

stephenr• 4 years ago

I don’t know for sure about the memory and cpu rationing but doesn’t systemd expose the rest of those limits for service units?

anderspitman4• 4 years ago

Not sure. Even if it does, it's going to be built using the APIs I already mentioned, and therefore not available in userspace. It doesn't make any sense for a process to need special privileges in order to limit a child process to strictly lesser privileges.

dogleash• 4 years ago

Someone

Why not you?

anderspitman4• 4 years ago

Because I'm not a security expert. Gaining the knowledge necessary to do it well would probably require a specialization change and years of prep, which would be an inefficient division of labor. Plus, it's an interesting problem, but not one I'm particularly drawn to. I'm currently focused on solving other problems[0] higher up the stack.

[0] https://boringproxy.io/

wahern• 4 years ago

The state of userspace sandboxing across all operating systems I'm aware of is dismal. As a developer it should be trivial to spawn a process with a subset of the privileges of the parent process.

It's a breeze on OpenBSD, at least as an application developer. The key is realizing that you can't effectively sandbox a process without the process cooperating. That often means refactoring the source code to reorganize how and when it acquires resources. If you're refactoring the source code, then in many cases it's easier and most effective for the process to sandbox itself, thus pledge(2) (https://man.openbsd.org/pledge.2) and unveil(2) (https://man.openbsd.org/unveil.2). Afterall, the process and its authors are the final and best arbiters of the resources it will need to initialize and acquire during runtime.

All the pain on Linux comes from people struggling to sandbox uncooperative processes (i.e. processes you can't modify and refactor) by trying to construct a fake environment. That's ultimately a losing battle. And quite ironic that people instinctively seek that route even in the context of open source applications where source code is readily available and upstreaming patches and improvements extremely common. OpenBSD has managed to sandbox to some extent almost every program, including command-line utilities, that ship with the OS. And they continue to incrementally improve the those sandboxes over time--IOW, pledge and unveil are amenable to iterative, steady development of capability management, rather than requiring herculean effort up front, potentially resulting brittle architectures that make feature additions excessively difficult.

While there's nothing quite as simple as pledge on Linux (seccomp is close but has several problematic caveats), something very similar to unveil was upstreamed into Linux last year: Landlock. See https://docs.kernel.org/security/landlock.html Unfortunately, AFAICT, Landlock, like seccomp, is mandatorily inherited on fork (no option to disable), which means it can be difficult to integrate into existing applications that might fork or exec other processes--they might be completely unrelated and require a privilege set that isn't a subset of the invoker. Reordering the sequence of process spawning to before the privilege drop can be impractical--effectively impossible if it means basically rewriting the entire program, for example with a read-eval loop (e.g. procmailrc). Spawning processes indirectly through, e.g., systemd or pkgexec, simply shifts the problem and can even increase exposure and bug risk. But for daemon services which are only leaves on the process tree, that's not a concern; Landlock and to a lesser extent seccomp can better approximate unveil and pledge, respectively.

anderspitman4• 4 years ago

Thanks for the info. I'll look into Landlock.

I'm totally fine changing the way I write my applications to make them more secure. I just don't feel like the APIs are there.

Personally I think capability-based security models is the best idea I've seen so far. Set up the file descriptors and network sockets your program needs at startup, then drop any other privileges. The problem is file descriptors don't cover things like directory trees AFAIK.

If performance isn't critical, what are your thoughts on using QEMU to wrap an application for security purposes? That might even work on Windows.

sporkle-feet2• 4 years ago

Not related to this technical note, but I was surprised to learn that the rsync service has scheduled downtime, like for hours at a time.

Is it possible to make your service highly available? Seems that is table stakes for other cloud storage services today.

rsync9• 4 years ago

Not sure what you mean… generally speaking we keep the storage servers up continuously - as you’d expect.

I see several with 80-100 day uptimes right now …

sporkle-feet2• 4 years ago

Then how do you explain this email sent by yourself?

    We are announcing an extended, off-peak maintenance window for your backup account.

    Beginning Saturday March 5th at 9:00a PT (UTC -8) the server will go offline for approx. 6 hours as we     perform data integrity checks and rebalancing.

    ...

helamonster• 4 years ago

However, if your workload consists of one or more zfs datasets that store, and subsequently access, many small files, you can actually store the files themselves on the device - not just the metadata.
For instance, if you have a pool named TANK and a filesystem named fs1 and you want to store all files smaller than, or equal to, 8K in the special vdev, you would run: ...

This sounds like the files themselves must be of size 8K or less in order to be allocated on the metadata vdev. But doesn't that include all individual file blocks of size 8K or less, not just entire files?

ohghiZai• 4 years ago

Now that borg 1.2 is out, are there any plans on supporting that?

lygten• 4 years ago

Rsync.net is an awesome service. Ive been using their 100GB for 18 USD offer for the last few years to host my RESTIC backups. Have not had a single problem.

Rsync.net Technical Notes – Q4 2021

Related Stories