FreeBSD-rustdate, a reimplementation of FreeBSD-update

by ggm • 2 years ago

101 41 2 years ago

toast0• 2 years ago

Thank you, thank you. I haven't reviewed this, and I may or may not use it, but wow was freebsd-update from 13.0 to 14.0 slow, especially since I run my freebsd systems off spinning drives, and most of them have src.

I did some looking at freebsd-update, and it's quite the shell script, which makes it daunting to consider changing. All the forking and whatever doesn't seem great; using data structures should be a lot simpler!

But my systems running freebsd-update spend an awful lot of time on I/O, and not so much time on cpu, as I recall. I think the general flow for a replaced file is gunzip to the update dir, then install to the destination. This means all the files are written twice, and I thought there was a third write that I can't remember at the moment. Install doesn't have an option to decompress, but it'd be super handy if it did --- then you could write the file only once to the destination directory and move it into place.

I'm never a fan of quite as many files in a directory as freebsd-update gets up to either. Given that everything is already named by hashes, it's not too hard to have 16 or 256 directories named 0-f or 00-ff that get a slice of the files. Large directories have gotten better over time, but they're still slow.

There's some ability to exploit the embarassing parallelism available too, but that's hard in a complex shell script, so some other environment makes sense. I didn't look, but hopefully that's a knob in rustdate ... Sometimes you'd want it and other times one thing at a time is better, even when it's slow.

One feature I know update can do that I want to look at more but haven't had time is the zfs boot environment stuff --- seems like with the right arguments you could do the update on an inactive environment and reboot to switch, instead of the default way which does the install on the live environment but you could reboot to switch back. I'd be much more patient if the live environment continued working and I could reboot at my convenience once the process finished. Especially if I might be able to run pkg upgrade in the boot environment before the reboot. If rustdate supports that too, that'd be neat.

deaddodo• 2 years ago

I did some looking at freebsd-update, and it's quite the shell script, which makes it daunting to consider changing. All the forking and whatever doesn't seem great; using data structures should be a lot simpler!

It doesn't look too crazy:

https://github.com/freebsd/freebsd-src/blob/main/usr.sbin/fr...

It's fairly well broken down, and pretty well organized. I think the most difficult part is that so much of it relies on shell nice-isms that are a pain to re-implement in something like C (although, I guess you could just call fetch and the like directly via 'exec' or 'system'; rather than using their libraries to manage things directly).

throw0101d• 2 years ago

It's fairly well broken down, and pretty well organized.

Not surprising given that it's written by Colin Percival, who was the Security Officer for FreeBSD for a while, and who runs/develops Tarsnap:

* https://www.tarsnap.com

* https://en.wikipedia.org/wiki/Colin_Percival

* https://news.ycombinator.com/user?id=cperciva

kstrauser2• 2 years ago

I do have to say that's about the cleanest shell script of that size I've ever seen. cperciva, if you're reading this, that's pretty.

cryptonector2• 2 years ago

Hmmm, lots of `for C in $@;` that should be `for C in "$@";` instead.

mustache_kimono• 2 years ago

Would you mind mirroring to a repo, with a license, so others might view?

stackghost2• 2 years ago

The source tarball, on the downloads page, contains a license.

ThePowerOfFuet• 2 years ago

Shouldn't have to download and unpack a tarball to see how it's licensed.

jeroenhd• 2 years ago

The license:

---

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

---

erk__• 2 years ago

It is on Launchpad here: https://launchpad.net/freebsd-rustdate

ode• 2 years ago

See https://rustdate.over-yonder.net/speed.html for the numbers.

frankjr• 2 years ago

Impressive. The current state doesn't really bother me given the frequency of system updates but I do sometimes wonder what on earth it's even doing that's taking so long.

alberth2• 2 years ago

This is freebsd-rustdate, a reimplementation of freebsd-update.
In usage, it’s expected to be similar, but not identical to freebsd-update.

Doesn't reimplementing it to not be identical, defeat the point of reimplementing it?

(since it can't be used as a drop-in replacement)

youwot0• 2 years ago

The ability to rewrite incompatible subsets of basic Unix utilities is where Rust truly shines.

jchw6• 2 years ago

At the very least, the uutils project seems to be making good progress on becoming a "true" drop-in replacement for the GNU coreutils that will pass the GNU test suites and treat compatibility issues as bugs.

Honestly though, as time goes on, I'm not sure if rewriting UNIX utilities to be memory-safe actually gets you that much. I mean, having multiple implementations of something as important as the coreutils is good: nobody complains about the co-existence of e.g. BSD and GNU utilities, and IMO there's really no reason to feel any different about RiiR rewrites. That said, I can't really figure out what attack model would really make memory safety a particular priority. In most cases you would be isolating things that require these sorts of utilities using namespaces/jails/etc, and I wonder if memory bugs in the coreutils is actually a serious attack vector.

Of course in this case (FreeBSD-rustdate) it's literally for performance reasons rather than safety, so everything sort of makes sense (not necessarily using Rust for safety per-se, but for robust threading. That does make sense.)

nmz• 2 years ago

Given that sh can print everything as it runs, it would be nice to know what is it that takes so long, instead of doing a full rewrite.

I remember once replacing a filter written in pure bash by an awk one, the slowdown was around 2 minutes v awk which was a second.

kelsey98765431• 2 years ago

edit the .sh, add `set +x` then pipe it to ts

```

$ cat > example.sh ; bash example.sh |ts

set -x

echo a

sleep 3

echo b

sleep 1

echo c

#end

+ echo a

+ sleep 3

Aug 27 09:48:02 a

+ echo b

+ sleep 1

Aug 27 09:48:05 b

+ echo c

Aug 27 09:48:06 c

```

kstrauser2• 2 years ago

You can do all that without modifying the script or using a separate tool:

  PS4='+ $(date --iso-8601)\011 ' bash -x example.sh

alberth2• 2 years ago

Suggestion: call this something else - because 'rustdate' makes me think 'rust date', which makes me believe this is date/time related.

gonzo• 2 years ago

Given the official stance of moving to pkg-base, updating freebsd-update with an incompatible rewrite in Rust seems like yak shaving.

FreeBSD-rustdate, a reimplementation of FreeBSD-update

Related Stories