NFS > FUSE: Why We Built Our Own NFS Server in Rust

https://about.xethub.com/blog/nfs-fuse-why-we-built-nfs-server-rust

by ylow • 3 years ago

141 91 3 years ago

about.xethub.com

rajatarya• 3 years ago

Repo link: https://github.com/xetdata/nfsserve

ylow13• 3 years ago

Hi! Author here. Happy to answer any questions!

ReactiveJelly2• 3 years ago

You simply need to implement the vfs::NFSFileSystem trait.

Wait, what......

.. So not only is it an easy-to-run Rust NFS server and client (I gave up on NFS some years ago because I couldn't figure out how to run Samba) but it's also meant to use as a _substitute_ for FUSE in stacks like SSHFS or an app wanting to serve debug data as files?

Awesome!

eqvinox4• 3 years ago

I don't think this includes a client, in fact the post argues that not having to implement the client because it already exists in the kernel was a major benefit of the approach…

ylow13• 3 years ago

Exactly. You build the server-side (as a FUSE alternative), and just use the OS's NFS client to mount it.

lukax4• 3 years ago

This library is amazing. Also very happy to see such a permissive license.

Btw, why did you decide to use NFS instead of WebDAV? WebDAV should be easier to implement and is also supported on all platforms.

ylow13• 3 years ago

I have not looked at WebDAV closely. I was looking for something that is already supported in all the major operating systems without additional libraries. I don't believe WebDAV is commonly supported out of the box?

eqvinox4• 3 years ago

Have you run any POSIX compliance checks against it? In particular, how are you handling assignment of inode numbers (fileid in NFS speak) to "virtual" files and making sure those stay reasonably consistent? (I vaguely remember this being a bit of a tripwire)

Also did you look at NFSv4 / what made you decide to go with NFSv3? My (superficial) impression was that NFSv4 did a lot of simplifying and throwing out legacy stuff (e.g. rpcbind), but I don't know too many details on this…

[edit: NFSv4 answered on parent / https://news.ycombinator.com/item?id=37575304 ]

FWIW I think NFSv4 would be a rewrite rather than an extension :D

ylow13• 3 years ago

I tested by uh... building a "mirrorfs" and compiling this project in it :-) (cargo does stress it pretty hard). I have not looked for POSIX compliance testers. Is there one you can point me to?

NFSv4 did throw out a lot of the legacy stuff, but also added a bunch of other state information like locks and delegation which is quite a bit more annoying to implement. Technically cos this is meant for "localhost-mounting-only", delegation should be easy to build (since we are expecting only exactly 1 mount). But v3 had far fewer APIs to implement, and so is a good starting point.

mprovost6• 3 years ago

NFSv4 is a completely different protocol and basically doesn't share anything with v2/v3. While technically it's still implemented using Sun RPC, it only has two procedures: NULL and COMPOUND. Version 3 is actually an RPC protocol, with a different procedure for each filesystem operation (READ, WRITE, CREATE, ...). One of the genius moves that Sun made with NFS was to distribute the protocol files (rpcgen) which autogenerate stubs (in C) making it easy to implement servers and clients. Nothing like that exists for v4.

hn927268192• 3 years ago

I believe you can install an NFS client on windows home if you just go to programs and features > add a feature > NFS client, but I don't have a way to check.

The fact you just need to impl one trait is amazing. I will definitely be testing this out

hn927268192• 3 years ago

You're right, there is no way to do it in windows home edition

teddyh• 3 years ago

NFSv3? Wouldn’t NFSv4 be more appropriate?

ylow13• 3 years ago

NFSv4 is quite a bit more complicated and is also stateful which makes it a bit more challenging to implement. NFSv3 is a good starting point, and is still supported on all the major platforms. Certainly, extending with a v4 implementation would be great future work.

duped• 3 years ago

You will have to throw away almost all of your code for NFSv4, it's a completely different protocol. One of the advantages of 4.x is that you don't need the portmapper or mount protocols, you just need to know the hostname/port of the server.

mprovost6• 3 years ago

Version 3 will be supported until the end of time. Version 4 was an evolutionary dead end that never took off or saw much adoption.

EdSchouten6• 3 years ago

Also keep in mind that NFSv4 is mostly adds necessary complexity. Not accidental complexity.

For example, NFSv4 supports accessing files that have been unlinked, but still have one or more open file descriptors. NFSv3 does not. It needs to resort to a technique named ‘silly renames’.

http://linux-nfs.org/wiki/index.php/Server-side_silly_rename

LeoPanthera• 3 years ago

NFS4 is a huge ugly bloated mess of a protocol, and they keep adding more stuff to it.

NFS3 is small (I wouldn't say "elegant" but by comparison...), works, and doesn't have anything wrong with it.

chungy4• 3 years ago

NFSv4 has sane, good locking out of the box. Implementing lockd on an NFSv3 server is a big world of hurt.

andrewstuart• 3 years ago

So it’s a local file system driver accessed via NFS? Is that right? I get that NFS is on one side, but what’s on the other?

ylow13• 3 years ago

This NFS Server on one side, and the OS's builtin NFS client on the other.

lukax4• 3 years ago

It appears as a network drive on all major operating systems and can be accessed as any other local filesystem (through kernel syscalls).

iroddis• 3 years ago

FUSE is used to present a file system interface of something that isn’t a file system. E.g. present an LDAP directory as a file system.

Writing FUSE drivers is pretty frustrating, though, and not well supported by all OSs.

My takeaway is that this project discards FUSE in favour of making an NFS server that can be easily extended to do the same thing: present non-file system systems as mountable nfs remote file systems.

It’s a neat idea, and does avoid any extra kernel drivers, etc. A safer abstraction at the cost of adding a network stack and NFS’s sometimes wonky effects (especially with hard mounts).

Snarwin• 3 years ago

Page is completely unreadable in Firefox for Android.

DistractionRect• 3 years ago

No, it's that dark reader doesnt play nice with the background texture on that site. Perfectly readable with dark reader disabled.

rantingdemon• 3 years ago

Thank you kind soul!

renewiltord2• 3 years ago

This is clever, considering the constraints. We use a fork of catfs write now to have a read through cache over NFS, but we could, in theory use a similar flow as here to do it and this will be a good example. Nice. Part of the problem, though, is that we currently use extended attributes, and I think you need NFSv4 plus a modern Linux to get those across NFS.

ylow13• 3 years ago

Doing some quick googling, I see https://lwn.net/Articles/353831/ which sounds like an xattr protocol extension to nfsv3. I do not know the extent to which it is supported though...

renewiltord2• 3 years ago

Couldn't make it work. But works with v4 on Linux 6.

EdSchouten6• 3 years ago

For people who are interested in doing something similar in Go, some time ago I implemented a generic VFS that can be exposed both via FUSE and NFSv4.

It’s part of Buildbarn, a distributed build cluster for Bazel, but it can also easily be used outside that context.

Details: https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4....

My recommendation to the authors would be to use NFSv4 instead of NFSv3. No need to mess around with that separate MOUNT protocol. Its semantics are also a lot closer to POSIX.

ylow13• 3 years ago

Very nice. The mount protocol is pretty minimal, though different OS clients seem to use it differently. The NFSv3 chattiness is definitely a thing, but since its all localhost its not "too bad". v4 will be nice and is something I will love to try to implement. Its quite a bit more complex though.

ylow13• 3 years ago

Another possibility I am interested in is SMB3. (SMB4 is rather nasty). Similarly, supported by most operating systems, and does not look too horrifying to implement (The protocol documentation is really really hard to read though). And has the advantage that it does not require Windows Pro edition.

NoahKAndrews• 3 years ago

It appears that Windows still doesn't have an NFSv4 client: https://learn.microsoft.com/en-us/windows-server/storage/nfs...

formerly_proven2• 3 years ago

I guess these are used as a soft mount, considering that if you terminate the application or it crashes a hard mount would just hang indefinitely. That's probably fine for a read-only snapshot, most problems with soft mounts are around writes.

loeg• 3 years ago

I wouldn't recommend "hard" mounts to anyone.

formerly_proven2• 3 years ago

Well you trade system lockups with sometimes getting IO errors under load, which one is better ... shrug

minroot• 3 years ago

"I love files"

Who doesn't?

fragmede• 3 years ago

Uncompressed DPX video files. For those, a SAN exporting a block device (iSCSI, AoE) is preferable.

barryrandall• 3 years ago

People waiting on a backup process to restore over a billion tiny files.

ComputerGuru• 3 years ago

Another option is plan9; Windows uses the p9 protocol to mount host/guest directories and drives for IO in WSL.

Karrot_Kream• 3 years ago

Wow I didn't know that! https://superuser.com/questions/1749690/what-is-this-weird-p... that is really interesting.

peoplearepeople• 3 years ago

This is interesting, is it possible to mount NFS as a client in an unprivileged user namespace on linux?

lukax4• 3 years ago

On Gnome you can use `gio mount` otherwise afaik no.

vamega• 3 years ago

NFS seems to be a good fit for their use cases, however I don't believe that NFS supports an `inotify` like interface which makes it less useful for usecases like virtual filesystems for developer environments.

Maybe newer NFS revisions allow for the NFS server to notify clients of changes like this, which would be really open things up here.

hathawsh2• 3 years ago

Very good point. Is there anything out there that's like NFS but also supports inotify? By "like NFS" I mean a remote filesystem protocol (not a block-level protocol) that works across platforms and can support nearly all filesystem features.

EdSchouten6• 3 years ago

NFSv4.1 and later do, via a callback operation named CB_NOTIFY.

hathawsh2• 3 years ago

That's great to hear. Thanks!

MayeulC• 3 years ago

Anyway, interesting idea. As another comment pointed out, I wish there was an easy way to mount NFS and 9p shares as a user.

I've been toying with 9p+Wireguard lately, as an alternative to SMB/Webdav/sftp/etc for sharing files with friends over the Internet. Wireguard+NFS works well enough, but it's relatively hard to integrate in userspace. Toying with rust-9p[1], I obtained acceptable performance. I still have to experiment with putting Wireguard in userspace as well (probably with Tailscale's implementation), but I come back to the difficulty of mounting 9p and NFS shares as a user. I also wish it was simpler to export directories over both protocols, on an unprivileged port, with access rights limited by those of the user.

Regarding the article content, I find 9p to be even simpler and probably more ubiquitous than NFS, though I don't know if MacOS and Windows natively support it.

[1] https://github.com/pfpacket/rust-9p

mato3• 3 years ago

Anyway, interesting idea. As another comment pointed out, I wish there was an easy way to mount NFS and 9p shares as a user.

Add e.g. foo.bar.com:/path /net/foo.bar.com nfs to /etc/fstab with options=user ?

Regarding the article content, I find 9p to be even simpler and probably more ubiquitous than NFS, though I don't know if MacOS and Windows natively support it.

Windows certainly does not. MacOS I don't know, but if you mean "natively" as "stock install without third-party software", then probably not.

mato3• 3 years ago

Windows certainly does not.

Actually, scratch that, comments below suggest that WSL2 uses 9P for file sharing with the host, so it's possible that a version of Windows with the WSL2 bits installed will support 9P.

chatmasta• 3 years ago

If you haven't seen it, Cloudflare also has an implementation of Wireguard in userspace, called Boringtun [0], written in Rust and successfully deployed to millions of iOS and Android devices running the 1.1.1.1 app.

For reference, another Rust project that depends on Boringtun is Onetun [1], which uses it to encrypt packets sent over a virtual smoltcp interface. I imagine you could follow a similar approach to integrating rust-9p with Boringtun, and you wouldn't need to leave the Rust ecosystem (whereas you might face more obstacles integrating it with Tailscale's wgengine, which is written in Go).

[0] https://github.com/cloudflare/boringtun

[1] https://github.com/aramperes/onetun

mato3• 3 years ago

This is awesome. NFS is very much undervalued as a cross-platform interface for talking to a (not necessarily remote) filesystem.

~20 years ago I was using SFS (Self-certifying file system), which was a SSH-like Internet-usable TOFU filesystem, using NFS as the "backend" for talking to the host OS. The site has since disappeared, but is archived at https://web.archive.org/web/20080330201843/http://www.fs.net....

appplication• 3 years ago

My only experience with NFS is it being excruciatingly slow. Is that not a universal issue?

snapplebobapple• 3 years ago

When i shared the same conrwnr over nfs and cifs from truenas scale nfs was faster by about 15%..

selfhoster11• 3 years ago

If anything, it's probably one of the fastest ways to access files remotely that I have come across.

notacoward3• 3 years ago

Gluster did this about a decade ago, though not in Rust. Actually twice - home-grown NFSv3 and Ganesha-based NFSv4. At Facebook we used the NFSv3 version to run dozens of clusters serving tens or perhaps hundreds of thousands of machines. My own involvement with that piece was minimal (I was a maintainer of Gluster overall for a while) but it worked pretty well for those users who needed a real mountable filesystem with real files instead of vaguely file-like objects that let you down every time you try to step beyond whole-file read and write.

ylow13• 3 years ago

Nice! I did look at Ganesha when I was exploring other NFS server implementation I could perhaps borrow and simplify. It looked great, but pretty much all implementations are too "real-filesystem" for me to convert. I needed something which I can make a very simple virtual filesystem interface over, and so wrote my own.

ylow13• 3 years ago

(Love the Gluster project btw. Thanks for maintaining it!)

znpy• 3 years ago

Did i read correctly, nfs v3?

IIRC the statelessness of nfs was largely considered a design mistake, and there usually are a large number of hacks to overcome the consequences.

So much so that nfsv4 moved to a stateful protocol…

ylow13• 3 years ago

I don't know that its considered a "design mistake". It is not without flaws, but it does make it extremely simple to implement. Its a very acceptable point as a tradeoff between simplicity and performance. Will certainly love to implement NFS4, or even SMB3 at some point though.

amadio2• 3 years ago

XetHub has the world’s first natively cross-platform, user-mode filesystem implementation, allowing you to mount arbitrarily large datasets on your machine.

Not really world's first. CERN has developed EOS (https://eos-web.web.cern.ch/) for many years, and even though it's not available natively on Windows, it is available on Linux and macOS. EOS uses FUSE, though, not NFS.

This enables you to, in just a few seconds, locally mount ~660 GB of Llama 2 models or write DuckDB queries to analyze large parquet files and scan just the data you need.

If you mount all instances of EOS at CERN on your machine with the FUSE client, that in principle mounts hundreds of PB of data from LHC experiments, although much of this data requires special permissions to be accessed. However, there's also a lot of open data. See https://opendata.cern.ch/.

eviks• 3 years ago

If it's not available on the most popular OS, it's not cross-platform

amadio2• 3 years ago

The client is available on Windows via a samba gateway. There's also a sync client for CERNBox for Windows, and even Android and iOS (https://cernbox.web.cern.ch/cernbox/downloads/). The server is not available on Windows, though. However, I'd say that among physicists Windows is definitely the least popular OS.

KirillPanov2• 3 years ago

So glad there is at least one other human being out there who understands what a huge step backwards NFSv4 was.

NFSv3+wireguard is approximately the perfect network filesystem protocol.

10000truths• 3 years ago

Care to elaborate on why NFSv4 was a huge step backwards compared to NFSv3? I'm not familiar with the details of either protocol.

infotogivenm• 3 years ago

Nice. A similar project, but more license-constrained, is fuse-t for macos: https://github.com/macos-fuse-t/fuse-t

NFS > FUSE: Why We Built Our Own NFS Server in Rust

Related Stories