Writing a FUSE filesystem in Python

http://www.stavros.io/posts/python-fuse-filesystem/

by stavros • 13 years ago

142 48 13 years ago

seiji• 13 years ago

I did one of these a while ago that lets you mount a redis server as a local filesystem: https://github.com/mattsta/redisfuse

It's a great way to do FS-like things ("edit X in vim", "run aspell across all keys in Y") on very non-file-like objects.

StavrosK12• 13 years ago

Oh, wow, that's a fantastic idea! Well done, I will play around with it. I love the idea of exposing disparate datasets as filesystems.

cdjk• 13 years ago

You should take a look at Plan 9. Everything is a file, and service are implemented as file servers - including the windowing system (rio).

Some of the ideas are available in Plan 9 from User Space.

BHSPitMonkey• 13 years ago

I did something similar with a digital repository server software for a conference hackathon. PyFUSE is pretty fun to play with, despite the documentation gotchas and warts.

https://gist.github.com/BHSPitMonkey/01cef0d528f374cca8cb#fi...

stevekemp• 13 years ago

I did the same thing, but I used C.

http://www.steve.org.uk/Software/redisfs/

hemancuso4• 13 years ago

I write ExpanDrive, http://www.expandrive.com - which mounts SFTP/S3/Dropbox/Openstack Swift/Google Docs/Box on Mac/Windows & soon Linux all in Python/FUSE.

Ctypes is the way to go for interacting with libfuse. Even better would be speaking the FUSE protocol directly to /dev/fuse*, but is rather complicated by our Windows support.

StavrosK12• 13 years ago

Oh, congrats, I used ExpanDrive when I was using Windows and I remember it being the best S3 client I had tried. I'm very excited for Linux support. Feature request: please make sure it supports ACLs properly, nothing else I've tried does. I'll set a folder as world-readable, but, when I upload a file into it, the file won't have inherited that permission and I have to go and do it manually every time...

samwillis• 13 years ago

Out of interest what do you use for mounting on Windows?

I have found Dokan (http://dokan-dev.net/en/) and pyFilesystem (http://pythonhosted.org/fs/expose.html) has a clever binding there you write your fs once and can mount it with FUSE or Dokan.

Is there something better?

hemancuso4• 13 years ago

Nothing open-source, unfortunately. I originally wrote a Windows fuse-like filesystem driver that we used in our 1.x product line, but that was a lot of work to maintain and have since switched to CBFS https://www.eldos.com/cbfs/ - which I wrote a libfuse binding for that we directly mount through via ctypes. They are actually in the process of formally building out FUSE support for CBFS but it's not officially supported.

casca• 13 years ago

For those who don't know the space, ExpanDrive used to be SftpDrive. I've been recommending it for years to people who are not technical for accessing sftp shares and have never been let down. Thanks for a great product and sharing some detail about how it works.

hemancuso4• 13 years ago

Thanks :)

voltagex_3• 13 years ago

Hey, I had thought that ExpanDrive was using Dokan, but I think I'll pick up a copy now that I know some of the backstory. Thanks for posting here!

lamby3• 13 years ago

Another "me too", but I really enjoyed writing this: https://chris-lamb.co.uk/projects/aptfs

StavrosK12• 13 years ago

Hey, I love "me too"s, they are all interesting projects. That's a very novel way of managing packages, does it work if you copy all the symlinks out and then restore them (so you can back up/restore your installed packages)?

lamby3• 13 years ago

It's simply a view "remote" APT repositories, not your local package setup.

fuzzix• 13 years ago

Have yet another "me too" - I wrote up writing a FUSE module in Perl in a little more detail than The Fine Article:

http://blogs.perl.org/users/fuzzix/2012/08/fun-with-fuse.htm...

The info should transfer to other FUSE bindings.

nickthemagicman• 13 years ago

Ha working on a fuse file system for one of my projects in grad school.

notacoward• 13 years ago

FWIW, you can also write a GlusterFS "translator" in Python.

https://forge.gluster.org/glupy

This allows you to add your own functionality alongside everything GlusterFS already has - e.g. distribution, replication, handling for all sorts of annoying VFS/POSIX special cases - instead of having to do everything from scratch yourself. I'm not saying it's the right option for everyone who might just use FUSE directly, but it might be an option to consider.

Disclaimer: I'm the original author, though others have taken over since.

wizzardy• 13 years ago

If anyone interested to see some real-world open-source FUSE example (but written in C), take a look at our project RioFS: (https://github.com/skoobe/riofs): an userspace filesystem for Amazon S3 buckets that runs on Linux and MacOSX (FreeBSD support is comming as well).

StavrosK12• 13 years ago

Yes! Fantastic, thank you, I've been looking for something like that.

gpsarakis• 13 years ago

Nice post! You mentioned that you implemented this wrapper for backups. Apart from creating a virtual file system for sandboxing perhaps, isn't this generally slower? Maybe I am not getting the exact purpose of this task.

StavrosK12• 13 years ago

Slower than what? This is for creating encrypted versions of your files before you back them up, so your backup program will only see encrypted files.

aray2• 13 years ago

Neat project and great post. One thing I'd like to point out is that there is another mechanism for implementing userspace filesystems in Linux: V9FS. FUSE has a lot of documentation -- much of it in various stages of bitrot -- and many of the comments here point to other FUSE projects as well!

The underlying mechanisms for FUSE are pretty convoluted and difficult, so much more than trivial examples can break the abstractions afforded by the libraries like the one used in the post.

V9FS is in contrast exceedingly simple and basically unchanged. And, yes, it's based on the Plan 9 filesystem protocols :)

http://v9fs.sourceforge.net/

marbu• 13 years ago

This is interesting. I'm aware of the fact that some plan 9 protocols were ported to linux, but are you suggesting that this v9fs is a replacement for fuse in common use cases? Do you know about simple comparision of fuse and v9fs implementation of the same userspace filesystem?

aray2• 13 years ago

V9FS and FUSE both enable userspace filesystem drivers. They are actually pretty interchangeable for 99% of use cases, and the rest of those cases are where you'd directly interact with both, so it wouldn't be fair to compare a FUSE library abstraction with V9FS.

FUSE is too complicated to directly interact with, so everyone (OP included) is using libraries and abstractions on top of it. So what you'd really be comparing is that abstraction, as well as possibly how it limits actual FUSE interaction. Additionally, the performance of these abstractions is hard to predict, as many do lots of work (threads/IPC/etc) silently in the background, so the apparent workload on your filesystem driver is far from actual workload.

V9FS is simple enough that drivers often just talk it directly. The protocol is a simple RPC mechanism, and parallelism happens naturally.

As far as a direct comparison example, I don't know of one.

frio2• 13 years ago

What kind of performance did you get out of this? I wrote a naive implementation of a FUSE filesystem in Python a while back -- it read the sickbeard DB and built a directory of the last snatched programs. The performance was pretty shocking, but workable for a single user. I always assumed that was down to Python, so I'd be interested to hear how well yours performs.

StavrosK12• 13 years ago

I haven't tried it at all, since it just needs to be faster than my Internet connection (it's for backups), but you can just run the file provided there and it should work properly. Please post a benchmark if you do (I'm not at home now)!

frio2• 13 years ago

I'll give it a try tonight :)!

bambambazooka2• 13 years ago

Fun fact: bup has a fuse module to access saved files: https://github.com/bup/bup/blob/master/Documentation/bup-fus...

StavrosK12• 13 years ago

That's pretty great, I think I will use bup when EncFUSE is done, rather than rdiff-backup. It's also very actively maintained, which is great.

IgorPartola4• 13 years ago

I found rsnapshot to be far superior.

j_s• 13 years ago

Does encryption prior to backup mean less de-duplication?

StavrosK12• 13 years ago

It does. encbup uses CBC mode, which means that you'll only have per-file deduplication, but EncFUSE uses the less secure ECB mode, which means per-block deduplication (roughly almost as good as unencrypted).

Writing a FUSE filesystem in Python

Related Stories