An in-memory distributed Posix-compatible file system

http://bshafiee.github.io/BFS/

by shafiee01 • 12 years ago

31 13 12 years ago

bshafiee.github.io

dsr_• 12 years ago

The standalone mode is boring -- it's a ramdisk, about the same as you get from tmpfs.

The standalone_swift mode is interesting -- it's a ramdisk with eventual commit to disk -- but I don't see performance figures or essential config details, like how often it flushes.

The distributed mode is surely cool for some use cases, but it doesn't seem to commit to disk and it neither replicates nor works in parallel.

shafiee016• 12 years ago

1-Well if you don't want to use a backend storage what else do you expect?

2-It will flush as soon as you call flush or close your file. It's not necessarily disk and can be any configured backend (right now only swift is supported; however, it's easy to develop other backends as well)

3-Replication is done in the backend storage. Once your data is flushed it's replicated to your desire in Swift, Amazon or whatever backend you are using. What do you mean 'nor works in aralllel'?

notacoward5• 12 years ago

If the back end is eventually consistent, how can you claim POSIX compliance on the front end? User writes and fsyncs, BFS node goes down, somebody tries to read. What will they get? Who the heck knows? The only in-memory copy is gone, and the back end you've chosen doesn't guarantee that you'll get the most recently written data on the next read.

It's perfectly OK if you decide on a different consistency model, but then you can't say you're POSIX compliant (actually you can't anyway for legal reasons but that's another topic).

notacoward5• 12 years ago

The obvious, comparison for standalone mode would be to a regular file system built on a ramdisk. The obvious comparison for network mode would have been Plain Old NFS with a ramdisk on the server. Both comparisons are missing. Also, this statement is flat-out untrue.

IOZONE performs all of its operations on one file

Getting iozone to use multiple files and threads is trivial (-t option), and the results for that would be much more interesting than those for a single file/thread. Unfortunately, such benchmark errors are extremely common. I don't fault the author so much as his advisor, who seems to have a background in computer networking (hence the strong focus in this work on alternative network interfaces) but none in storage. My advice to the author would be to find a more appropriate advisor for this project.

shafiee016• 12 years ago

1- This is just a preliminary evaluation. I just made the repository public 3 hours ago. I am planning so many experiments including the ones you mentioned.

2- As I said more benchmark including multiple files with iozone, postmark are coming. This is just to give viewer a sense. P.S I am happy with my supervisor :) thanks for your recommendations though ;)

notacoward5• 12 years ago

As you continue your testing, I strongly recommend that you use different numbers of threads/files and queue depths, as well as different file and I/O sizes. If you want to make scalability claims, you'll also need to test with multiple clients and servers. It's also good practice to show the exact iozone commands used, or (even better) fio command line and profiles.

More generally, I think it would be very helpful if you could be more explicit about how your system works. For example...

* What are your consistency/durability guarantees? I mean really, since they're clearly not POSIX.

* How do you detect and respond to faults?

* How would you describe the system in CAP terms?

* How do you reconstruct file system state from the backing-store object state after a node failure or cold restart?

* How do you decide which node should hold a file? How do you re-make that decision when servers are added or removed? Can they be?

These are the decisions that every distributed file system must make (and an explicit comparison to others sure wouldn't hurt). While it might seem like a lot, you did say this is for a master's thesis and it's stuff your advisor should already have required. It will certainly come up when you go to defend that thesis. If this truly advances the state of the art in some way, you should already know exactly how and be able to explain it.

Disclaimer: I work on GlusterFS. On my own, I've worked on two toy projects (CassFS and VoldFS) that explored ideas in how to build a distributed file system on top of another kind of data store. I might be able to help you, if I knew more about which issues you'd already addressed and which remain.

virtuallynathan• 12 years ago

Are you planning to benchmark this on 10G/40G networks? looks like on first glance it is limited by the available network bandwidth.

shafiee016• 12 years ago

I wish if I could :) but right now in our lab I don't have access to such equipments. Honestly, that was the plan because using 10/40G networks will significantly reduce the delay in remote operations as well as increasing the throughput in remote operations.

notacoward5• 12 years ago

Might I suggest a round of testing on Amazon or Rackspace? They both have 10Gb-equipped instance types with SSDs and everything, for a few bucks per cluster-hour. That shouldn't break even an academic budget. You might not be able to do your special ZERO networking in that environment, but you'd be able to explore other interesting corners of the performance space and your experiments would be reproducible by others.

An in-memory distributed Posix-compatible file system

Related Stories