Seaweed-FS: simple distributed file system written in Go

by thinxer • 11 years ago

67 11 11 years ago

sargun• 11 years ago

Self-plug. This is based off the Haystack paper: https://www.usenix.org/legacy/event/osdi10/tech/full_papers/... from Facebook. Relevant blog-post is here: https://code.facebook.com/posts/685565858139515/needle-in-a-...

I gave a talk about Haystack at PWL: http://www.meetup.com/papers-we-love-too/events/220795812/ with the relevant video here: https://www.youtube.com/watch?v=EuNumdi1Do0

CSDude• 11 years ago

I have read that paper a year ago, in Cloud Computing course. Does Facebook still use it? I see pictures are now at Akamai.

reactor• 11 years ago

I was looking for something like this (esp to store and server images) for our new application. Looks interesting, thanks for the work.

notacoward• 11 years ago

Already posted on HN (albeit quite a while ago when it was still called Weed-FS) and still not actually a file system.

decwakeboarder• 11 years ago

It looks like a filesystem to the extent that HDFS is a filesystem, it's just not a POSIX filesystem.

tinco• 11 years ago

It's a filesystem like HDFS in the sense that seaweedfs and HDFS are systems that have to do with files, like dropbox and explorer.exe

pram• 11 years ago

Looks very interesting, thanks. Kind of hilarious he didn't make a native Go API though..

This is the only one AFAICT: https://github.com/ginuerzh/weedo

chrislusf• 11 years ago

The "operations" package is more sort-of "official" client, used by SeaweedFS itself.

The feature list is still growing. I prefer not to really have an official GO API before I can have time to polish the API design.

millstone• 11 years ago

Honest question: why is this called a file system and not a file server? It looks closer to Apache than, say, ext3.

e12e• 11 years ago

Looks interesting. I'm curious about why there's a 32bit cookie added to the volume,filename -- it seems like a rather weak protection -- and as such, it becomes an unnecessary complication? As have been shown with facebook, relying on (permanent) secret urls to grant/deny access is a bad idea.

So, why not just use volume,id, and then deploy a proxy that handless access based on tokens in front -- if access control is wanted? (Not all uses of files will need/want access control).

I suppose one reason for a "cookie" would be cache invalidation in case of volume,id reuse.

fit2rule• 11 years ago

I'm thinking that IPFS (http://ipfs.io/) has more of an opportunity to fill the distributed-filesystem role de jeur, alas .. Seaweed-FS seems flakey and not quite ready for primetime. Poor conclusion?

Seaweed-FS: simple distributed file system written in Go

Related Stories