SeaweedFS – A simple and highly scalable distributed file system with S3 API

by dragonsh • 6 years ago

106 75 6 years ago

ganafagol• 6 years ago

Geez, what's with those weird project names? For a second I expected/hoped this would be some cool hack storing data in actual seaweed. (You know like pingfs..) No it's not! It's some S3 k8s ... thing. Nothing wrong with that but come on, choose a better name!

And no I'm not particularly fond of the namr CockroachDB either.

heipei4• 6 years ago

I'd rather have weird names that are unique than the incredible name overload from generic / cool product names that are used from flagship products all the way down to 5-line npm packages.

corobo• 6 years ago

I'm on the other side of this. Do a Google search before naming your thing!

Make it as silly and unique as possible, I'll be able to search for solutions later

taxcoder• 6 years ago

Make it as silly and unique as possible, I'll be able to search for solutions later

Silly? Not my preference. Unique? Absolutely. Something made up or completely out of context is likely to be a whole lot easier to search for.

kermatt• 6 years ago

On the other side of this, SeaweedFS appears to be largely implemented in Go, an useless keyword to search on.

"Golang" works much better, but sometimes results in a pedantic correction "The language is Go not Golang", to which I can only reply that the project is hosted on golang.org. Weird a.k.a. unique names are useful.

CockroachDB: Cockroaches are hard to kill. The vendor wants to imply this is also the case for their platform.

chrislusf17• 6 years ago

haha. Actually SeaweedFS started with a worse name. But some users warned me of how bad the name was in Japanese language. So I added "Sea" in front of the old name.

dragonsh3• 6 years ago

Thanks for good work with seaweedfs, I shared it with the community as was working on using it personally in our startup to replace minio.

It will be nice if you can integrate it with pyfilesystem [1]. It’s a small library providing unified file interface in python for local or cloud storage without fuse. It can already support seaweedfs using S3 API, it will be nicer to just directly have a driver for seaweedfs removing S3 translations.

I like dried crispy seaweed wafers, they do produce umami flavours, so it is a good name, reminding them. The quality of software isn’t based on name but by what it does, so don’t bother about people talking about name.

[1] https://docs.pyfilesystem.org/en/latest/

heipei4• 6 years ago

This looks almost exactly like the kind of data store I need for an application. I have previously considered using minio (too inflexible wrt to adding more shards / replicas), a homebrew system based on something like ScyllaDB (needs code on top to act like a blob store) or S3/B2 (too slow and/or expensive wrt to transfer costs). Is anyone using this in production and can share a story of how stable and hard to run it is?

drenvuk• 6 years ago

Scylla wouldn't even be good for blob store if you're dealing with megabytes. If you're dealing with kB range Cassandra is stabler and pretty much what you want. For big (MB+) Seaweed is up there for the best option.

heipei4• 6 years ago

Yes, this is another reason that kept me from doing it. I have on class of files (images) that go from 1kB to ~ 1MB in size, but never above. But then I have another set of files that go from 1B to several MB in the worst case (with a normal distribution around 13kB). Ideally I'd like to have one store that can handle all of these so I don't have to worry about worst-case performance too much.

dragonsh3• 6 years ago

Like you we also started with minio, but as it was not fully S3 compatible (list folders and few more related S3 API) and do not have FUSE or NFS support either. So we started moving our infrastructure to seaweedfs. We have not completed the move yet, once done might share some benchmarks. Some of the benchmarks you can also see on their GitHub project.

3np3• 6 years ago

Have you looked at Ceph?

PeterCorless• 6 years ago

Fun fact: Ceph is being rebuilt with the core engine of Scylla, Seastar, at its heart:

https://docs.ceph.com/en/latest/dev/seastore/

0x008• 6 years ago

Whenever you introduce a new solution into a problem space that already has plenty of options, you are obligated to state why your (new) solution is needed in the first place, IMO.

They did it well:

Most other distributed file systems seem more complicated than necessary.
SeaweedFS is meant to be fast and simple, in both setup and operation. If you do not understand how it works when you reach here, we've failed! Please raise an issue with any questions or update this file with clarifications.

https://github.com/chrislusf/seaweedfs#compared-to-other-fil...

However, since I never had to touch hdfs after installing it in the first place, I wonder what the difficulties in operation are, that they tried to overcome here?

rawoke0836002• 6 years ago

Whenever you introduce a new solution into a problem space that already has plenty of options, you are obligated to state why your (new) solution is needed in the first place, IMO.

Agreed ! I'm just very glad, they didn't use the word "lightweight", it's a pet irritation of mine, when ppl "advertise" their software as "lightweight" !

The "first few" versions are always "lightweight" once they included all the bug fixes and corner cases of the other solutions over the next few releases, they seldom stay "lightweight" :/

chrislusf17• 6 years ago

Agree!

Yet I do try very hard to keep each layer separate from each other. So each layer can be "lightweight". :)

threeseed3• 6 years ago

HDFS famously suffers from a major "small files" issue where if you have lots of files that are less than the block size you i.e. 64MB you will have production instabilities due to excessive namenode memory, read/write performance and disk efficiency.

Also when you lose datanodes you will need to rebalance the data which SeaweedFS apparently does not need to do.

chrislusf17• 6 years ago

In SeaweedFS, rebalancing data is via admin commands. The data nodes only act as just as storage devices.

anorwell• 6 years ago

I really wish this project, or other object storage systems modelled after haystack, would get more traction. I think it is reasonable to expect that your object storage system should support both small objects (< 10k) and large objects (> 1MB) transparently, but in my experience none of the heavily used open-source object stores (ceph, swift) can actually support small objects adequately.

ddorian435• 6 years ago

Yep. Not even minio, or aistore(nvidia), ceph etc

Only ambry(linkedin) AFAIK, but has no erasure coding.

chrislusf17• 6 years ago

Thanks!

There are many problems with large number of small files. It's more efficient to batch them together.

jonawesomegreen• 6 years ago

This is really cool. The killer feature I see is to be able to have a cloud storage tier for warm data that goes off to s3, while keeping the hot storage local. Does anyone know of another option that allows this kind of hybrid local / s3 storage that also has a filesystem interface?

ddorian435• 6 years ago

There is also aistore which you can use as a distributed cache in front of different systems (s3, https, seaweedfs).

threeseed3• 6 years ago

Alluxio (https://github.com/Alluxio/alluxio) does this.

But note that it doesn't work well on Kubernetes and POSIX FS interface isn't perfect but then I haven't found any that are.

fh9733• 6 years ago

Their architecture descriptions starts with a strawman:

Usually distributed file systems split each file into chunks, a central master keeps a mapping of filenames, chunk indices to chunk handles, and also which chunks each chunk server has.
The main drawback is that the central master can't handle many small files efficiently, and since all read requests need to go through the chunk master, so it might not scale well for many concurrent users.

The chunk server architecture has been first put into production with the Google File System AFAIK. And it has been designed specifically for large files (what search needed at that time). So no surprise.

But that's only one architecture for a DFS. There are also block-based DFS (like GPFS), object-based DFS (Lustre.), cluster file systems (OCFS), and other architectures. They exhibit other characteristics.

Telling from the architecture and Wiki, it does not seem to be a file system at its core, but an object store with a file translation layer. One of the core problems of this approach is that in-place updates usually mean read-modify-write (if the object store has immutable objects, like most have, with Ceph being a notable exception).

From the replication page:

If one replica is missing, there are no automatic repair right away. This is to prevent over replication due to transient volume sever failures or disconnections. In stead, the volume will just become readonly. For any new writes, just assign a different file id to a different volume.

This sounds like the architecture and implementation is still pretty basic. Distributed storage without redundancy (working redundancy!) is not that interesting.

Sorry to be that critical (great that someone writes a distributed file system!), but I think it is important to add some context. And the seaweed auther seems to have problems with bold statements either...

Disclaimer: I also work on a distributed file system (with unified access via S3 ;)

threeseed3• 6 years ago

a) Project was created in 2012 so these statements may have been made around the time when the most popular DFS was HDFS and GCP was brand new.

b) It does have redundancy through data replication unless I am missing something ?

fh9733• 6 years ago

a) I wonder in which sense HDFS was ever the most popular DFS. You could argue that it is not even a DFS, because it is not a general purpose / POSIX file system. In 2012 there was Lustre, GlusterFS, GPFS, ... Not many DFS were built since 2012.

b) It is completely unclear how the replication works, and which properties it has (split-brain safe?). From what it states (see citation), the "repair" is not automatic.

q3k• 6 years ago

For a distributed filesystem's redundancy to be working in my book, it has to actually be able to reconstruct itself after unavailability/failure, which Seaweed seemingly doesn't do.

Additionally, something more complex than N-replication (eg. M.N erasure coding) is also a strong 'redundancy working' requirement for me.

chrislusf17• 6 years ago

Thanks for educational comments! I need to update those old statements. It was started originally as an object store only.

The blobs are read-modify-write, but a file can have many blobs. So the whole file does not need to be read-modify-write, unless it is a small file.

The redundancy is managed via scriptable admin commands.

fh9733• 6 years ago

Great to see you here!

I understand your unit of smallest write is a blob. How large are blobs? How does their identifier look like?

3np3• 6 years ago

Some differentiators that aren't immediately obvious in the comparison:

SeaweedFS Filer metadata store can be any well-known and proven data stores, e.g., Cassandra, Mongodb, Redis, Elastic Search, MySql, Postgres, MemSql, TiDB, CockroachDB, Etcd etc, and is easy to customized.

I'm not very familiar with other DFS's but at the very least glusterfs stores metadata as xattrs on an underlying filesystem and so has no need of an external data store.

Also, SeaweedFS has a "master" server (single centralized with failover to secondary) and "volume servers" (responsible for data).

chrislusf17• 6 years ago

In SeaweedFS both metadata and data are portable.

You can move metadata to any faster scalable data store. And/or move the data to any cheaper/faster/larger storage, in cloud or in your garage.

gbrits• 6 years ago

Currently using Minio which is a relative success. However as part of quality checks we're using a lot 'listObjectsV2' S3-calls to check for gaps in our data, which is pretty slow. When using SeadweedFS plus, say, a Filer meta-data store based on Redis, this would presumably be faster?

3np3• 6 years ago

Sure. Not really saying one is better than the other; SeaweedFS approach allows you flexibility with these decisions and offloading, GlusterFS has no dependencies on additional services to reach its full potential.

arnoooooo• 6 years ago

I wish all these file systems told me clearly if and how they guarantee file integrity over time.

bzikarsky2• 6 years ago

We are running SeaweedFS successfully in production for a few years. We are serving and storing mostly user-uploaded images (around 100TB). It works surprisingly stable and the maintainer is usually responsive when we encounter issues.

heipei4• 6 years ago

Are you running it on multiple nodes? If so, how confidently can you handle typical "cluster" operations like adding nodes, removing broken nodes, etc? Happy to hear via DM as well.

fastest9632• 6 years ago

We're running across multiple nodes. Removing and adding volume servers is pretty simple. You can manually fix replication via a cli command after adding/removing a node.

bzikarsky2• 6 years ago

Yes, we scale regularly, though we usually only add nodes. We are slowly approaching 100 seaweed nodes. We are running in k8s on local SSD storage, managing failures is easy this way.

teknopurge• 6 years ago

I like it. What advantage does this have over something like Storj? (aside from the obvious operate differences)

chrislusf17• 6 years ago

Predictable latency? Hadoop Support? Too different to compare.

fastest9632• 6 years ago

We've been running SeaweedFS in production serving images and other small files. We're not using Filer functionality just the underlying volume storage. We wrote our own asynchronous replication on top of the volume servers since we couldn't rely on synchronous replication across datacenters. The maintainer is super responsive and is quick to review our PRs. Happy to answer any questions.

some_test_user• 6 years ago

So weird...I was just thinking about this yesterday.

talhof8• 6 years ago

Looking good!

Take a look at Gasper (https://talhof8.github.com/gasper).

ddorian435• 6 years ago

Your link doesn't work. Are you spamming ? At least write something.

marcoherbst• 6 years ago

Evercam has used Seaweed for a few years. We've 1344TB of mostly jpegs and use the filer for folder structure. It's worked well for us, especially with low cost Hetzner SX boxes. I'd echo other people's positive comments about the maintainer's responsiveness & support. Happy to (try and) answer questions.

ddorian435• 6 years ago

What replication are you using ? What about your recovery times when a server fails on 1Gbs port ?

ijunaidfarooq2• 6 years ago

we don't use any replication, Also in almost 5 years we only had one server crash which was due to file-system corruption, and we overcome that as well, it was a few leveldb files which got corrupt due to which the whole XFS file-system was went down, but we recovered it. Just one drawback was: We never used the same filer for saving files, and Get speed was also quite slow on that one, but with time, the volume compaction and vacuum, everything works fine on GET requests.

didip• 6 years ago

I have been following seaweedFS since forever. Played with it on my own homelab.

But I don't know if there's a major shop that uses it. Anyone knows?

chrislusf17• 6 years ago

From a user who worked in Apple as a contractor, they have used SeaweedFS object storage to store build artifacts, grown to about 200 nodes.

There are a few other companies, but not as famous as Apple. :)

ijunaidfarooq2• 6 years ago

Evercam.io is using it for past few years, i.e 5 years almost, And I have been following seaweedfs for almost the same number of years, I think we are the largest one, this is what I think, as I have not seen anyone else, coming back and forth on GH issues :)

dcveloper2• 6 years ago

If Oracle wins the Supreme Court case against Google, aren't all these "like S3" or S3 API compatible solutions (whether block storage competitors or file systems) at risk?

chrislusf17• 6 years ago

Oracle wants to lock Java, while Amazon wants to open up S3 API, so more people can use S3.

On the other hand, SeaweedFS is has no API access fees and faster than S3 with your own hardware. Not sure what Amazon may do.

pjmlp• 6 years ago

Oracle is fine with others using Java, provided everyone plays ball.

https://adoptopenjdk.net/sponsors.html

https://en.wikipedia.org/wiki/List_of_Java_virtual_machines#...

It is Google's own version of J++ that needs to be taken care of.

philjohn• 6 years ago

Oracle Cloud have an S3 API compatible service, don't they?

dcveloper2• 6 years ago

Amazon wants to open up S3 API for now. I guess my point is launching products with S3 API combability I feel is a legal liability if Oracles wins the case.

I was specifically implying block storage solutions that offer S3 API combability (i.e. use our AWS S3 competitor with matching S3 API).

[1] https://www.digitalocean.com/products/spaces/ [2] https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-...

dragonsh3• 6 years ago

My suggestion don’t rely too much on S3 API after this ruling and better to have seaweedfs API as primary interface, until Amazon S3 release their API interface as true open source with right to modify for one’s own use.

This is one of the biggest problems with Java, it’s free but not open source (Eric Raymond has been fighting this for a while now that free does not mean open source). Oracle can come after anyone using Java if there is substantial money to be made.

So Java is only free as long as money keeps coming in for Oracle, once it dries like for SCO, they will go after anyone using Java API to extract money.

Also this shows subtle difference between open source and free. In a legitimate open source licensed software copyright and right to change is granted and company cannot sue others just based on copyright like Oracle can sue anyone using Java API.

davidcollantes• 6 years ago

Is it possible to mount a SeaweedFS?

chrislusf17• 6 years ago

yes. Just run "weed mount". Actually the same binary to start "weed server".

pmiller22• 6 years ago

This is interesting. I've been looking for a file system for a non-RAID disk array I want to set up at home, and this seems to have some of the characteristics I'm looking for. The primary downfall for my particular use case seems to be that I want to use partiy-based error correction rather than (or, in addition to) replication, because I want the array to be able to survive a failure of any N disks in the array.

Is there anything like that out there (other than Unraid, which I kinda don't like)?

chrislusf17• 6 years ago

SeaweedFS does support erasure coding for warm data.

aftbit• 6 years ago

Are there any docs on this feature? I couldn't find anything in the readme other than the blanket statement that it existed.

thrusong• 6 years ago

I am not a large user whatsoever but I've been using SeaweedFS for a few years now.

It is archiving and serving more than 40,000 images on a webapp I built for the small team I work with.

I run SeaweedFS on two machines and it serves all images I host.

I wanted to kick the tires because I was always fascinated by Facebook's Haystack.

It has been simple, reliable, and robust. I really like it and hope if one of my side projects ever take off at some point, I get to test it with a much bigger load.

pinmingsanlang• 6 years ago

two years ago, In DiDi, it has been storing and serving 10 billions of files.

jasonjayr• 6 years ago

The architecture reminds me of 'mogilefs', which has a similar mechanism of filename to file storage.

https://github.com/mogilefs/mogilefs-docs/blob/master/HighLe...

It's an old system from the folks @ Danga, but the mailing list still sees random activity now and then...

1MachineElf• 6 years ago

If you want something similar that also supports NFS, then there's leofs: https://github.com/leo-project/leofs

SeaweedFS – A simple and highly scalable distributed file system with S3 API

Related Stories