Distributed FS: MogileFS

http://www.danga.com/mogilefs/

by astine • 17 years ago

20 5 17 years ago

jobenjo• 17 years ago

We've been using MogileFS for the last few years to host all our avatars. It's fast and powerful, and has had very few issues.

It's definitely a solid system, though was certainly overkill for our needs (if I could do it again, I would just use s3).

jbellis2• 17 years ago

Mogile relies on a central mysql db for metadata (e.g. replica locations) so it doesn't scale well at all, especially not when your workload emphasizes many small files ( because then you exhaust mysql that much faster relative to your disk capacity).

recurser2• 17 years ago

We're running about 90 terabytes of mogile storage containing 20-30 million files and haven't had any problems at all. If we do, the plan is to shard them along the same lines as our database sharding. I don't know of any other products apart from S3 that could have made our file hosting as stress free as it is. What kind of scaling problems have you had?

jbellis2• 17 years ago

so your average file size is 4.5MB. that's a pretty good fit.

> I don't know of any other products apart from S3 that could have made our file hosting as stress free as it is.

I think you're right about that part. Better solutions don't come off the shelf. Yet. :)

> What kind of scaling problems have you had?

the system I wrote at Mozy needed to scale to PBs of space and billions of files, so it was obvious that the mysql part would be a problem.

more generally I think mogile is in a pretty narrow niche: typically either you really need a lot of space and files, like flickr or facebook or ..., in which case mogilefs doesn't cut it, or you have a smaller data set but you want random access to your files in which case mogilefs also doesn't cut it.

Distributed FS: MogileFS

Related Stories