HN user
Built a secure shared filesystem on top of Amazon S3:
https://objectivefs.com
One advantage over S3FS would be that multiple filesystem mounts would see a consistent view of the filesystem, but it looks like this advantage disappears when mixing direct bucket access with filesystem mounts. Given the famously slow small file performance of EFS it might have been better (and cheaper) to send all files to S3 and only use EFS for the metadata layer. Not having atomic rename is also going to be a problem for any use that expects a regular filesystem.
There is also ObjectiveFS that supports FUSE and uses S3 for both data and metadata storage, so there is no need to run any metadata nodes. Using S3 instead of a separate database also allows scaling both data and metadata with the performance of the S3 object store.
Congratulations on your launch from ObjectiveFS! There is a lot of interest in 1-to-1 filesystems for mixed workloads, hope you can capture a nice share of that.
Using NFS and being able to use an existing bucket is a nice way to make it easy to get started and try things out. For applications that need full consistency between the S3 and the filesystem view, you can even provide an S3 proxy endpoint on your durable cache that removes any synchronization delays.
For workloads with many small files, it usually is better to store many files in a single object. Filesystems with regular POSIX semantics, such as atomic directory renames etc, also makes it easier to integrate with existing software. We have seen a lot of scientific computing usage of our filesystem (https://objectivefs.com) and as you mentioned localized caching of the working set is key to great performance.
ObjectiveFS takes a different approach to high-availability durable file system by using S3 as the storage backend and building a log structured file system on top. This leverages the durability and scalability of S3 while allowing e.g. git cloning/diffing of a large project (even cross region) to remain fast. To compare we have some small file benchmark results at https://objectivefs.com/howto/performance-amazon-efs-vs-obje... .
For a high performance shared file system on AWS, an alternative is ObjectiveFS[0]. It uses memory caching to achieve performance closer to local disk.
For TB-scale heavy read loads ObjectiveFS[0] is an alternative to EFS that don't have EFS's IO/bandwidth limits. It is often used for distributing data for machine learning and large scale simulations on EC2 and GCE.
Another file system that works well with lots of small files and that doesn't require maintaining any extra servers is ObjectiveFS[0]. It is a FUSE based log structured file system (with snapshots) using GCS or S3 for log storage with memory cache and (optional) disk cache for performance.
We are looking into the best way to add native kubernetes support. Currently, you can add a mount on the host or directly mount the file system inside the container. Both approaches work well, so it mainly depends on your preferred architecture.
Using a clustered/distributed filesystem definitively simplifies persisting the state between EC2 spot instances. It also makes it easier to scale out the work load when you need more instances accessing the same data. To add to your list: there is also ObjectiveFS[1] that integrates well with AWS (uses S3 for storage, works with IAM roles, etc) and EC2 spot instances.
When you have an object store available (Ceph, Cleversafe, S3, etc), and need a POSIX file system you can also use ObjectiveFS[1]. By using an existing object store for the backend storage, it separates the concerns of reliable storage from the file system semantics which makes it (we think) easier to set up and use. With object and block storage working great for their use cases, file systems can provide some extra functionality, e.g. point-in-time snapshots of the whole file system, which makes it easy to create a consistent backup of your data.
If you have access to an object store, you can get persistent file system storage for your containers using ObjectiveFS[0]. You can run it either on the hosts or inside the containers depending on your goal.
Latency for small random reads (16 parallel threads), using the Linux kernel source tree, when hitting in the SSD instance store disk cache have a 95th-percentile latency of 6ms with ~50MB/s throughput. When hitting in the memory cache the 95th-percentile latency is <1ms with ~380MB/s throughput. We have more performance data available at https://objectivefs.com/howto/performance-amazon-efs-vs-obje...
An alternative to user5994461's suggestion would be to switch from CephFS to a different distributed filesystem. If you need POSIX and can't go directly to S3 you could try ObjectiveFS[1]. Using local SSD instance store and memory for caching you can get very good performance even for small file workloads[2].
[1] https://objectivefs.com [2]: https://objectivefs.com/howto/performance-amazon-efs-vs-obje...
We have some notes on small file performance for EFS and ObjectiveFS at https://objectivefs.com/howto//performance-amazon-efs-vs-obj... that you might find interesting. Scaling out PHP on AWS is definitively easier with a shared file system.
If you need more performance than EFS for your shared filesystem storage, you could give our ObjectiveFS (https://objectivefs.com) a try. We see significantly higher read/write performance, especially for smaller files.
Both GlusterFS and ObjectiveFS are shared filesystems instead of block storage, so they work with POSIX files/directories (like NFS) making them easier to scale out. If your application can work directly with an object store that is of course also a great way to go.
One way to handle persistent container storage is to use a shared filesystem such as glusterfs or objectivefs. It's easy to set up and can be very helpful when moving your infrastructure to containers. Once created you can mount your shared filesystem either on the host and share it with the container or directly in the container, see e.g. https://objectivefs.com/howto/how-to-use-objectivefs-with-do...
For stateful containers that need access to a shared folder you can also use a shared file system such as GlusterFS (http://www.gluster.com), EFS (https://aws.amazon.com), or ObjectiveFS (https://objectivefs.com). This keeps your state in one location and all your containers can access it through the regular file system interface, i.e. you can start using it before you have rewritten everything to be cloud native.
Great that you have been wanting to use ObjectiveFS. We are happy to talk with you about your non-S3 storage use case and see if we can find a plan that works for you.
Cool! If you are looking at shared filesystems on EC2, you might also want to take a look at ObjectiveFS[1]. Works in all regions and on GCS, and you can mount your filesystem securely (end-to-end encryption) between regions.
ObjectiveFS[1] is another option if you have access to an object store (e.g. S3 or GCS) from your containers.
If you are looking for a POSIX compatible file system for GCE or EC2, we think our ObjectiveFS[1] is the easiest way to get started and use. It is a log structured filesystem using GCS or S3 for backend storage and with a ZFS like interface to manage your filesystems.
You can use our ObjectiveFS[1] if you want a networked file system where multiple instances can simultaneously mount it read/write. It is backed by S3 and gives you a standard POSIX interface.
ObjectiveFS does all the coordination among the clients through S3, so there is no extra coordination service needed. This is why we are really happy that Amazon recently moved the S3 us-east-1 region to read-after-write consistency (like all their other regions).
We do write bundling before sending data to S3 so lots of small files would be packed together and stored in a single S3 object. This also helps reduce the number of object store operations.
You are absolutely right that frequent sync will necessarily create many small objects, which is why small objects will be combined into bigger ones (this compaction is done in the background). Sync latency is of course bounded by the S3 PUT time, since fsync(2) can only return after your data has been safely stored in S3.
It is a really interesting design problem. Thanks for sharing your ideas.
ObjectiveFS comes with fast compression that is always enabled, it works great for log files and other compressible data. Let us know what you think if you give it a try.
It uses FUSE under the hood. Thanks for the feedback, we will make it clearer on the website.
S3QL has a lot of great features. You are right that support for concurrent mounts is one of the main differences. ObjectiveFS also keeps the file system data consistent and stores it in S3 asap, for sharing and to handle EC2 unexpected termination. S3QL uploads all metadata changes at a regular interval (by default every 24h) or during unmount.
We built a shared file system with an S3 backend. We've been running our own servers for many years, and wanted something low maintenance, reliable, scalable and secure. So, we built this POSIX-compliant shared filesystem for Linux and Mac.
We love to hear your feedback. Feel free to try it out. We will also be here to answer questions.
As anishathalye said you probably don't want to keep your git working directory on Dropbox. If you want to share a git working directory between machines, you can use a shared filesystem such as our ObjectiveFS (https://objectivefs.com). Note that if you use git with an OSXFUSE filesystem you currently need to add the "--quiet" option to work around an issue with signals (https://github.com/osxfuse/osxfuse/issues/213).