Amazon Elastic File System – Production-Ready in Three Regions

https://aws.amazon.com/blogs/aws/amazon-elastic-file-system-production-ready-in-three-regions/

by jeffbarr • 10 years ago

232 119 10 years ago

aws.amazon.com

atrilumen2• 10 years ago

Can EFS be mounted from the Lambda environment?

jeffbarr2• 10 years ago

Not at present. I would be interested in learning about use cases for this. Post here or find me online, as you wish.

markonen• 10 years ago

You mention HPC and Big Data processing as use cases for EFS; Lambda is also very well suited for big processing tasks you want to execute quickly with a large amount of concurrency. You often need persistence across executions, for example when breaking tasks into multi-function workflows, and EFS would seem to simplify this over manually using S3 for it.

jedberg7• 10 years ago

One use case would be one that came up a while back here on HN -- The person had a large binary blob that they needed to load up every time their Lambda function ran. If they could mount EFS from Lambda, they could put the large binary blob there.

Depending on how they wrote their code, if the Lambda could seek to the right part of the file then this would work, but if it has to load the entire file into RAM anyway, then it wouldn't gain them much except perhaps a slightly easier way to get the file into RAM.

atrilumen2• 10 years ago

I want to run the Elm compiler in a Lambda, and maintain persistent environments so that subsequent builds can benefit from keeping the libraries and intermediate files available.

shatgupt• 10 years ago

I would like to have a media pipeline where a media goes through QC check, thumbnail, preview generation and some other processing which are independent of each other. My media can range from 1-15GB, so if I can store it in EFS which can be mounted in Lambda, I can use various Lambda functions which can simply read from a common storage without downloading every time from S3.

buremba4• 10 years ago

We use a columnar database and we insert data to it periodically (30s to 90s) using Kinesis as middleware. Files are immutable and each insert creates [count of columns] * [count of tables] files. The database also periodically merges the batch files in background.

Since EFS was not available, I was trying to develop a FUSE filesystem that writes data to both S3 and local filesystem (EBS) for durability and read only from local filesystem for performance. Even though it's cheaper than EFS, it's slow since each file needs to be sent to S3 individually. Our requirements are not that much because the files are immutable, there are not many files (I heard that EFS doesn't work well with many small files.), no need for concurrent access. I think I will give it a shot since it also fits our use-case.

takeda2• 10 years ago

If files are immutable, isn't s3 much better fit for it? s3 main weaknesses is that you can't modify part of a file, which makes it unsuitable for a filesystem, but looks like it satisfies your use case.

buremba4• 10 years ago

Maybe but unfortunately the database doesn't support "backup store" that we can use directly S3, it just writes the data to specified directory in a filesystem. We should either fork the database or implement "S3 backed" filesystem and we picked the latter one. Unfortunately the performance FUSE is not that good for read and writing data to S3 is somehow expensive.

Someone3• 10 years ago

They use S3. Their problem likely is that they write many small files to S3, and repeat it every minute or so. That's a combination S3 isn't designed/optimized for.

madbiz3• 10 years ago

Can EFS be used as a volume for ECS tasks?

bbgm• 10 years ago

Here's a post we wrote earlier in the year about EFS and ECS: https://aws.amazon.com/blogs/compute/using-amazon-efs-to-per...

madbiz3• 10 years ago

Thanks!

madbiz3• 10 years ago

In the blog post the EFS is mounted by the EC2 instances from the start but i expected to be dynamic depending on the scheduled tasks on the instance. I dont want to couple EFS volumes which might be used by the containers if they might get scheduled on an instance. I would like to have a general purpose cluster and let the scheduler instrument the EC2 instance to mount an EFS volume if its needed by a scheduled task dynamically.

illumin82• 10 years ago

If you use the Convoy NFS volume driver with EFS, your containers can use any NFS server (including EFS) as highly available persistent volume storage: https://github.com/rancher/convoy

jedberg7• 10 years ago

This is really cool but also kind of a shame. I feel like it will encourage bad behavior.

I can see people using this to deploy code by deploying onto the shared volume so that all their instances get an "instant update". Which is great when it works, but lord help you when EFS goes down and every app server you have is hung on a broken NFS connection.

oconnore2• 10 years ago

Isn't the whole point of AWS that you can blame AWS engineers for downtime instead of your own (in this case) code deployment thingy?

jedberg7• 10 years ago

I'm not sure if you're being sarcastic or not, but no, it isn't. It's to provide building blocks for infrastructure. Down time is still on you. Only a lazy engineer blames their tools for their failures.

sciurus• 10 years ago

http://www.whoownsmyavailability.com/

hdra• 10 years ago

I have to admit that is the first use case that I thought of when I read this. No more messing around with AMIs and launch config whenever an update is deployed.

You bring up a good point on the EFS being a point of failure though. What use case do you think the EFS is good for? It seems like even worse for data storage since IO can easily become a bottleneck.

jedberg7• 10 years ago

What use case do you think the EFS is good for

Out of band big data/data pipeline processing. Something where you want an easy way to sync your data but can handle extended downtimes.

KaiserPro5• 10 years ago

but thats the same with any component. how is that different to SNS et al?

NFS is actually multi homed, so there is no real reason why it can't be HA/clustered, apart from the block store and the underlying file system.

jedberg7• 10 years ago

how is that different to SNS et al?

It's not really, and you should write your software to account for SNS outages, etc.

That being said, this is presented as NFS. NFS has a nasty habit of freezing your system if it breaks. If SNS breaks, you get errors and timeouts in your logs, but can keep going. If NFS breaks, you pretty much just sit around waiting for it to come back.

Corrado6• 10 years ago

I think that deploying onto a shared volume is exactly the workload that EFS encourages. That things will stop functioning can be said about any (AWS) technology. "Lord help you when (S3|RDS|SQS|etc.) goes down and every (thing) you have is hung up on a broken (thing) connection."

The rational argument here is that if uptime is important to you, the solution is to utilize multiple regions, of which EFS is now in 4. You are using AWS because of the SLA and the assumption that when something goes wrong there are legions of technical folks trying to fix it as soon as possible.

jedberg7• 10 years ago

The difference is that all of those services are accessed via an API, which means your client can do clever things when it fails, like timeout, give a fallback, find an equivalent resource in another zone or region, etc.

With EFS, it's exposed as NFS, which hooks in much deeper. If it goes down, there isn't anything you can do to work around the problem, unless you start hacking your own file system kernel modules.

walden42• 10 years ago

Assuming all assets get stored on S3, the other common writing scenario is logging. How would multiple machines writing to the same log file work?

Thaxll• 10 years ago

Most app / language load everything at startup.

nemaar• 10 years ago

The instance can fetch the upgrade from the NFS mount and also work using the previous version when NFS is down. I see no problem here.

facorreia• 10 years ago

If people want to deploy that way, I suppose that using rsync to update a local clone, and running from that, would minimize this problem.

Cidan• 10 years ago

We've been in the EFS preview for quite a while now, and we are incredibly pleased with it. We do a few different things using EFS, including using it as an output for some of our Spark jobs.

It's really a top notch product; couldn't be happier.

kldavenport• 10 years ago

Pricing looks competitive https://aws.amazon.com/efs/pricing/ I wonder if this helps those who perceive S3 as vendor lock-in since it replicates a traditional file system.

LeoPanthera• 10 years ago

Does it? It looks significantly more expensive than, say, rsync.net.

fulafel• 10 years ago

Any ideas what they are using for NFS servers software?

boraturan• 10 years ago

Seems like no Windows support?

Corrado6• 10 years ago

That's a good question. Originally it was Linux only as the EFS client was locked to NFS 4.0 and the Windows Server NFS client was 4.1 (why 4.1 is not backwards compatible, I don't know). However, the announcement mentioned using the PowerShell cmdlets to attach EFS volumes, and in fact the PowerShell help pages have specific commands for working with EFS. So, my guess is that it is now available for Windows and Jeff didn't do a very good job of bringing that to light.

rincebrain2• 10 years ago

The EC2 example shows mounting with nfsvers=4.1, so I'm guessing it got resolved by bumping the version supported.

objectivefs2• 10 years ago

Cool! If you are looking at shared filesystems on EC2, you might also want to take a look at ObjectiveFS[1]. Works in all regions and on GCS, and you can mount your filesystem securely (end-to-end encryption) between regions.

[1] https://objectivefs.com

arqq• 10 years ago

ObjectiveFS looks awesome and I've been wanting to use it, but using non-S3 storage is locked to "enterprise, I can't afford it". :(

But I also can't afford S3's bandwidth pricing.. I could, however afford to run an S3-API compatible service myself with bandwidth pricing I could afford.

But then you wouldn't let me use it because then I'd be "enterprise".

objectivefs2• 10 years ago

Great that you have been wanting to use ObjectiveFS. We are happy to talk with you about your non-S3 storage use case and see if we can find a plan that works for you.

merb• 10 years ago

I wonder if some can run a postgresql database on top of that. This would make it relativly easy to have a master without downtime and be way cheaper than RDS.

Sanddancer3• 10 years ago

It claims to present itself as a full filesystem so I don't see why it wouldn't, however, you're probably going to run into pretty hardcore performance issues, because the bandwidth amazon provides in general is on the stingy side, but under a terabyte, they're downright miserly, with a 10 gig filesystem only able to provide 0.5 MB/s sustained. This product is problematic at best.

Someone3• 10 years ago

Also, it is a full-fledged NFS-mounted file system.

I would be very wary of running a database from that.

Andys• 10 years ago

How exactly is it "without downtime"?

Florin_Andrei• 10 years ago

It's essentially NFS, so yes.

Sanddancer3• 10 years ago

This feels rather...primitive for a network filesystem introduced in 2016. Where are the snapshots? Where are the subvolumes? Where's the ability to send/receive volumes/subvolumes to filesystems in other availability zones? There are also I/O limits if you have a smaller filesystem with frequently accessed data, an inability to mount on machines through vpn gateways, and a lot of other seriously rough edges.

It's an interesting start to a project, but the public features feel very much lacking.

rincebrain2• 10 years ago

It seems like this is more of a glue layer between a more interesting backend service (EFS) and clients, but I don't know that I'd expect Amazon to do something like write a native client, versus just probably improving the NFS client behavior on Windows/Linux/...

The neat trick would be if NFS 4.X grew support for communicating interesting operations like snapshotting.

KaiserPro5• 10 years ago

YES

Finally we can start using standard posix semantics for things.

Yes people might say its primitive, but currently all I can see is people trying to re-create shared files systems over protocols not designed for it cough cough HTTP

Finally I can have a shared home, with a shared environment.

A single readonly binary source (great for docker by the way.) also great for ensuring one version of scripts, without having to ansible everything.

jdub• 10 years ago

Congratulations, you've won a single point of failure!

willejs2• 10 years ago

HAHAHA. YES! So much this. THere are legit use cases for this, but there are many use cases that will cause you much pain.

illumin82• 10 years ago

It's a single point of failure that's highly available across multiple availability zones. I'm not sure I would even call it a single point of failure when it's running in multiple datacenters. If you need geo-redundancy, rsync it to another region nightly.

merpnderp• 10 years ago

At 10x the price for S3.

acdha2• 10 years ago

That's not a valid comparison for many applications: if you need POSIX filesystem semantics, it doesn't matter how much S3 costs because it cannot do the job, any more than you'd say S3 costs more than a USB key.

KaiserPro5• 10 years ago

You are correct. However its not for the same things.

S3 is high latency, HTTP semantics, no locking mechanism.

EFS is low latency, high throughput shared file system with standard posix interface. everything can write to standard files. Not many things can write using S3 effectively.

ngrilly2• 10 years ago

To store small files (10 to 1000 KB) uploaded by users in a web app, what is the best tool, S3 or EFS?

gedrap• 10 years ago

A benefit (usually) of S3 is that since it's operating over HTTP, you don't need any additional things to make the files accessible. In case of EFS you would need a HTTP server running. Maintaining a HTTP work is extra work, extra configuration but might give you additional benefits.

Thinking of small file size, EFS would probably work a bit faster (since you'd have some HTTP overhead) but I doubt that it would be significant, unless users upload hundreds of files at once.

EFS advantage in this case would be that you'd no longer need a S3 library to do it, since it's POSIX compliant. Given that most languages have simple, solid libraries for interacting with it, it's not a huge difference but still :)

All in all, I'd go with S3 unless there are some special requirements that are hard to satisfy using current S3 features.

RossM• 10 years ago

Object storage (i.e. s3+cloudfront) is still the correct way to store immutable files. that said, I know this will be used when the options are to rewrite filesystem access to an abstract (i.e. fs/s3/azure/etc) or just use nfs. People already use s3fs-fuse[1] which is dubious at best.

[1] https://github.com/s3fs-fuse/s3fs-fuse

ngrilly2• 10 years ago

Would you say that the main choice criteria is immutability?

Immutable files => Object storage => S3

Mutable files => File storage with POSIX semantics => EFS/NFS

cshimmin• 10 years ago

I have an application where users upload hundreds of small (~10kB) files every day. We quickly realized that S3 was not an option, since you get charged per request. As a workaround we batched the file uploads by archiving uploads each hour, and uploading a single .tar.gz to S3 at the end of the hour. The aggregated data gets processed offline later.

We have N upload endpoints (as part of an autoscaling group), it's a bit of a pain to reaggregate the data since each server uploads its own .tgz to S3. I'm very happy to have EFS now! It makes scaling out our upload endpoints much simpler, as they can just dump their data into a common directory (which can still get archived and persisted to S3 later).

s17tnet• 10 years ago

Finally! My client was unable to use AWS for lacking of real POSIX fs.

jakozaur2• 10 years ago

Price is a bit high $0.3 GB / month vs. $0.1 GB / month for SSD EBS and even less for local storage. S3 is even cheaper $0.03 GB / month. So EFS is 10 times more expensive than S3.

On I/O front also EBS (gp2) and (st1) are several times cheaper.

So most of EFS advantages are in flexibility and sharing same file system across many instances. This may be great e.g. for sharing readonly binaries, but large scale data pipeline likely can be done much cheaper using different technologies.

KaiserPro5• 10 years ago

You are right for small scale things.

I'll have to double check my working out, but you're only paying for what you use, not provision, which is one big things.

Also, if you have > 3 machines working from the same dataset (ie docker image/video/large binaryblob) there is an instant saving, without factoring in provisioned vs useable space with EBS

but, your original point, as a 1-1 replacement of a properly sized EBS mount, is correct, its more expensive.

pmalynin• 10 years ago

Nitpick

"large binaryblob" is literally "large binary binary large object"

Corrado6• 10 years ago

I think EFS is meant for large read-heavy patterns, at least that's what we used it for. For example, I help run an e-commerce site that has a lot of content (JS, CSS, images, etc.) that we serve up from 10 dedicated servers as well as 30 general servers. When we release new content we have to update each of the servers and make sure that everything got installed properly everywhere. With EFS, we just have to update one thing and everyone gets it automatically.

lucaspiller• 10 years ago

Why don't you use S3 and Cloudfront to serve all the assets?

saurik• 10 years ago

On I/O front also EBS (gp2) and (st1) are several times cheaper.

Isn't I/O for EFS free?

jakozaur2• 10 years ago

You don't pay extra for IO on EFS and some types of EBS (like mentioned here gp2). However, for some applications you optimize your storage to get most IO per $. In that terms EFS comes with some IO capacity as well as EBS. The point is that EBS is way cheaper in that use case.

Corrado6• 10 years ago

I am excited by EFS but during the beta it didn't perform as well as I would have liked. I had a small Wordpress install that I wanted to make elastic in that I could have multiple web servers and all of the data was stored apart from the AMI. After testing my setup I found that latency was way too high and the performance of my site was terrible. I'm guessing it's because the amount of data was really low (100MB) and the requests were infrequent.

I ended up going with a shared volume, which works fine and performance is great, but I can only attach the volume to a single instance at a time which prevents me from running multiple instances. That said, it looks like there has been some performance tweaks to EFS so maybe it might be better this time.

lsaferite2• 10 years ago

Was it not possible to use the S3 plugin for WP to offload all the content files to S3 + CloudFront?

jeffbarr2• 10 years ago

We addressed this use case during the preview. Could you try it again now?

jtrtoo• 10 years ago

I think you might be going about solving this problem in a sub-optiimal way. WordPress data consists of db + asset files. Db gets addressed outside of this and asset files either go in s3 or are synced (rsync, etc)

TheGuyWhoCodes• 10 years ago

Does EFS provide better throughput and latency than S3?

bhouston• 10 years ago

Are there any third party performance metrics yet? Read/write ops per second, and read/write bytes per second from one or many machines?

Would love to see this in comparison to a 10Gbps (or 1Gpbs) attached spinning rust or SSD NSF drive. That would really help me understand what tradeoffs are happening here.

deskamess• 10 years ago

Can you expose the EFS file systems to CloudFront? For a future migration (local to cloud) I would like a backend application running on EC2 to use standard path names, but public access would be through CloudFront. It would be easier to migrate existing apps to the cloud with that approach, and then slowly work on changing the apps to use S3 (for cost reasons).

sheraz• 10 years ago

I think it is worth mentioning that Azure offer something similar with Samba / CIFS/ SMB protocol. [1]

This is the underlying tech that powered their Docker volume plugin as well [2]

I'm using in production for serving up small images to some web servers, and I'm currently in playground with docker stuff.

So far, so good. I'm impressed.

[1] https://azure.microsoft.com/en-us/documentation/articles/sto...

[2] https://azure.microsoft.com/en-us/blog/persistent-docker-vol...

edit: footnotes

aluskuiuc• 10 years ago

A notable difference is that Azure file shares have a 5 TB limit for the whole share, and a 1 TB limit on the size of any given file [1], while EFS has no limit on the size of the file system, and a 52 TB limit on the size of any given file. [2]

Disclosure, work for AWS.

[1] https://azure.microsoft.com/en-us/documentation/articles/sto...

[2] http://docs.aws.amazon.com/efs/latest/ug/limits.html

agperson• 10 years ago

Glad to finally see this in production, wish it was available across VPC peering, VPNs, etc.

gtirloni• 10 years ago

I don't have a lot of experience with pNFS but it's supposed to be an extension to NFS 4.1, which Amazon EFS is supporting. However, pNFS is not mentioned in this blog post, their website or the official docs.

That would be an interesting feature.

api• 10 years ago

This is pretty awesome.

Even though we don't use AWS much due to lack of IPv6 support, we've wanted to have a giant bottomless FS for various purposes. It'd be easy to set that up in here and export it over a virtual network.

spullara• 10 years ago

I really want to be able to access EFS file systems from Lambda. Has anyone figured out a way to access NFS?

rymohr• 10 years ago

Still no support for incremental snapshots though right?

mike503• 10 years ago

Why hasn't anyone mentioned ObjectiveFS? When EFS was in preview just a month or so ago it was absolutely abysmal with performance for basic smaller web stuff.

OFS feels nearly like an attached disk, and uses S3. You only pay per mount. Their support has been awesome and very personal.

It provides a fuse-based full POSIX filesystem backed by S3 and can be mounted from multiple places in multiple regions. Some basic things like snapshotting they said are coming soon too.

I gave up on EFS and even though it's finally out of preview, I think I still am going to prefer OFS based on what I've seen so far...

Amazon Elastic File System – Production-Ready in Three Regions

Related Stories