How to directly upload files to Amazon S3 from your client side web app

http://codeartists.com/post/36892733572/how-to-directly-upload-files-to-amazon-s3-from-your

by tadruj • 14 years ago

194 36 14 years ago

brettcvz3• 14 years ago

Alternatively, use Filepicker.io.

filepicker.store(myFile, {location: "S3"}, function(FPFile){console.log("File stored in S3 at "+FPFile.key+" and available at "+FPFile.url);});

tadruj4• 14 years ago

filepicker.io is a little bit more heavy-weight on the browser. But I really like it. It's THE start-up of 2012 for me.

This post was intended for the people who want to have a specific and stand-alone building block just for S3.

philfreo3• 14 years ago

unfortunately they don't return the full url to the file on YOUR s3 bucket (but you can piece it together)

brettcvz3• 14 years ago

Our concern is that if we return the direct S3 url, people less experienced will assume that the S3 url should work by default, when it will 403. So we return Filepicker.io urls which alias to your S3 bucket

templaedhel• 14 years ago

I came here to say this exactly, and was a little disappointed to see the mixed bag of filepicker impressions.

Filepicker is awesome.

At my job I just had to implement direct to s3 uploads, and filepicker cut down the work by 10x. It's basically a one line implementation to save the contents of a file input to s3. That's awesome.

The pricing is extremely liberal, and the docs are good. I haven't used many of the more advanced features, but even at its simplest form it's an excellent tool.

As for the complaints here, they seem to be mostly pedantic - I would much rather concat a few strings to build the proper s3 URL when the alternative is building the whole upload infrastructure, and once again, that's without considering all the advanced features filepicker gives you.

JuDue• 14 years ago

Filepicker is a great idea, especially if you want to alter the image size or make thumbnails without the hassle of setting up the processor.

Just with their image manipulations were a bit more fully featured, like imagemagic

modarts2• 14 years ago

Tangentially related, but does anyone know of an easy way to push a github repo to an S3 bucket (configured as a public site)?

I'm pretty new at git/github, but understand that the solution may involve a post commit hook that somehow calls an S3 file upload service.

kmfrk• 14 years ago

One way is to use something like vaporfile (https://github.com/EnigmaCurry/Vaporfile) to create a post commit hook that lets vaporfile push the new files to S3.

It doesn't do delta updates, which could be a problem depending on the kind of project, though.

gdubs• 14 years ago

I was actually reading up on this earlier; I was led to jgit http://stackoverflow.com/questions/7031729/publish-to-s3-usi...

modarts2• 14 years ago

Sorry, I should have clarified a little better. I'd like to avoid installing anything on a server to make this happen.

(My development workflow/environment is completely decoupled from any servers I control: I use Parse.com as my backend with static HTML pages served from S3, and cloud9 as my IDE, with Github as my source host)

vinhboy2• 14 years ago

Last week I was trying to figure out how to do this with dropbox. Anyone know a way?

tadruj4• 14 years ago

Dropbox has a filepicker https://www.dropbox.com/developers/chooser , you can also use filepicker.io for more services support. But Dropbox doesn't support per-request CORS as I read on their Forums.

vinhboy2• 14 years ago

G@#$D!T I wished I had known about this last week. I already implemented my own file browser! Thanks for the tip though.

mjtokelly2• 14 years ago

Direct uploading like this has been possible for at least two years. However, in our tests it turned out that Amazon was silently losing some files >100MB (and almost all files ~1GB).

On the AWS forums it was clear that AWS was aware of this consistent problem, but was not able to fix it, and not willing to document it.

They eventually released "multipart upload", which remains the only reliable way to get large files to S3. Unfortunately, multipart upload is nearly impossible to implement as a web app (short of resorting to e.g. Java).

kzahel• 14 years ago

Multipart is not at all impossible from a webapp (depending on your browser requirements). FileReader API is available across all browsers (except IE < version 10, which I admit is a big problem). For IE, you can resort to some other solution (perhaps Silverlight, or Flash?)

carson3• 14 years ago

I will second this. At the bottom of this article there is a reference the article I wrote a while ago when Amazon released this feature. I was planing on doing multipart uploading at that time using the FileReader but there was a bug in the way S3 did CORS so I didn't want to continue until that was fixed. They fixed it and I never came back to it. Maybe be a good time to try it again. Resuming a partial upload seemed like a good win to me.

tadruj4• 14 years ago

Our video bucket is a few TB in size right now, file-sizes ranging from 10MB to 1GB. Not one file/upload broken until now. So I guess they fixed it.

akoumjian• 14 years ago

Direct uploading like this has not been possible through XHR as CORS support was only added this past year. However, you could mitigate it by handling the CORS headers through a proxy, using flash, or a couple other methods.

You could, however, perform a traditional POST.

tomsaffell• 14 years ago

>Unfortunately, multipart upload is nearly impossible to implement as a web app

I am right now writing a library to do just that. I already have a prototype working, and we'll probably open source our library once it's reasonably tested. But perhaps you know something I dont... is there anything in particular that makes it "nearly impossible"?

mjtokelly2• 14 years ago

Sounds like my understanding must be out of date as of this year! (Which makes me very excited about your upcoming library.) My conclusion was based on research ~1 year ago finding no OSS or commercial software supporting it, and forum posts for those projects by developers sadly explaining why they just couldn't do multipart yet.

My bad experience with files disappearing was entirely with web-based POST requests, ~2 years ago. Large file transfer from EC2 to S3 was reliable, but our POST requests on slower connections (even those that were very reliable) would return with a false report of success.

mattdeboard• 14 years ago

What? Why is it hard? I just wrote a web app a couple weeks ago to enable my co-workers to upload large files (.5-1.5GB) to S3. Like, I don't doubt you, I'm just curious why my experience was so different than what you describe.

philfreo3• 14 years ago

Thanks for posting... I was looking for something just like this:

https://github.com/tadruj/s3upload-coffee-javascript

tlrobinson• 14 years ago

I really like CoffeeScript, but do we really need to advertise every JavaScript library as a CoffeeScript library as well?

Feel free to use CoffeeScript to implement your libraries, but lets just call it a JavaScript library and assume everyone who uses CoffeeScript will understand they can use any JavaScript library.

tadruj4• 14 years ago

I think we need to. It deserves a credit, because we use it and we didn't pay for it. CoffeeScript makes our lives easier, almost eliminates bad practices and removes brace clutter from the screen, making the code more readable, understandable, better.

When having to choose between a library written in JavaScript and a library written in CoffeeScript I choose the latter, because it statistically has a probability to be a better library, just because the coder didn't have a chance to use bad JavaScript practices, when writing it. Assuming I usually don't have time to read the code of the libraries I'm using.

pud• 14 years ago

Any recommendations for how to show a progress-indicator to the user during a direct-to-S3 upload?

carson3• 14 years ago

I wrote the example referenced at the bottom of the article and it has an example of indicating progress to the user. You can see it in app.js here: https://github.com/carsonmcdonald/direct-browser-s3-upload-e...

pswenson3• 14 years ago

Coolest would be an ability to have 1) a great filepicker UI (drag target + multi file upload w/ progress bar) 2) resizing/thumbnails in the browser 3) no server side other than s3

guessing #2 isn't possible?

carson3• 14 years ago

#2 is possible but #3 isn't unless you have a completely open bucket and that wouldn't be good.

brettcvz3• 14 years ago

I hate to be so self-promoting, but http://jsfiddle.net/GTYX5/2/ should do everything you're asking for

pswenson3• 14 years ago

how do you do #2?

ayushgta• 14 years ago

Thanks for posting this. I was looking at adding uploads just this morning, so this is very timely. It looks like if you combine the direct upload to S3 as outlined in this article with bitline (http://www.blitline.com/), you get some pretty cool abilities. I especially like that bitline supports batch operation (for example create five sizes of my image) in one call.

secabeen• 14 years ago

Here's the other page on the AWS documentation on this: http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentic...

dgoodlad• 14 years ago

I did a demo similar to this a few months ago:

https://github.com/dgoodlad/s3-photo-upload-demo

http://s3-photo-upload-demo.throwawayapp.com

How to directly upload files to Amazon S3 from your client side web app

Related Stories