Ask HN: Cheapest way to host and serve large-ish files?

https://news.ycombinator.com/item?id=14764015

by ageitgey • 9 years ago

6 7 9 years ago

news.ycombinator.com

What options exist for serving large files on the web without getting killed on bandwidth costs?

For example, let's say I want to host a 5GB file that contains an educational ML data set on S3. Assuming 1000 people download it in a month, that would cost me $450/mo in just outbound bandwidth fees. That's almost $0.50 per download.

Obviously there must be ways to serve files cheaper than physically mailing an SD card to someone. What have you done to serve large files cheaply? Is the only option to maintain a server (either colo or virtual) that has better bandwidth pricing?

icebraining• 9 years ago

Well, there's Backblaze B2, it would only cost $0.10 per download: https://www.backblaze.com/b2/cloud-storage.html

Alternatively, there's bittorrent, which was pretty much designed for this.

But considered you can find decently priced dedicated servers with unmetered gigabit ports, seems like installing Linux + nginx and enabling automatic security updates might still be the best option.

weinzierl• 9 years ago

Amazon has Public Data Sets on AWS. I don't know the current status, but it used to be free for eligible projects [1].

BitTorrent would be the natural thing to do, but it has it's own set of issues. Jeff Atwood summarized them nicely in [2]. This was written in 2009, LegalTorrents doesn't exist anymore and you don't need a tracker anymore, but the rest of the points are still valid. His summary is:

It's ironic that BitTorrent, which is justly famous for equalizing the highly asymmetric nature of most people's internet connections, is itself so asymmetric when it comes to sharing: trivially easy to consume, but awkward and confusing to share. That's too bad, because BitTorrent is such a powerful tool for sharing. Hopefully this post demystifies the process a bit!

[1] http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=irol-ne...

[2] https://blog.codinghorror.com/sharing-files-with-bittorrent/

EDIT: Thinking about it, there is no shortage of free public data sets. I believe many of them are provided by individuals and organizations which wouldn't be able or willing to pay the bandwidth cost out of their own pockets. Maybe you could research the solutions they are using. A starting point could be the [3].

[3] https://news.ycombinator.com/item?id=2165497

ageitgey• 9 years ago

I have used Internet Archive in the past for this kind of hosting. They charge nothing and provide a great (though sometimes confusing) service.

But I'm all for paying my own bills as long as they are reasonable. It seems crazy that IA can afford to host huge files for free for anyone but S3 charges so much for so little. I was hoping there was a middle ground. I guess the middle ground is just running my own dedicated server. Thanks for the reply!

mozumder• 9 years ago

Static business home IP address from your ISP and your own server.

lovelearning• 9 years ago

1 TB storage on Google Drive is $10 (100GB is $2). No bandwidth pricing.

sixhobbits• 9 years ago

To add to the other comments about using bittorrent, Academic Torrents [0] is trying to solve this problem. I don't think they're that popular yet, but they aim to address some of the issues with bittorrent already highlighted in the other posts.

[0] http://academictorrents.com/

ieee8023• 9 years ago

I think averaging 2TB of data served a day is popular!

Ask HN: Cheapest way to host and serve large-ish files?

Related Stories