Ask HN: Where can I host a large data-set (35GB) in public?

https://news.ycombinator.com/item?id=1617618
by donohoe • 16 years ago
10 12 16 years ago

I've been scraping various parts of a web site since 2008 and have accumulated an interesting data set (tracking changes every hour, and in some cases every 6 minutes) covering published RSS, markup changes etc.

The data is cool and I think people would have fun analyzing and making mashups. It consists of JSON and HTML fragments with a simple API of sorts to navigate:

  2010/index.js
The content of "index.js" is an array showing what months, within 2010, are available:
  [ "01" , "02" , "03" , "04" , "05" , "06" , "07" , "08" ]
Knowing the Months, you can then look up the days:
  2010/08/index.js
  [ "01" , "02" , "03" , "04" , "05" , and so on... , "17" , "18" , "19" ]
Within each day you can check the Hours:
  2010/08/19/index.js
  [ "00" , "01" , "02" , "03" , "04" , "and so on... , "09" , "10" ]
Within each hour, you can get the JSON file:
  2010/08/19/10/index.js
and so on and so on...

Freebase and Amazon Public Data Sets have been suggested but don't look like a good fit. Right now the file size of the 3 TAR files is about 35GB aprox.

Any further suggestions?

Related Stories

Loading related stories...

Source preview

news.ycombinator.com