AWS Public Datasets

https://aws.amazon.com/public-datasets/

by darshanrai • 8 years ago

390 45 8 years ago

aws.amazon.com

marcinzm• 8 years ago

Interesting, they used to have Wikipedia data and now it's gone from the list, anyone know why?

arxpoetica• 8 years ago

Also curious...

sitkack• 8 years ago

Google Cloud has something similar, https://cloud.google.com/public-datasets/

I think it would show a kind of gilded age maturity if all the cloud providers cooperated on their public datasets, because they are for the public good.

vgt• 8 years ago

And the first terabyte of analysis for free in BigQuery!!

(Work at g)

diggan4• 8 years ago

That's all fine and dandy, but I assume I could also pull down the entire dataset for free and run unlimited analysis locally for free?

CobrastanJorji• 8 years ago

Hacker News is a member of this dataset, but it's only updated daily. If you want to get meta about this discussion, I guess you'll need to wait until tomorrow.

https://cloud.google.com/bigquery/public-data/hacker-news

chrisbaglieri• 8 years ago

A lot of these datasets have been available for some time: https://aws.amazon.com/blogs/aws/new-aws-public-data-sets-tc.... Perhaps what is most surprising is that the list hasn't grown a ton since then.

yellowbkpk• 8 years ago

As someone who has spent a considerable amount of time on data that has ended up on this page, I think the fact that the list hasn't grown says more about the priorities of other companies than of AWS. Amazon doesn't (yet) have time to build and maintain these datasets themselves: they work with others to build and maintain it and then fund the storage and transmission fees.

I helped build the Terrain Tiles dataset as part of Mapzen, which recently shut down. The OpenStreetMap data exists on the AWS Public Datasets page because it's useful to Humanitarian OpenStreetMap Team. If you're able to convince your company to generate and work with a public dataset, consider reaching out to the AWS and Google public datasets teams to get it hosted and publicized.

neuromantik80862• 8 years ago

In my experience, the list that AWS keeps on their public page isn't completely up to date. There are two major datasets in the neuroimaging community that are hosted in S3 (see my comment elsewhere on this page), but AWS hasn't widely publicized this fact for some reason.

Mizza2• 8 years ago

If anybody is interested in AI/bioinformatics projects on AWS, I'm currently involved in a project to harmonize _all_ of the publicly available RNA data (many petabytes) into easily sliced, AI/ML-ready datasets for anybody to use:

Website: http://www.refine.bio/

Source: https://github.com/AlexsLemonade/refinebio

eggie2• 8 years ago

Would this allow rapid realignment of the data against a given reference (graph?) model? Or is the alignment somehow baked in?

Mizza2• 8 years ago

We do the alignment too, yes.

Nashooo• 8 years ago

Very interested. Looking forward to following this!

dekhn3• 8 years ago

How in the world do you have "petabytes" of RNA data?

vamin2• 8 years ago

An RNA sequencing run generates on the order of 10GB of data, a typical study requires many runs (treatments, controls, replication of results, etc), and posting the raw data is required by most biology journals. I'm not surprised that there is over 1PB of data available to curate.

yinyang_in• 8 years ago

Anybody to use, who is paying for bandwidth here?

gajju3588• 8 years ago

Adding one more source of open datasets in the thread: Manually tagged high-quality datasets :

https://dataturks.com/projects/trending

ah-• 8 years ago

Seems to be down. What do they have?

benhamner• 8 years ago

Over 13,000 community-uploaded public datasets: https://www.kaggle.com/datasets

(I work at Kaggle)

santiagobasulto• 8 years ago

I take the chance to ask you: is there any command line utility similar to pip to import datasets from kaggle? Something like:

`kik pull titanic`

Godel_unicode• 8 years ago

https://github.com/Kaggle/kaggle-api

minimaxir• 8 years ago

How much does it cost to export/process this data from Amazon? Unlike GCP/BigQuery which has free cloud processing built in, downloading/analyzing these GB/TB datasets for personal analysis can’t be cheap.

The descriptions note “Educators, researchers and students can apply for free promotional credits to take advantage of Public Datasets on AWS.” which is not a good sign.

pletnes• 8 years ago

I guess they want you to run your analyses on AWS directly.

keypusher• 8 years ago

AWS has free tier, in this case 1500 hours free of a DC2.Large on Redshift.

isseu• 8 years ago

You can use Athena, is cheap and easy to use

sjburt• 8 years ago

It's the normal data cost. EG the Landsat data is stored in S3, so it's free to EC2 (in the same region), and somewhere around $0.09/gb to the public internet.

However, the idea is not that you download it all (there's probably cheaper ways to acquire Landsat data), the idea is if you want to do whatever analysis on AWS, they've already got it neatly ingested for you.

pacificleo11• 8 years ago

Importance of data for machine learning algorithms can’t be stressed enough. I remember talking to a friend in Google Translate team. they had a good also but they were struggling to get quality translation data to train their service. the problem was more severe when it came to language which was not very popular. translation set was next to nothing for Say something like Turkish, Hindi, Latvian etc.

They finally solved this by using meeting notes from UN Assembly. Which were transcribed by best of the translators? that access to meeting transcription was (unfair ??) advantage Google had over other tools. Was it wrong ? I don’t think so. Should have been those meeting notes be public: Yes

solarkraft• 8 years ago

The German ML translation company of ex-googlers called DeepL (formerly Linguee) providing the currently best machine translation service apparently got a lot of their data from (I assume public) EU documents which had to be translated to all member languages.

pavlov• 8 years ago

When Google Translate was new in my language Finnish, their translation quality was atrocious.

I tried this sentence: "Hei äiti, puhun suomea". The expected translation would be "Hi mom, I speak Finnish".

Instead Google's result was: "July's mother, I speak English".

Obviously the engine had been trained on unvetted data sets where the word "English" occurred in translations in a position where the original had the word "Finnish", and no context was provided to avoid this kind of mistake.

The word "July" came about because "hei" is also used as an abbreviation for "heinäkuu" (July). It was sobering that a supposed world-class AI couldn't distinguish between these two usages. Machine learning needs a lot of old-fashioned handtuned human-made heuristics.

komali22• 8 years ago

Sobering perhaps to learn what the state of the art is for "world class" AI indeed ;)

Cortana when??

elorant• 8 years ago

I wonder if anyone has processed Common Crawl's dataset for a list of all the existing domains (I don't care if the data are old).

RobAley• 8 years ago

You can get the URL data (and parse out the domains yourself) at http://index.commoncrawl.org/ Be aware that commoncrawl, while huge, only crawls a fraction of the web, and so you won't get all of the "existing domains" (i.e. all domains that exist) if that is what you are after.

neuromantik80862• 8 years ago

This doesn't appear to be publicized on the linked to page, but two of the major open access fMRI dataset repositories (FCP-INDI [1] and OpenfMRI [2] are hosted in S3.

[1] http://fcon_1000.projects.nitrc.org/ [2] https://openfmri.org/

tracker1• 8 years ago

I wish they'd centralize WHOIS data and clean it up... that is some messy stuff there.

AWS Public Datasets

Related Stories