Yahoo Releases the Largest-Ever Machine Learning Dataset for Researchers

http://yahoolabs.tumblr.com/post/137281912191/yahoo-releases-the-largest-ever-machine-learning

by tchalla • 11 years ago

397 76 11 years ago

yahoolabs.tumblr.com

frik• 11 years ago

Sad the future of "Yahoo!" (the tech company, not the Alibaba stock) is uncertain. They were always very open with their research. Thinking back to 2008/09 they had the biggest Hadoop clusters, etc. even the first edition of O'Reilys Hadoop books says "Yahoo press".

alceufc• 11 years ago

Flickr was -- and maybe still is -- very useful for the computer vision research community.

tedd4u• 11 years ago

This is old news but we put together a 100M item dataset from Flickr as well, all Creative Commons licensed, including lots of metadata from users as well as some pre-computed features. It's called the YFCC100M.

http://yahoolabs.tumblr.com/post/89783581601/one-hundred-mil...

TeMPOraL• 11 years ago

How do they use it? Does it have a good programmatic access? Because UX-side, I'm surprised they still exist. I don't know of a single photo-related web site that has worse UI and is more annoying to use than Flickr.

mintplant• 11 years ago

Flickr is great for finding Creative Commons-licensed images as well.

aswanson• 11 years ago

Amazing...did not know that. Thanks.

magicmu• 11 years ago

As a brand, I'm honestly impressed that they've lasted through so many sea changes.

blazespin2• 11 years ago

That's really irrelevant here. I think we should focus on what an incredible contribution this is. Perhaps a sign of good things to come from Yahoo.

wahsd2• 11 years ago

I hope they revive Yahoo tubes, if not as an actual tool, maybe as a teaching device. I was sad to see it go even though it was seriously falling apart at the end.

guelo• 11 years ago

The owners of Yahoo don't care if there are more good things to come or even if Yahoo is gutted and killed as long as they don't have to pay taxes on their windfall Alibaba gains.

BetaCygni• 11 years ago

I'm torn. I love open data, but I fully expect that someone will (partially) deanonymize this.

rectang• 11 years ago

I share your concern.

Once data like this is deanonymized, it's out there forever -- there's no going back to fix it like you would a software bug. So you need perfect understanding and provable security at release time to guaranteed safety into the indefinite future. That's not an easy constraint to satisfy.

joatmon-snoo• 11 years ago

The thing that terrifies me specifically is that there's been work done - I believe it was a branch of the US military studying network traffic patterns - showing that you can reconstruct profiles based on behavior patterns and link them back to the original user with high success rates.

fweespeech• 11 years ago

Yep, and this is why stuff like this should have a formal opt-in process.

japaw• 11 years ago

Probobly. Hopefully so have they learned from the fallout from the AOL search log case ( https://en.wikipedia.org/wiki/AOL_search_data_leak ). That case was certainly a big mess.

jacquesm• 11 years ago

I'm not sure why you're downvoted, the AOL search log case was a huge mistake on the part of AOL and I'm quite surprised that Yahoo! would take a risk like this. The real risk is never in just this data but in combining it with other public datasets. I haven't looked at what is in this particular dump in detail but if there is data that had to be anonymized (as they claim) then you can bet that there will be people already busy trying to reverse that.

glxc• 11 years ago

13.5 TB - that is pretty huge!

Great to get some truly "Big Data" sets out there. I consider "Big Data" to be data that can't be conventionally processed on a commodity machine, else it's just analytics

Yahoo must be applauded for supplying various data sets and helping progress machine learning research

collyw3• 11 years ago

I saw a course advertised in my email yesterday. Big data with MySQL. The description talked about queries and aggregate functions. That isn't big data - that's just "using a database" before the term "big data" appeared in the mainstream.

sbarre• 11 years ago

Ugh.. I had a boss years back who insisted on using "big data" to refer to our analytics and reporting work (which was nowhere near big data in terms of data size - we had maybe a million rows in our database across all our tables), and I fruitlessly tried for months to explain to him that anyone who really knows what "big data" means would immediately see through his bullshit..

collyw3• 11 years ago

https://www.coursera.org/learn/analytics-mysql?utm_medium=em...

Ok, they are comparing MySQL to Excel, so in relative terms the data could be bigger...

rodionos2• 11 years ago

Note that they're sharing this dataset only with *.edu, which is unfortunate for the rest of us. I wish they would allow access to a fraction of the dataset, e.g. 5% of records, for the rest of the community.

boomzilla• 11 years ago

Would it be suffice with an .edu email, or does one need a formal document from university officials? I tried to click through from the sandbox link, but a Yahoo account is required.

nolok• 11 years ago

Do you mean to technically get access, or to be legally allowed to use it ? Because for the latter, merely having a .edu isn't enough.

Namrog84• 11 years ago

Most places only care that you were a student with active edu. I graduated years ago and I still use the edu without problem. Most things say you need to be a student from university. Which I am/was. Very few places say actively enrolled student. It's a legal gray area in my opinion.

rodionos2• 11 years ago

I have a lifelong edu address but it's not what they mean of course. My reading is that you have to be an enrolled student or researcher.

hellameta3• 11 years ago

To clarify from source[1]

TO BE ELIGIBLE TO RECEIVE WEBSCOPE DATA, UNLESS SPECIFIED IN A PARTICULAR DATASET, YOU MUST: Be a faculty member, research employee or student from an accredited university Send the data request from an accredited university .edu or domain name (for international universities) email address

UNLESS SPECIFIED IN A PARTICULAR DATASET, WE ARE NOT ABLE TO SHARE DATA WITH: Commercial entities Employees of commercial entities with university appointment Research institutions not affiliated with a research university

[1] http://webscope.sandbox.yahoo.com/

knughit2• 11 years ago

Since nowadays universities are run as start-up accelerators with closed source software, the .edu restriction is antiquated and obnoxious

dave_sullivan• 11 years ago

To yahoo: that seems like an unbelievably lame restriction. Even aside from commercial entities, there are many many people working to improve machine learning with no university affiliation.

You're basically taking something really cool here and shooting yourself in the knee with a shotgun.

I hope other companies don't think open data initiatives count if they're not actually open. If you want to keep your data internal and top secret, totally fine, but open data should be available to anyone or it doesn't count.

Almost yahoo, almost...

Edit:

> I didn't see the word "open" mentioned once in this article...

Touché sir. Sentiment still stands: "released to researchers" and "released to the public" should not be different things.

corp-a• 11 years ago

I didn't see the word "open" mentioned once in this article...

mrfusion• 11 years ago

That really sucks. I hate the whole attitude that you need to be a Ph.D. To do research.

criley22• 11 years ago

I mean that's what Ph.D's do in most fields, the whole point of getting a Ph.D is so that you can do research in that field. Learning how to write well researched white papers that can withstand peer review.

Prior to that, you can do some research as perhaps with a Master's or as a graduate student, but usually you're assisting on a Ph.D's project.

The process of getting a Ph.D in many fields is basically the process of learning how to research and report your research.

So the idea of limiting high level research tools to those who've proven to be capable of high level research isn't that ridiculous to me.

If you want to research for a living and report it, is it so ridiculous to ask that you train traditionally in researching and reporting?

Estragon• 11 years ago

I imagine you'll be able to torrent it fairly soon.

mooreds• 11 years ago

It is actually 1.5TB compressed. Direct link to the dataset:

http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did...

"The dataset may be used by researchers to validate recommender systems, collaborative filtering methods, context-aware learning, large-scale learning algorithms, transfer learning, user behavior modeling, content enrichment and unsupervised learning methods."

Edit: added quote

astazangasta• 11 years ago

I am so sick of the implication that all data is equivalent, and there is some generic notion of "big data" that we generic "data scientists" can learn how to "mine" using some generic technique called "deep learning" that will give us all the answers we need like some kind of oracle.

I study biology. The shape of the data, the way it is structured, the problems we face in analyzing it, are quite different than the ones faced in user-news interaction data. Techniques that are useful for reshaping and summarizing one dataset are not necessarily applicable to another.

blazespin2• 11 years ago

Word2Vec!

jedberg• 11 years ago

It's too bad they aren't publishing this as an EBS snapshot. That would probably be the most useful way their intended audience could consume it given that most universities get a ton of free Amazon credits for exactly this type of research.

lqdc13• 11 years ago

My university had no Amazon credits (2 years ago). I did have access to several supercomputers though, which would work out much better for this type of data.

Yahoo is also somewhat closer to Microsoft than to Amazon.

IshKebab• 11 years ago

Not the most interesting dataset though.

inglor• 11 years ago

People who downloaded this - does this contain any form of tagging of the data? For example, do news articles contain visit counts? Article sentiment? Any form of structured information?

Otherwise, what benefit does this have over scarping news sites?

GrantS• 11 years ago

The interesting data here aren't the news articles themselves, but the news-browsing history of 20 million people over a 4 month period.

To answer your first question, though, according to the official description of the dataset [1], "On the item side, we are releasing the title, summary, and key-phrases of the pertinent news article."

[1] http://webscope.sandbox.yahoo.com/catalog.php?datatype=r&did...

redrummr• 11 years ago

Did you read the article? It seems like they are providing data on the stories people clicked, and at what time, so you can draw temporal and recommendation hypotheses. Some device and location specifics are provided. Scraping can only tell the scraper's story. This data tells millions of people's stories.

boltzmannbrain2• 11 years ago

It's unfortunate Yahoo assumes only those with .edu email addresses make up "the research community".

rovr138• 11 years ago

No they don't.

http://webscope.sandbox.yahoo.com

boltzmannbrain2• 11 years ago

You're kidding... "TO BE ELIGIBLE TO RECEIVE WEBSCOPE DATA, UNLESS SPECIFIED IN A PARTICULAR DATASET, YOU MUST: - Be a faculty member, research employee or student from an accredited university - Send the data request from an accredited university .edu or domain name (for international universities) email address"

fsaintjacques• 11 years ago

I really need a yahoo account with verified sms to download this?

wahsd2• 11 years ago

Can someone please explain to me why this dataset needs to be one big file? They couldn't have broken it down? I need to download the full 1.5TB? Also, they couldn't have simply made the data available on one of the "big-data" services? Seems to redundant and inefficient.

wdr1• 11 years ago

Is it possible to get the readme w/o downloading the entire thing?

They state "The readme file for this dataset is located in part 1 of the download. Please refer to the readme file for a detailed overview of the dataset.", but I only see an option to get the full 1.5T.

jonesb6• 11 years ago

1) Begin registration to a community college.

2) Get .edu email address

3) Profit

satyajeet232• 11 years ago

Released Publicly?

You need an .edu mail address, a yahoo account with verified sms to download this!

Very unfortunate.

zo1• 11 years ago

It is unfortunate. But who knows what sort of restriction have to be imposed by the various sources of the data and other various contractual obligations? I'd imagine most of us would feel quite differently if we knew that we were sources for certain parts of the dataset.

Yahoo Releases the Largest-Ever Machine Learning Dataset for Researchers

Related Stories