Foursquare dataset free to download and analyze

http://www-users.cs.umn.edu/~sarwat/foursquaredata/

by rsobers • 13 years ago

113 35 13 years ago

www-users.cs.umn.edu

That doesn't look like Foursquare has handed that over. What's the legality of scraping a service for their data in this way?

onedev• 13 years ago

That doesn't look completely legal to me either.

joshu• 13 years ago

then re-releasing it with distribution terms is probably not legal at all.

ozh• 13 years ago

I don't get why scraping publicly and freely accessible data would be illegal. The redistribution under their own terms is another matter, though.

jorgeortiz85• 13 years ago

I'm a Foursquare engineer. We have explanations of our API policies here: https://developer.foursquare.com/overview/community

We'll be contacting this researcher to ask where they got this data and whether it conforms to our policies.

nicholassmith2• 13 years ago

Thanks for responding, I didn't realise Foursquare actually gave so much of their data away freely. Which makes it slightly more seedy someone has scraped the rest and dumped it online.

boothead• 13 years ago

No mention of the data format. Is it json, csv what? I know you can always head -n the file but a little hint would be helpful!

nachi• 13 years ago

It looks like an ASCII-formatted table. Pretty disappointing that it isn't machine readable out of the box.

timthorn2• 13 years ago

What's not machine readable about an ASCII table? A fixed width table has its own advantages over eg CSV - for instance, to read a specific field you can reach it by offset rather than having to count delimiters.

aw3c2• 13 years ago

sed 's/[[:blank:]]//g' dataset.dat | sed 's/|/\t/'

sneak2• 13 years ago

Obligatory:

https://en.wikipedia.org/wiki/AOL_search_data_leak

http://techcrunch.com/2006/08/06/aol-proudly-releases-massiv...

http://www.nytimes.com/2006/08/09/technology/09aol.html?page...

TL;DR: It's fairly easy to deanonymize datasets like this, provided they are somewhat complete.

route66• 13 years ago

The dataset comes with GPS locations of users and venues. With that data alone you can retrieve individual addresses in not too densely populated areas. Missing links will get caught in the social net.

We already learned that Warhol's 15 minutes of fame should read 15 megabytes , but, to cut the "it's the users choice to post that data" apologists short: almost no-one I speak to understands the implications of all possible interpretations, classifications and groupings that their online traces allow.

sneak2• 13 years ago

Furthermore, it's not the users' fault that 4sq isn't sufficiently rate limiting or otherwise protecting this data. Why should arbitrary users be able to see the social graph of others they're not friends with? Also, why should people outside of your immediate one-hop social graph be able to see your checkins?

Giving it to 4sq for data mining is different than giving it to UMN and/or the whole internet for data mining and/or deanonymization.

r721• 13 years ago

Also this:

http://33bits.org/2009/05/13/your-morning-commute-is-unique-...

davidmat• 13 years ago

Could anyone recommend some solid introductory material on data analysis/data visualisation?

I'm thinking this data set seems like a fun way to fill a rainy weekend, going for a dive into these worlds :)

benmanns• 13 years ago

The Social Network Analysis course on Coursera started yesterday. [0]

0: https://www.coursera.org/course/sna

m4tthumphrey• 13 years ago

Look's like it's been removed. Damn.

Edit: Not removed, just unaccessible. 403.

annnnd• 13 years ago

I'm quite sure it will surface somewhere. Looking forward to it. :)

EDIT: I would love to get my hands on this data... anyone? :)

mvanvoorden2• 13 years ago

I have it, but "The user may not redistribute the data without separate permission."

But if you have any questions about it, I can try to answer them, my username here also corresponds to a gmail account I use ;)

danso• 13 years ago

Jesus Christ. The bulk scraping in violation of the TOS is egregious enough, but redistributing it with a mandate that the researchers get credit? For what, scraping a generous public API?

Irishsteve• 13 years ago

This is usually common practice. It serves two purposes 1) Makes it easier to find the dataset that was used for experiments. And 2) Improves citation count for the author which is usually important in research.

mzs• 13 years ago

That's true, but here is the paper from Microsoft Research and it seems to be lacking in how those data files were generated:

http://research.microsoft.com/pubs/156453/icde12_lars.pdf

rajbala• 13 years ago

The data set has been removed?

lgas• 13 years ago

http://archive.org/details/201309_foursquare_dataset_umn

xntrk• 13 years ago

sounded too good to be true. I guess we'll have to find it on bittorrent.

dotBen• 13 years ago

filename was "umn_foursquare_datasets.zip" in case that helps

galapago2• 13 years ago

http://ul.to/betfn1vh

galapago2• 13 years ago

http://webcache.googleusercontent.com/search?q=cache:hLI5FqD...

(the direct link is not working, but this confirmed that was freely available)

interskh• 13 years ago

This data set contains 2153471 users, 1143092 venues, 1021970 check-ins, 27098490 social connections, and 2809581 ratings that users assigned to venues

The number of check-ins seems to be low compared to other numbers.

renownedmedia• 13 years ago

Looks like the data is only up-to-date as of July 2012 (judging from the zip compression times).

waynesonfire• 13 years ago

why was this not posted as a torrent?

Foursquare dataset free to download and analyze

Related Stories