AggData: Datasets created from scraping the web

by adamhowell • 17 years ago

62 26 17 years ago

utnick• 17 years ago

I have looked into making a business like this before, there are quite a few of them and I do like scraping.

But don't you have to break a lot of 'terms of use' agreements to scrape this data? Could you get in legal trouble for that?

ashishk• 17 years ago

I think that's something to worry about if sales grow to significant levels.

There are probably several solutions for that problem.

cschneid4• 17 years ago

Maybe, but there's a decent legal argument that it's not copyrightable data (facts of where stuff is for example), and that it's publicly posted ("find a store near you" links).

I suppose it gets more and more fuzzy if you move into non-fact data, and you open yourself up more to lawsuits.

lonestar• 17 years ago

By the same legal argument, couldn't anyone buy one of AggData's datasets and then publish it for free on the internet?

MrMatt• 17 years ago

I don't think that something being a fact makes it exempt from copyright. Map data is copyrighted, and often has minor inaccuracies in order to indicate copyright infringement.

evgen2• 17 years ago

The facts themselves are not subject to copyright, although if some business decides it does not like how its TOS is being violated it would probably stuff a few "fake" entries in there that would be returned to IPs that trigger a spider warning and then later go back and see if they were put into the db; if so you go after aggdata for violating a copyright on the collection as a whole. The data being provided by the businesses that are being scraped is not much different than that provided by a map service provider or phone book provider -- the individual facts are not subject to copyright, but the collection as a whole is and if aggdata is basically doing the web equivalent of photocopying a phone book and selling it to you then they are open to attacks from this vector.

toppy• 17 years ago

Hey, AggData guys, why not change business model and sell your data in bulks? Wouldn't be nice to use it that way?

from aggdata.dealership_locations import cadillac

print "Cadillac Dealers in NY:"

for loc in cadillac:

    if loc.city == 'New York':

        print loc.address, loc.phonenumber

timmaah2• 17 years ago

Their "FreeData" sets could use some attention.

I realize it is free, but if you are going to have it as an example of what you do, have it correct and up to date.

The headers for the congress data are completely off, and is not current.. Franken, Kennedy.. etc..

mitko• 17 years ago

Only "Locations" kinds of data? And the search is awful- it couldn't find anything for McDonalds for example (http://aggdata.com/search/node/McDonalds)

I was hoping to use it as a possible alternative to http://archive.ics.uci.edu/ml/ for ML data sets but now I am kind of disappointed.

jrwoodruff• 17 years ago

I just used the search to find 'mcdonald' and it came right up with the McDonalds data.

timmaah2• 17 years ago

He forget the '

searching for mcdonald's works fine as well.

weirdwes• 17 years ago

Before the App Store hit and iPhone web applications were all the rage, I started working on a restaurant locator. Oddity Software was a company I came across that provides datasets like the ones from AggData, though I'm not sure if it's from scraping the web. I figured I'd give it a mention in case people came here searching for additional resources. They definitely have more in the way of free lists (http://www.odditysoftware.com/free_lists.html), though I can't personally vouch for accuracy or timely updates as I haven't used them.

Listable (http://www.listable.org) is another list type service, though it's lists are much less complex and are user created.

I'll be adding AggData to my bookmarks, though. I could see myself using at least one of their "FreeData" lists in the future and possible some of their paid ones.

aggdata• 17 years ago

Wow, this discussion is way deeper than we have ever gotten into at AggData. In fact, "frig", I think we may need to hire you. :) We have been very particular in the type of data we collect for some of these very reasons, and we feel that the location data was enough in the public domain to protect us from infringement allegations. We don't currently have much in place to pursue those trying to resell our data, and it hasn't really been a problem yet. I think, like mentioned, it doesn't make much economical sense.

A couple of other quick responses: yes, we know our search is kind of lacking now, and we're working to fix it. Also, we have major plans of offering bulk data and specific regional data; we're currently just working on expanding our library, though.

Thank you, everyone, for your insight! -Chris Hathaway, AggData LLC

(and seriously, frig, send us a message on our contact page, I have more questions for you)

vijayr• 17 years ago

If you are interested in data, here are some sites to get them from

http://theinfo.org/get/data

http://infochimps.org

http://developer.amazonwebservices.com/connect/kbcategory.js...

http://ckan.net

EDIT: Comprehensive list here http://www.datawrangling.com/some-datasets-available-on-the-...

AggData: Datasets created from scraping the web

Related Stories