The Scraping Problem and Ethics

http://blog.osvdb.org/2014/05/07/the-scraping-problem-and-ethics/

by codezero • 12 years ago

241 123 12 years ago

blog.osvdb.org

elwell2• 12 years ago

Is it ethical to scrape from a scraper?

programminggeek• 12 years ago

Who watches the watchmen?

drdeadringer• 12 years ago

Is it ethical to scrape from a service you DO pay for, instead of using their API or other interface?

andreasvc3• 12 years ago

Ethical at least requires respecting robots.txt.

elwell2• 12 years ago

If robots.txt said to jump off a bridge, would your spider do it?

duey2• 12 years ago

The OSVDB website contains no signup page for commercial access. No pricing either, purely sign up via contacting someone. From my experience whenever I see this, I just refuse to use the service and look elsewhere. Contacting someone is annoying and opens you up to repeat sales calls. Perhaps they should make commercial API access easier to access rather than complain about scrapers.

awor• 12 years ago

Did you read the article? In both cases the offending parties were told to contact RBS to obtain a proper license/account.

kijin2• 12 years ago

That's exactly the problem that GP is talking about.

McAfee were just being the usual assholes, but the first guy mentioned in the blog post could have been converted into a paying customer if the pricing scheme were clearly outlined on the web. Since "Contact us" account tiers are usually reserved for the very high end, he probably assumed that it would cost him an arm and a leg.

It shouldn't be too difficult to come up with a handful of moderately priced account tiers, each targeting a different type of customer, as well as an open-ended tier at the end ("Contact us") for those with special needs and deep pockets.

lukejduncan2• 12 years ago

It's not really reasonable to say "I don't like the way you market your goods... so you really shouldn't be concerned with people stealing them."

duey2• 12 years ago

Not in the long term obviously, but you can't make a product extremely difficult to buy and then complain that everyone is taking the easy route.

mehwoot2• 12 years ago

Tell that to the 95% of the population on this site who torrent TV shows, movies and music.

johnhess• 12 years ago

I have to disagree. As creators, we should expect to see the behavior our designs encourage.

The fact is, if it's harder to buy something, would-be buyers choose another route.

Imagine something of trivial value that was very arduous to obtain. Say, the scores to last weekend's football game are only available via sending the NFL 2 cents taped onto a postcard then getting a user account and password back in the mail. Yes, you could apply for an account and who cares about the 2 cents? But most people wouldn't and who could blame them? There'd certainly be a market for pirated 'score data'.

But, if you just put ads on your site (the NFL does), voila! You make the same amount of money and people are happy to use your service. Netflix, Hulu, iTunes Store and Spotify all get this. If you make it a pain to do something, you can't feign surprise when everyone goes around you.

Is it legal to circumvent something just because it's arduous? No. Is it ethical? Not completely, but downloading pirated football scores wouldn't keep me from running for office.

lugg• 12 years ago

Pretty sad. McAfee I dont really expect much less from. They've always been a bunch of scam artists / con's in my mind.

I dontbhave time to look but couple of things striked me as odd. Is there a reason you dont lock down your request limits? Also why dont you secure access? Allow free accounts to be made and issue apis keys for them, at least that way you can much more easily rate limit access to the apis and heavily rate limit front end web requests down to reasonable numbers.

DigitalSea• 12 years ago

If I were in OSVDB's shoes, I would call these people out in an email and ask them to pay a licencing fee. McAfee have always had a shady past, even when they shook themselves clean of John, they have a history of scam-like behaviour to make a quick-buck.

You should be rate-limiting how many requests free API users can make, like; Twitter, Facebook and every other Internet provider does via their API. Make it harder for people to obtain the information (to the best of your abilities) and paying will become more of an option because they won't go to the trouble of scraping it if it's impossible and will take a lot of time to do so.

Think of your offering as a car. You currently have no car alarm or immobiliser, if you install a immobiliser and car alarm, you will make it very hard for a thief to steal your car.

userbinator2• 12 years ago

On the other hand, large (even non-profit) organisations are precisely the ones who have the resources to scrape steathily and widely, as would a loosely-organised community of users... it's not hard to come up with algorithms to respect the rate limits, balancing the load across multiple IPs and accounts, and producing access patterns that don't look any different from the rest of the site traffic.

rtpg2• 12 years ago

like much security, it's not about making it impossible, it's about making it a lot less convenient/a bit harder.

At one point the effort to circumvent would cost more in man-hours than just buying the product.

andrewfong• 12 years ago

Large organizations also tend to have risk-averse lawyers. I'm not a fan of the CFAA, but if what Aaron Swartz did was illegal, then so is this.

gadders• 12 years ago

If I were in their shows I would track their IPs and send them bogus data along the lines of "Please pay for a commercial license."

dspillett5• 12 years ago

You need to be careful sending bogus data: in some jurisdictions this could be argued to be deliberate targeted commercial sabotage. You would no doubt eventually win any resulting legal argument, assuming you could afford to carry the argument on to that conclusion. Sending no data, or limited data, would be safe though.

A better method would be to set "default" pricing (something high but not ridiculous, that could easily be negotiated downwards if they contact you) and make access beyond a few requests a click-through (or better: have them respond to an email before progressing further) where they agree to that pricing if they are using the information commercially.

newaccountfool2• 12 years ago

Even if you limit the rate, they will just create multiple accounts on multiple servers. There is no way you will make them pay for it as you could find it hard trying to prove it's them scraping in the first place.

jwcrux• 12 years ago

As a somewhat unrelated side note, I'm actually working on providing a free API written in Golang for CVE and CVSS entries. You can find it here: https://github.com/jordan-wright/cve-api

kevcampb• 12 years ago

I see that the "Open Source" Vulnerability Database changed it's name to the "Open Sourced" Vulnerability Database in July 2013.

https://web.archive.org/web/20130714002216/http://www.osvdb....

I guess they were hoping no one would notice the subtle but substantial change to their service

yaur3• 12 years ago

Shouldn't you at least be disallowing /show/* in robots.txt? Not that scrapers are necessarily going to respect this... but the way your set up it seems like this is semi-legit behavior.

userbinator2• 12 years ago

Maybe they want Google to be able to crawl their database (which it has clearly done, as you'll see if you do a search.) That also raises some questions...

yhager2• 12 years ago

Not completely by the specification, but I think this one works as expected.

    user-agent: *
    disallow: /

    user-agent: Googlebot
    allow: /

kylemaxwell• 12 years ago

Web scraping isn't a crime - the simple act of downloading the data should not be a problem here. (The reuse of the data might be, depending, but we don't have that information right now.)

This doesn't even rise to the level of what Weev did.

Afforess• 12 years ago

Scraping isn't a crime, but by the same token, neither is rate limiting and banning ips.

codezero3• 12 years ago

It can be a crime if it is a violation of the site's terms of service and the requests violate the host's robots.txt.

I'm not sure whether there is a legal precedent though, in some cases you could call it a denial of service if the requests are not rate limited, and in other cases it might be considered an inappropriate access (see Weev, though he eventually won appeal).

stronglikedan• 12 years ago

He's just making the point that it's unethical, which it is. Even if it boils down to simple bandwidth theft.

__david__• 12 years ago

I feel the opposite. I don't think it's unethical, even if it might be illegal.

Even calling this "bandwidth theft" is quite the hyperbole—if the server can't handle the bandwidth, then rate limit the requests.

I think if you're serving out pages to the public, you don't really get to tell me what kind of browser I'm allowed to download it with. As long as I'm speaking HTTP, it seems fair.

Sadly, that law has been slowly creeping against this mentality... Lately I feel like I'm some old internet hippy with these views. On a site called "Hacker News", no less.

thaumasiotes2• 12 years ago

In agreement with my sibling comment: if I ask you politely for some bandwidth, and you give it to me, it's hard to describe that as "bandwidth theft".

msie• 12 years ago

Why not put all information behind a login page and force people to sign up? It looks like this site will be used by only a few people anyways. You can then also track scrapers by login.

paulhauggis• 12 years ago

What about Aaron Swartz? He essentially did the same exact thing to the computers at MIT and he is somehow a freedom fighter when someone doing the same thing to the osvdb website is considered "unethical".

This is straight from the open security foundation website:

"We believe that security information and services should be easily accessible for all who have the need for such information and services"

codezero3• 12 years ago

While your sentiment is reasonable, I think the main difference here is that McAfee and others mentioned host their own private vuln databases and do not share them with anyone, so they were scraping to increase their own private resources for commercial use.

Aaron was scraping private resources to share publicly.

jamra• 12 years ago

More than that, he was scraping private resources that were freely populated. He was not robbing content creators of their money. He was circumventing a paywall to what should be free data.

pmorici• 12 years ago

You might argue the difference is that the information that Aaron was after was already paid for with public money. The OSVDB sounds like it is a bunch of people who aren't otherwise paid to maintain this information.

hueving2• 12 years ago

You might argue the difference is that the information that Aaron was after was already paid for with public money.

This is naive. Just because research is backed by public money doesn't mean the publications are automatically free to the public. If your argument was valid, you could use it to demand access to the emails of every FBI employee. Just because something is funded by the public doesn't automatically make every component of it open to the public.

cproctor• 12 years ago

I can't help recalling a post here a couple of years ago about the concept of "hellbanning" scammers on ecommerce sites--in short, making it look like everything is going fine, while actually isolating them completely from your business logic. Orders with stolen cards appear to go through, and send confirmation emails, but no real order is generated... In this case, you could transparently poison the results served to identified scrapers with 5% bogus vulnerabilities.

Or is that about as ethical as spiking trees to prevent illegal logging?

lukejduncan2• 12 years ago

I've definitely seen this done in practice

freshhawk• 12 years ago

Heh, I've always called this the map makers trick (found an article about it here: https://theweek.com/article/index/241967/trap-streets-the-cr...) although I guess that is specific to putting a small amount of fake data in your dataset to prove someone else used it. The hellbanning metaphor does fit for return large amounts of poison results.

It could be like spiking trees I guess, but that depends on the potential for harm. I just looked it up and was surprised to find that only one injury has ever been reported due to tree spiking. I guess it's a better talking point than actual tactic.

And like lukejduncan said, this is definitely done in practice.

calroc• 12 years ago

The point of spiking trees isn't to hurt people, in fact the tree-huggers tell the loggers that the trees have been spiked so that they won't attempt to cut the trees. The point of the tactic is to prevent damage to the forest, not to actually hurt anyone.

ChuckMcM3• 12 years ago

This is one of the more interesting policy questions on the web. Our search engine crawls a lot of blogs and what not on the web, criminals who want to find unpatched wordpress sites try to acrape our crawl by sending automated (scripted) queries to find them. We have developed a number of defenses over the years and pretty regularly ban them[1]. Here is the weird part though, if they hired 300 people on mechanical turk and paid them each a dollar to do one search, it probably would get them more information. Look at folks like 80legs or other 'distributed' scrapers. They exist almost solely to subvert these service terms. Are they evil? Creative?

One of the things that stands out is called out in this article. The people involved really want this information, so much that they are willing to expend time and effort to construct scraping bots and what have you . Why not just buy it? How is it that someone gets a request from their boss to get some information, but their boss expects them to get it for free? Can you imagine if they said, "We need pens, pencils, notebooks, staplers, the works for the office here. Oh and you can't spend any money getting that stuff, just get it here." Would they construct some elaborate raid on a nearby Office supplies store using a mercenary army of criminals? Why do that with information?

We did an experiment where we would 'grep the web' for you, basically run a regex over a multi-billion page crawl, give you the first 50 results for "free" and you could buy the complete set. I think we sold exactly one of those.

It is a weird thing, the OP captured it perfectly.

[1] Its a violation of the terms of service.

bizark1342• 12 years ago

Looks like 80legs that you mention rebranded themselves as "Datafiniti" at some point recently.

http://blog.datafiniti.net/?p=230

aruggirello• 12 years ago

which in Italian literally means 'end of Data'... kind of ironic isn't it?

meowface2• 12 years ago

I'm assuming that the vast majority of the time, it's a case of an individual or small group that wants the information, not a real corporation. Usually they have no intention of paying for anything or playing by the rules.

I have absolutely no explanation regarding McAfee though, considering the billions McAfee and its parent company, Intel, makes in revenue yearly.

stuaxo• 12 years ago

Could this be a company culture thing at McAfee ?

drewcrawford• 12 years ago

I think there is a great meta-question in here, about business models for digital data and software.

Here you have a great case study, about an organization that tried to do a volunteer model, and it didn't work. Then they pivoted to a commercial model, but fundamentally they still believe in a free tier. But they have to cripple that free tier pretty thoroughly, and even still people abuse it.

I have a product I'm working on, that some people are apparently willing to spend lots of money on. Ideally I would have some kind of low tier, so that people without lots of money would be able to use it too. But I can't figure out a way to segment the product so that everybody pays what they can afford without bad apples abusing the low tier and ruining it for everybody. The result is that I may end up only selling it to customers with deep pockets, even though the product is much more broadly applicable.

edraferi• 12 years ago

perfect price discrimination is hard.

MetaCosm• 12 years ago

They made the process of paying for the software a laborious pain in the ass. They are desperately trying to extract money from those who can pay, which sadly drives away those who can pay but don't want an involved process.

I have worked at lots of companies where I had a monthly budget of 10k+ that I could spend on whatever I wanted, but if I wanted any sort of complex deal (can't just put on CC with a line item) -- had to bring in legal and other groups -- instantly killed any interest.

"Licensing is based on the data needed (e.g. all of it vs subset), how it is used (e.g. internal only, external, product integration), etc."

What a goddamn horror show. I simply want a product, I want to pay for it, and I want to use it. Turning on Dropbox for Business was a decision made in about 5 minutes... "You all like it, already using it, awesome! I will get team setup." -- 5 minute later I had given Dropbox $3800.

I really think they are getting in their own way for no benefit. They have created a very high barrier to EVEN HAVING A DISCUSSION about buying the product. So, if I don't know exactly how will use it -- I can't purchase it. Stupidity.

ChuckMcM3• 12 years ago

Perhaps it won't come as a surprise but I've been toying with this question for a couple of decades now. Specifically what are the economics of information? In the 'goods' economy there are some interesting mechanisms that inform the question of value, these include but are not limited to, the cost to acquire materials, develop expertise in manufacturing, and managing the supply lines between raw material to finished good. Accountants will talk about the "Cost of goods sold" as a grouping function for these costs. In the 'information' economy the manufacturing part it pretty trivial, you just replicate copies, but the assembling part can be quite difficult. This leads to an interesting inversion where it can cost a lot to assemble something and nothing to 'manufacture' it. And that doesn't even begin to touch on what it is about information that makes it valuable in the first place.

What is the difference in value between a CD with the latest release of Ubuntu burned on it, and the download? download and a bootable Flash drive?

There is a great experiment you can run which goes like this; At one end of an athletic field, place a chess board with a queen on it on one of the squares. At the other end of the field have a table where people can get a quest. Offer to pay a person $5 if they will walk to the end of the field, note where the queen is, and come back and tell the quest giver. At the mid point of the field set up an information seller. They offer to sell you the location of the queen for anywhere between 20 and 80% of the reward price.

This simple experiment lets you see all sort of mechanisms in play that control information value. On the one hand you can see the range of values people apply to their own time (acquisition cost), their willingness to retain value (do they then go back mid-queue at the sign up table and start offering to sell the information for some fraction of the price to anyone?) At what threshold to people start trying to break the rules (a notion that is similar to price inelasticity but has a component like the 'black market demand').

Interesting questions to be sure.

wslh2• 12 years ago

Since you work at Blekko, there are more points to discuss that were not addressed in the article.

For example:

i) Are search engines web scrapers?

ii) Should search engines pay the scraped sites if they are charging to access their indexed data? probably some of the scraped sites has a specific license forbidding the search engine to sell their information in any way.

iii) Regarding Internet policies, is it fair/unfair that a site has a robots.txt configuration to avoid being indexed by a search engine other than Google? I would call this "search neutrality".

owlmanatt• 12 years ago

ii) Should search engines pay the scraped sites if they are charging to access their indexed data? probably some of the scraped sites has a specific license forbidding the search engine to sell their information in any way.

Any reputable search engine will respect robots.txt.

ChuckMcM3• 12 years ago

i) no, search engines articulate the web. They are essentially 'pre-paid' computation.

Take the classic example, a search for 'bilbo baggins'. What that is, is a request to identify documents on the web that have referred to Bilbo Baggins and return their locations.

1) It is absolutely true that you could sit down at your computer and look at each site, from aol to zillow, read all their pages, and note the ones that mention Bilbo. Then you could go back and order the list by the ones that had more of the information you were looking for to the ones with less useful information. Along the way you would find some sites that would not open up to you unless you had an account, those you could not visit.

2) A search engine can look at all the sites, it can note which sites mention Bilbo along with a bunch of other terms and can essentially "pre-compute" that list you were looking for. Along the way it will find sites that, through their robots.txt file, will say "We'd rather you not look here." and it will respect that, thus not indexing those sites.

In both cases figuring out which web pages have information about Bilbo on them is creating 'new' information out of existing data. You can do it on your own and it will cost you time, or you can do it with a few thousand machines and it will cost you money. Either way you get a list of possible sites.

That list forms a distribution, where there are a lot of sites that don't care one way or the other if you read them, sites that won't show up because they asked to be excluded, and sites that will show up because they paid to be included. Some sites really want you to find them, some sites really don't. A good search engine caters to both types.

ii) If a search engine was taking the page, copying it, and then showing that instead of showing the page (this is what got Google's news product in trouble) then its pretty clear that they should not do that. But in terms of location information? The sites themselves derive a huge economic benefit from being in the index that isn't reflected at all back to the search engine that sent traffic there [1], so on a pure economic basis the search engine is on the losing side of that transaction. However, the marginal cost of additional transactions is small (search engines are general purpose) so they make a small amount on large volumes.

To put your question in more specifics, where is the economic value in the list; bobs middle earth atlas, wikipedia entry on bilbo baggins, middle earth web ring, imdb pages on characters in the "Lord of the Rings" movie.

Is it that Bob has an atlas of Middle Earth? Or is it the list itself? Who made the list? Bob or the search engine? (or some human curator of a bookmarks page[2])

iii) It is completely up to the site to allow or disallow access to its content by search engines. Some sites do only allow themselves to be indexed by Google and they find they get less search engine directed traffic that way. Some sites don't allow anyone to index them and they get no traffic (sometimes they are surprised by this, sometimes they don't care, sometimes they are angry that the only way for people to find them is to be in a search engine index)

[1] Google broke that by creating AdSense for Content and created a pretty interesting conflict of interest for themselves.

[2] Good luck finding a book marks page these days :-)

lazyant• 12 years ago

Look at folks like 80legs or other 'distributed' scrapers. They exist almost solely to subvert these service terms.

Not sure I understand, is there a whole ecosystem of web businesses that feed off for free of your search engine, or you meant from several legit search engines or ... ?

yp_all2• 12 years ago

Pot calling the kettle black.

Crawling others' websites then selling ads. <--- Blekko

Scraping others' websites then selling ads. <--- ????

Selling access to user-generated content. <--- OSVDB

Amusing to watch these folks argue about ethics.

Who owns the copyrights in this data? Surely not the one who is demanding that you pay a license fee. These "services" are middlemen, plain and simple.

This might be why McAfee was wondering about how much manual curation is done.

Maybe that is the only possible theory of how OSVDB could assert any rights in the data (and only in a select few jurisdictions).

scott_karana• 12 years ago

If they're a middleman, and it's such an easy "service", why didn't McAfee just bypass them and get the data from the original sources, rather than do the wrong thing?

notastartup4• 12 years ago

This is exactly the argument MPAA makes against people torrenting movies and tv shows, using words such as "criminals", "evil", in an attempt to polarize people who are simply consuming information over the internet in the most efficient way possible, without any 3rd party oversight and censorship, the way information on the internet is always bound to be, the way equilibrium is achieved.

The argument that if you really want this information, they should pay for it doesn't work. Basically, you are punishing people for automating manual labor. The fact that you can hire bunch of people and tell them to manually copy data from the website and achieve the same result means that it shouldn't be any different from automating the process itself.

Terms of service on a website is not the law. If there is something you don't want people to have access to, then don't publish it online at all.

buro9• 12 years ago

We did an experiment where we would 'grep the web' for you, basically run a regex over a multi-billion page crawl, give you the first 50 results for "free" and you could buy the complete set. I think we sold exactly one of those.

Sounds like you didn't advertise it right. I've been looking for a "grep the web" service for a while.

We're going to use https://builtwith.com/ to achieve that exact thing. It will cost me $295 : https://builtwith.com/plans

For that $295 I will get a list of all domains using a rival technology... a list of sales leads.

I'm still short of what I want to have... a prioritised list of sales leads.

So I will write my own scraper to go through those results and scrape every one of those so that I can pull some info from the HTML page to tell me how large those customers might be. I'm not aiming for the largest (easy to find, costly to win), nor the smallest (time consuming and pointless to win), but the median.

I would definitely pay for a "grep the web" that allowed me to match pages by text signatures in the HTML, and then extract part of the DOM as values, and return the list of "url + extracted values" for the matching hits.

I'd consider that to be worth similar amounts to what BuiltWith are chaging, but I'd add more and would go up to $500 assuming that the results come with the extracted values as a CSV file of some kinda and the quality and completeness of the report is high.

People will pay for "grep the web", especially if you sell it to them as something they know they really want: "sales leads".

diziet• 12 years ago

What is the price? I assume the price is in the 5-10k+/month range, which is a reasonable amount for data like this.

alexnking• 12 years ago

I'm more and more concerned that the legal and cultural environment for web scraping would make it hard for a company like Google or Yahoo to be founded today.

The internet isn't about "don't take my stuff", it's about spreading that stuff around. I'm confused by people who want to make their data public, but want to control exactly how people access it.

ksk3• 12 years ago

make it hard for a company like Google or Yahoo to be founded today.

Is that a bad thing?

The internet isn't about "don't take my stuff", it's about spreading that stuff around.

Try asking Google if they want to "share" their database of crawled data.

I'm confused by people who want to make their data public, but want to control exactly how people access it.

Me too.

newaccountfool2• 12 years ago

Is that a bad thing? Yes, no question about it. More players in the market means more competition and more competition means a better service.

TheLoneWolfling• 12 years ago

Agreed.

I talked about this a while back: https://news.ycombinator.com/item?id=6572937

Someone decided to ban all bots from accessing their site except for Google and Bing. So much for "if you're so worried about Google just use another search engine".

hnal943• 12 years ago

I'm confused by people who want to make their data public, but want to control exactly how people access it.

You mean like TV? or Radio? or print....

johnward• 12 years ago

To be fair those mediums all kind of suck in their own way.

binarysolo• 12 years ago

Given these people's immediate reaction to resort to scraping, I'm honestly kinda surprised they didn't setup basic practices of at least going through proxies (or free ride off of Tor, ugh).

im3w1l3• 12 years ago

including an expansive watch list capability,

I would avoid the word expansive. Too similar to expensive. Especially when emailing people who are not native English speakers.

im3w1l3• 12 years ago

I was once developing a program for automated trading for my own personal use. In the beginning I made periodical requests to my broker, scraped out the price info, and made orders if the price was right. Once I had something sort of working I decided that I wanted to do this the right way and ask them for permission. They told me I would have to subscribe to a feed through some other program. After many hours of reading the manual and trying to figure out how to transfer the data to my application I again got it sort of working. But I had already started to lose interest and other things had come up in my life.

How is this relevant? Well I would have much preferred paying for scraping, than trying to learn some new api. It increased the transaction cost. If you further have to negotiate with a partner company, that sounds like even more transaction cost, from having to send emails back and forth and the mental effort of negotiating.

wslh2• 12 years ago

Beyond the moral discussion, I think a market for web scraping is a good thing because currently there are a lot of unconnected people trying to buy/sell this service.

Freelancer sites has a lot of offerings for web scraping but this niche has its own issues.

im3w1l3• 12 years ago

paying [the site in question] for scraping I mean

bitJericho• 12 years ago

I would hope they sent a bill.

NoMoreNicksLeft• 12 years ago

Maybe it's the supervillain in me, but I think I would try to come up with software to recognize the scraping attempts, and rather than ban them just have it generate fake data on the fly.

detroitcoder• 12 years ago

Amateurs. If you are going to crawl distribute it across unique IP addresses.

e28eta• 12 years ago

We had a former Google Maps PM give a talk about building APIs, and he addressed scraping. Something he pointed out is that the traffic pattern from scrapers differs from regular users. Regular users make requests for the same things often. Scrapers usually only request any given item once. The really obvious ones do it in order from the beginning. He had a hilarious map showing the requests of someone requesting business information starting from the top left corner of the map. They got really good information on all of the businesses between the North Pole and Greenland before being blocked.

notastartup4• 12 years ago

Scraping is not an ethics discussion. It's what you do with the data that falls into the topic. Selling email lists scraped from websites to spammers would be one. Using email lists to prevent spam would not be one.

The article is basically saying "I want to charge people for information I have made public online, they won't pay, so they are obviously thieves by refusing to do it manually by hand like they are supposed to."

Gimme a fucking break here. If you don't want the information to disseminate, DO NOT PUBLISH IT ONLINE.

pbreit• 12 years ago

It's hard to get excited about this. The data owner supposedly has a mission to make the data available but then is concerned with such a thing. It doesn't strike me as that difficult or expensive to make that sized database available.

jmzbond• 12 years ago

This also goes way beyond scraping, in the sense of small start-ups doing all kinds of hacks or clever manips to stay within the bounds of a free trial service.

As someone who sees both sides… I don't know what to say. I'm running test landing pages on Heroku with New Relic that pings the sites every minute to ensure the dynes keep spinning and my users don't experience downtime. While I'm careful to stay within fair use, this is at best obnoxious, because if everyone did this, Heroku would certainly need to redefine what's free. From my POV though, I am a bootstrapped entrepreneur and supporting 5 landing pages. I simply don't have resources to pay for a dyno and test everything I have in my head, especially not combined with the many other resources I'd need to start paying for as well.

Or consider the kid in Florida who used Parse' free account for hundreds of thousands of users. [1] (The article was on HN a few weeks ago, this was not it's central point, just something I took away relevant to this comment.)

Part of the cause I think is that we live in a world where we're so used to having things be free, it becomes an entitlement. Another is that all these examples of start-up hacks and hustle stories, we kind of laud, don't we? Everyone talks about how Airbnb scraped Craigslist and got a huge boon that way, but few in critical tones. Should we? Or is that how competition and new products get created (i.e., if the scraping hadn't happened, perhaps Airbnb and the whole sharing economy would be less successful today).

These are philosophical questions, and I don't really have a solution, but they are things to think about.

[1] http://pando.com/2014/04/30/how-a-florida-kids-stupid-app-sa...

Gigablah• 12 years ago

Looks like they forgot to conceal the full name and email of the guy from s21sec in one of the quoted emails...

The Scraping Problem and Ethics

Related Stories