With the rise of AI, web crawlers are suddenly controversial

https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders

by leephillips • 2 years ago

90 84 2 years ago

throwup238• 2 years ago

For decades, robots.txt governed the behavior of web crawlers. But as unscrupulous AI companies seek out more and more data, the basic social contract of the web is falling apart.

The basic social contract of the web fell apart long ago when almost everyone decided that Google was the only search engine worth serving and started aggressively blocking other crawlers.

zarzavat• 2 years ago

Add to that the removal of public APIs and things like RSS feeds. I would much rather use an API than scrape you, I will even pay a small fee, but if you don’t provide anything, then you’re getting scraped.

1vuio0pswjnm72• 2 years ago

"Add to that the removal of public APIs and things like RSS feeds."

You mean like this public API

https://hacker-news.firebaseio.com/v0/item/39421253.json

Or this RSS feed

https://www.theverge.com/rss/index.xml

Plasmoid• 2 years ago

I'm out of the loop on this. What has been happening? Do you get IP blocked? Rate limited?

realusername• 2 years ago

You do and since the rise of Cloudflare, you need their approval to create your search engine

ec109685• 2 years ago

Are there services that have a copy of the Internet for a fee?

How did perplexity.ai crawl the internet for their AI?

aaronrobinson• 2 years ago

Drama. Crawlers have always been controversial.

elpocko• 2 years ago

robots.txt is relevant and effective, as is my DNT header.

naiv• 2 years ago

Proxy companies are a big winner now

linkjuice4all• 2 years ago

"With the rise of AI, photos of the exterior of your business are suddenly controversial"

Many revenue-based websites tried to have it both ways with web crawlers wherein they wanted to block automated access or repeat viewers while letting first time viewers get a free taste. Others have noted that basically Google gets a free pass for all the traffic it brings in but everyone else has to respect robots declarations.

It seems like a no brainer - if your web server is configured to reply to GET requests with a 200 status and some content then they get to do pretty much whatever they want with it.

Don't want to give access to everyone? Stop sending your content for free and get them to agree to some contract and authorize/license their access to your stuff.

spencerflem10• 2 years ago

I don't think this is that cut and dry. Its perfectly reasonable to be fine with your content being linked to in an index like google search does, but not fine with the content being read by humans for free or used to train AI.

I'm sure the sites would rather that google paid them in licencing too, but without changes to our laws that's not going to happen

matthewrobertso6• 2 years ago

I want to research a topic, so I do a google search about it. If 100% of the search results are readable to google, so that they are indexed, but unreadable to me, due to paywalls, the google search is useless.

I'm not sure what the solution is, but paywalled articles in search results are bad. If they want to be indexed they should have to offer that same indexed content to anyone browsing the index.

postalrat• 2 years ago

It is also reasonable to serve the same content to regular visitors as you do to Google?

Is ok to serve a full article to Google and put visitors behind wall?

hobs3• 2 years ago

I deeply 100% disagree with you - if you want to show google something else that is a spam tactic that only ends in bad outcomes. If your content is good enough to pay for its good enough to hide from everyone.

A search based on the results being something otherwise than the contents of the search is misleading, a spam tactic, and bad.

CharlesW4• 2 years ago

Stop sending your content for free and get them to agree to some contract and authorize/license their access to your stuff.

Agreed, we need to wrap the public web with DRM immediately. We can't expect companies like OpenAI to waste time worrying about pesky legalities like copyright and content licensing.

spencerflem10• 2 years ago

I don't think there's a technical solution that would work here. This is a law & enforcement problem

BeFlatXIII• 2 years ago

Perhaps this can accelerate DRM hacking if multiple billions of startup money are put behind breaking it to stealthily add to the data sets. All the more fun if it's done somewhere extradition-proof from the US.

flir4• 2 years ago

You're not even sure this is a copyright issue in the US yet, let alone every other country on the planet. So yes, if you don't want it freely consumed, don't freely publish it.

o11c• 2 years ago

There's a different between "I allow people to access my data" and "I allow people to create products based on my data"

amelius• 2 years ago

When did robots.txt get a legal status?

Or did it ever?

franze2• 2 years ago

eBay v. Bidder's Edge (2000)

nadermx2• 2 years ago

That was overturned, https://en.wikipedia.org/wiki/Intel_Corp._v._Hamidi and White Buffalo Ventures LLC v. University of Texas at Austin

calibas• 2 years ago

For decades, robots.txt governed the behavior of web crawlers.

It never governed anything, web crawlers were never under any obligation to follow robots.txt.

This article seems like they took an existing controversy, rebranded it as something new, then blamed in on AI.

fallingknife• 2 years ago

they took an existing controversy, rebranded it as something new, then blamed in on <new thing>

Basically describes most of what passes as journalism these days.

franze2• 2 years ago

web crawlers were never under any obligation to follow robots.txt.

other than the fact that you could get successful sued if you do not follow them i.e.: eBay v. Bidder's Edge in 2000

nadermx2• 2 years ago

That was overturned, https://en.wikipedia.org/wiki/Intel_Corp._v._Hamidi and White Buffalo Ventures LLC v. University of Texas at Austin

geor9e• 2 years ago

The Internet Archive never follows robots.txt and they're still around

janalsncm2• 2 years ago

No, I think this is something new. It’s an existential problem for newspapers. Most newspapers have never thrived online but they got a stream of clicks and a few 100x stories every once in a while. But with language models and RAG, we may create a world where no one has to visit their site at all.

This really hit home for me in Hard Fork’s interview with the CEO of Perplexity AI [1]. The crux of the problem is, if Perplexity does what it strives to do, provide answers rather than links, no one will be clicking on links and generating revenue for newspapers anymore. I encourage you to watch him struggle with this question because this is really an irreconcilable problem.

[1] https://m.youtube.com/watch?v=AJuv_UwxELA

andybak• 2 years ago

But as unscrupulous AI companies seek out more and more data

I'm not sure I'm ready to concede the fundemental value judgement being made here. At least I refuse to accept it as a given rather then the core issue to be decided.

micromacrofoot• 2 years ago

many crawlers have always ignored robots.txt, if you’re monitoring any moderately visited site you’re bound to see random spikes of bots hammering your server no matter what text file or headers you set

datavirtue• 2 years ago

This. I stopped reading after the first few sentences. Whoever wrote that verge article is clueless.

Data accessible. Data free. Period.

phone86753092• 2 years ago

Maybe it was the same person that put together the Verge PC build guide (not the person presenting it, the person who wrote the script for it)

lewhoo• 2 years ago

I don't get it. The crux of it all seems to be that Google isn't competing with owners of data it crawls using the very same data. The crawl part isn't as much of a controversy as usage, isn't it ? The mentioned eBay v. Bidder's Edge (2000) seems to be a dispute over usage.

pixl97• 2 years ago

I mean Google is making AI models and Google is summarizing sites so they may well be competing. The difference is Google has a near monopoly on search and ads.

mediumsmart• 2 years ago

The web comes in 2 versions. One of them has a basic social contract. maybe

With the rise of AI, web crawlers are suddenly controversial

Related Stories