The AIpocalypse is here for web sites as search referrals plunge

https://www.theregister.com/2025/06/22/ai_search_starves_publishers/

by rntn • 1 year ago

75 80 1 year ago

I wrote about this last year that SSO will give way to LLMO. https://matrix.ai/learn/blog/content-commoditization-and-tru...

ujkhsjkdhf234• 1 year ago

I think you mean SEO not SSO. I know there is company that already is working in this space. I can't remember the name though.

Nicholas_C2• 1 year ago

How would LLMO and SEO strategies differ?

adaptbrian• 1 year ago

People need to corroborate facts you can't do that with just a box, you need someone else's UI to shape the trust you are talking about. That can't be done by showing a simple favicon next to a blob of text.

ed_mercer• 1 year ago

Who knew??

pingou• 1 year ago

"Google's average ratio of pages crawled to visitors referred was 2:1. Six months ago, that ratio had increased to 6:1. Today, according to the Prince, it's 18:1", that's interesting but are we sure it has something to do with AI?

yorwba• 1 year ago

Compare to "According to Prince, OpenAI's ratio of pages crawled to visitors referred has gone from 250:1 to 1,500:1 while Anthropic's ratio has gone from 6,000:1 to 60,000:1."

wiredfool• 1 year ago

I’d be stunned if my sites have as much as 1% non-bot traffic, even outside of the ones that are vaguely straddling the border between bot and Ddos.

cko• 1 year ago

In the past two years most of my Google searches were: "______ vs ______", <product name> review, and <subject> reddit.

Sometime in the last two months I noticed myself going straight to ChatGPT, and Gemini flash when my free credits ran out. And ocasionally some Reddit threads.

chrismorgan3• 1 year ago

And then the question is: why? Is it because the LLM is better, or because the search engine is worse?

I just purchased a monitor, from among the cheaper 4K 27" ones I could find. I’m in India, so they may not be the same models you’d get in places like the USA: maybe the situation is better there. When trying to compare them, or even find reviews of them, I could not find anything useful through a search engine. Exactly zero useful things. Not a single review of my four candidates. Oh, a couple of video “reviews” probably in Hindi… but I have learned to utterly scorn that category as consistently devoid of content; and a couple of “review” sites… with only copies of the first-party marketing material.

This part of the web seems to be just dead.

As for LLMs, well, I’m sceptical that they actually are all that much better: where is the information coming from? I’m deeply suspicious that they just look better. It’s well-established that they’re magnificent at conning.

For my own experience, I’ve tried DuckDuckGo’s Assist a few times, which is GPT-4o, and I haven’t been impressed with it. Almost everything it comes up with is superficial, or wrong.

nottorp• 1 year ago

Product reviews have been dead before "AI".

Even if you search for english reviews.

I don't think "AI" will help at all with this since it's trained on the same polluted space.

Ferret74462• 1 year ago

Because the search engine is worse, obviously. A search engine can only find things that exist, so unless someone wrote an article that was exactly what you need, you'll have to find multiple sources and compile the info yourself. LLMs do all of that work and generate the ideal report you were looking for (minus hallucinations).

jasonsb2• 1 year ago

Search results are only useful for product searches these days. Everything else is low-quality, bot-generated content designed to maximize ad clicks or affiliate earnings. Web search was a waste of time long before the AIpocalypse. These so called SEO experts who ruined the web for the last 10 years are now complaining that AI is ruining their business. Good riddance.

whatevsmate• 1 year ago

Exactly my sentiment. The web being crummy is not because of AI. The mainstream web - Google search and ad-infested web sites - was already crummy. AI has a higher tolerance for the sea of detritus than I do, so I lean on it to avoid that garbage.

transcriptase2• 1 year ago

Yep. Nowadays even if you try to do your research on something like a foam mattress/bed in a box… be aware that the same company owns virtually all the ranking and review sites as well as all of the dozen brands they’re pretending to compare. Even the old method of appending Reddit to search queries hasn’t returned genuine results for the last few years since companies know to use aged and otherwise organic looking accounts to subtly name drop themselves and reply in agreement throughout threads alongside keywords that will bring in google traffic and upweight their odds of LLM mention.

jfengel2• 1 year ago

Mattress stores have been famously dubious since before the Web. I'm still not exactly sure why they're always going out of business but never actually go away. One near me has been having a bankruptcy sale for three decades.

VladVladikoff3• 1 year ago

As a side note on this, I’m a huge fan of pure latex mattresses. Just a solid slab of latex. Maybe a few slabs of different densities if you want to get fancy. I’m 4 years in on mine and it still holds up really well, no sag issues etc. if you’re in Canada there is MFC (memory comfort Canada) this is the cheapest place I could find, but there might be a better deal if you dig deep enough.

chrismorgan3• 1 year ago

Also the reviews will be rigidly milquetoast, deferential nonsense, giving every product a participation award in a tailored category of its own, and refusing to ever call one superior to another, because the word “better” is so judgemental and might hurt someone’s feelings.

croes2• 1 year ago

But it won’t help the real useful sites either.

So we just get more losers and no winners

jasonsb2• 1 year ago

The fact that we have a lot of AI alternatives instead of a Google Search monopoly is a win in my book. Probably not the win you'd like to see, but a win nonetheless.

datavirtue2• 1 year ago

Exactly. Every time I follow a link to a link to a source I'm rewarded with a huge wall of text that finds ways to keep repeating the same phrases over and over to MAYBE reveal a little nugget that you need. Talk about having your dignity robbed.

JeanMarcS• 1 year ago

I would add the search of an error message while developing. Mostly using the search engines for that.

But hey ! Now you can ask ChatGPT, Copilot or whatever for the same thing !

We are surounded

spacemadness2• 1 year ago

The only good outcome of LLM content generation for society so far it seems. I remember some recipe site folks were here on HN complaining about ad blockers and it was hilarious following their reasoning. It’s ok to manipulate the entire internet if you’re a business but you’re not allowed any control individually as a consumer I guess. I’m sorry, but filling the internet with 90% fluff garbage doesn’t entitle you to any sympathy.

OptionOfT• 1 year ago

This is a great observation. Google established rules that would increase your website's pagerank. In response to that SEO was created, and with that the experts. Tags, inbound links, and then more content.

A good example of how this spawned articles that are absolutely useless: https://www.partitionwizard.com/news/

All those articles have nothing to do with application they're trying to sell, but they do (did) bump the pagerank.

Many companies do this, which has polluted search results for issues to the most common denominator, making it impossible to find that 1 post that actually has a working solution.

And now we've reached the next stage, those pages are no longer written by humans, but entire websites are being generated for the purpose of selling you stuff.

I was searching for the name of the bitter part in a peanut, and stumbled upon this website:

    https://www etprotein com/parts-of-a-peanut-anatomy-of-a-nutty-snack/

(URL is missing a dot after the www and before the com)

We have entered a situation where there is no longer 'no answer', there is always some answer out there, simply to sell you stuff.

In the past if you searched for Brand X at Retailer Y, and they didn't carry it, you'd get no results. Now they autocomplete Brand X's name, and show you a whole bunch of products related to Brand X, with no indication that they don't carry it.

mvdtnz• 1 year ago

It also impacts normal people putting human content on the web. It's not just SEO "experts" whose businesses or projects are being buried.

1vuio0pswjnm76• 1 year ago

"Web search was a waste of time long before the Apocalypse."

Difficult to find anyoone who will acknowledge this given the amount of mooney involved.

One search engine website pitched itself as having a mission to "organize the world's information". However search != organisation. The information the company was searching did not belong to them. It was outside their control. It generally remains disorganised.

The mission was actually to copy the world's information, intermediate access to it and sell advertising services.

The company did not sell "organising services" to information publishers. It sold online advertising services to advertisers.

Intermediating access to information was nothing more than a tactic to asemble audiences for advertising. The mission was to sell online advertising services.

Organising information for web publication is still an unsolved problem. (The problem of vast amount of disorganised information on the web remains unsolved.) Information that is organised is more easily searched. If organised well-enough, it can be generally be browsed without searching.

Utrustworthy, unethical coomputer nerds found ways to make money from the disorganisation. They became wealthy and were celebrated as being "successful". ^1 As such, it is difficult to find anyone who will acknowledge that relying on "search", where the algorithm is properietary, secret and can be still gamed by "SEO", etc., was deeply flawed. Flawed not in the sense of it offered no opportunity for making money. Flawed in the sense it ultimately does not work well for locating the information sought by the searcher.

If the wealth from intermediating access to information via "search" begins to dry up, then perhaps the fundamental flaws of web search will (again^2) be acknowledged.

1. NB. No one has been successful in organising the information published on the web.

2. As they were in the 1990s when one search project claimed (falsely) it wanted to counter the effects of advertising on "search" and remain in "the academic realm" only to then become the dominant web advertising services company.

It makes sense that product searches might be effective because product infomation tends to be organised.

1vuio0pswjnm76• 1 year ago

https://deskthority.net/wiki/Chatter

spwa4• 1 year ago

No worries! Of course governments, well especially EU, Australia, China, ... would NEVER let foreign companies take over their entire online marketplace (and a host of other things like ERP). So of course they have working well-funded local copies of Google, Amazon, Facebook, Tiktok, Salesforce, ... developed locally ready to go to secure their economy! Plus, of course, the internet infrastructure underpinning them.

Otherwise those countries would be SO fucked! So you can rest assured that their governments have worked tirelessly to prevent that from happening.

I think I'm going to dream of riding flying pink chocolate unicorns tonight.

smokel• 1 year ago

Most countries have lacked the capability to manufacture basic infrastructure for ages. It may look scary to you now, but it has been looking scary for decades to some.

Face it, we depend on a global economy. Every now and then something changes, and we adapt.

datavirtue2• 1 year ago

The only reason the US can build infrastructure is because we can print our own money.

heresie-dabord• 1 year ago

I don't in the least like the energy being consumed by AI business, but it appears to be the cost of undoing the enshittification of search.

dboreham2• 1 year ago

Temporary. AI will be enshitified just the same one day.

whynotminot2• 1 year ago

I was thinking the other day that I liked the web better when it was just people who were passionate or nerdy running a website. It wasn’t about ad revenue.

Maybe it’ll be better this way.

Nicholas_C2• 1 year ago

Are those passionate/nerdy people still running niche websites? A search engine that prioritizes that kind of content would be great.

dboreham2• 1 year ago

And it would make no money therefore it wouldn't exist.

viraptor2• 1 year ago

Kagi is fine doing that. See also https://kagi.com/smallweb

c54• 1 year ago

I haven’t validated it myself but Kagi is trying to offer this kind of small web search https://blog.kagi.com/small-web

user432678• 1 year ago

There’s a search engine for exact such websites — https://wiby.me

samrus2• 1 year ago

its not sustainable. any search engine that gets popular enough will get SEOd by bad actors eventually. we can enjoy it while its good though

ripe• 1 year ago

I've been using Marginalia to avoid commercial sites. Open-source and unencumbered by ads:

https://marginalia-search.com/

Pxtl• 1 year ago

No they're running YouTube channels and Substacks because of built-in monetization.

owebmaster• 1 year ago

No, they got rich adding ads, sold their websites and are now influencers

whytaka• 1 year ago

I've been building https://www.webring.gg . It's a webring creation and management tool that lets members vote new websites (and website owners) to join the ring.

I'm hoping that good webrings can hold the integrity of human networks on the web.

mschuster913• 1 year ago

Back Then (tm) you didn't have to fear getting hacked all the fucking time. Shodan didn't exist, Metasploit was in its infancy, you needed to know where to look to buy DDoS and even then, half the offerings were scams themselves. The biggest threat was people abusing your SMTP server.

Today? Run just a plain Apache without a domain name, just a plain old IPv4 and listen on ports 80/443. In seconds you'll get hit by a barrage of exploit attempts, scanners, god knows what. And manage to get something successful, you'll get a few emails that say "pay us X dollars in BTC and we'll leave you alone", and if not you'll get booted off the net if you don't hide behind Cloudflare.

Discovery is also not a thing any more. The old "rings" have long since passed, RSS feeds went down the drain with Google Reader (IMHO, that decision effectively killed off the blogosphere), and Google Search got completely poisoned with SEO.

The Internet has become the Wild West. Law makers have begun to notice but unfortunately they're so utterly braindead that fierce opposition to their regulation attempts is the only choice possible, so nothing happens. Rinse and repeat.

stirfish2• 1 year ago

I have a server in my pantry hosting some stuff, including an anonymous ftp server. I get a bot that uploads fake screensavers once in a while, and one time someone on 4chan made a million empty directories in it before I turned it off for 30 minutes and they got bored.

I might be naive about getting hacked, but it's been up for 5 or 6 years now and nothing bad has happened.

Edit: actually closer to 12 years total now, wtf I'm aging?

xtiansimon• 1 year ago

“The Internet has become the Wild West”

That phrase struck me as odd. I had to reread the comment. I already associated the metaphor with the 1990s web others are lamenting as gone. They refer to the WW of content; “wild” referring to sparsely populated lands. Your WW is “wild” or diverse lawlessness, vandalism and unruly (human) behavior. I suppose both are true—were true.

pabs32• 1 year ago

Webrings are still a thing. Same with RSS.

https://wiki.archiveteam.org/index.php/Webring

Ferret74462• 1 year ago

That web still exists and is still just as easy (read: hard) to find. I'm sure you can find hundreds of such sites by recursively asking people here. The commercial web didn't make it harder to spin up your own server after all, quite the opposite.

1vuio0pswjnm76• 1 year ago

https://172.93.49.252

https://172.93.49.252/settings

https://172.93.49.252/submit/

This website has some forgotten www features that are becoming rare thanks to Big Tech, Big CDNs, and their followers:

1. It does not require SNI (SNI was added to TLS in 2003)

2. It publishes a stable IP address. See https://172.93.49.252/submit/

One can use an IP address _or_ a domain name (any domain name will work) to reach it

3. It is available via HTTP as well as HTTPS

http://172.93.49.252

For old time's sake, try requesting via HTTPS and then HTTP and see if you notice a speed difference

To get a starting list of potential web sites, freely available DNS zone files (public information) remain available

Every major www search engine has used these files at some point

https://czds.icann.org/

People often refer to DNS as the internet's "phone book"

If that is true, then DNS queries are "directory assistance"

The actual phone book is the zone files

Perhaps we need a wiby.me for DNS to select from all those NS and A records that go undiscovered

djbdns has a program called "random-ip"

   random-ip|sed 1q

Besides PTR queries, there is an adundance of freely accessible databases to fecth information about an IP address

This can often lead to a discovering random website

The type of website that is worthless to so-called "tech" companies and their online surveillance advertising "business model"

1vuio0pswjnm76• 1 year ago

Note to self. Originally received +1

Havoc• 1 year ago

Sure feels like a key piece of the organic internet is dying.

Large amount of content has already moved behind walled gardens. AI is going to kill blogs etc. That doesn't leave much. Bit of niche stuff like lemmy/bluesky/hn I guess?

Gareth321• 1 year ago

I think even Lemmy and BlueSky are mostly bots. As much as I loved the anonymous wild west of the internet in my youth, I have come to accept that that is dead, and AI/LLM/bots will finish off its carcass. I foresee a future where humans are verified using some kind of trusted mechanism so that we at least know when a website, content, or comments comes from a real person. I value this far more and I think so do most.

spacemadness2• 1 year ago

Until we find an economic system that isn’t hellbent on individual greed and political and state groups hellbent on spreading propaganda, this is what we’ll have. The only thing that made the internet great in the before times was these groups didn’t take it seriously or know how to control it.

viraptor2• 1 year ago

AI is still bad at giving enough information and won't preserve the author's tone. I'll still take searching on Perplexity but reading the details on (for example) https://ridiculousfish.com/blog/ over reading only the dry, possibly wrong summary. Blogs will be ok. Maybe the spam blogs will die, which I'm happy with.

mbmjertan• 1 year ago

I can’t find the measurement methodology for Brightedge’s whitepaper, which concerns me because the margin for error seems huge, but Cloudflare’s numbers do make sense.

Both are however in line with my personal experience and observations regarding how people around me are using the web now.

I find myself using web crawling in LLMs a lot more, and search a lot less. My reasoning follows, and I think most people would agree.

- When I’m looking for some relatively obscure information which I’m not sure where to find or which would require hours of research for me to find, I use ChatGPT (usually o3 with deep research) and refer to citations for more information regarding a topic. This saves me hours of investigating, which I usually don’t have for something that’s just a curiosity. A friend also used deep research to find papers highly relevant to a topic he was working on for his med PhD in minutes, claiming that just searching through PubMed to find such papers would take him days - and probably less successfully.

- When I’m looking for something specific in regards to a topic I’m relatively familiar with, I use search (usually Kagi, unless at work where it is banned!) to quickly find reference material.

- LLMs (and engines like Kagi) let you skip through the SEO spam you’d usually skip through when using Google, as well as letting you search more easily (due to better natural language understanding than classical query engines). The quality of search results had been diminishing for years (geeks4geeks ranks higher than SO on Google), so it’s not surprising people turn to tools which produce better results. It’s like being shocked that people are driding cars instead of riding donkeys.

A particular example is that I looked up a DTC that my car was throwing. I googled it first, and got a results page consisting of forums that said nothing, paywalled generated sites that also didn’t provide any info, scammy Scribd clones hosting diagnostic manuals for the wrong car model and ad-ridden garbage sites that just claimed “oh it could be anything, just take it to a mechanic”. ChatGPT gave me an exact (and correct) answer in seconds.

This is an expected result of what we have done to the web, and it should surprise nobody. I’m only sad that genuine, small online communities are dying in favour of walled gardens, but that’s an entirely separate discussion.

jfengel2• 1 year ago

I read that as "ALPOcalypse". As in, we're all now dog food, perhaps.

I need to switch everything to a font with crossbars on the upper case I.

zkmon• 1 year ago

Every change will be met by other changes, ultimately leading to an equilibrium. A single change can not continue with its trend, as long it is connected with an ecosystem. It must make deals and find a balance.

In this case, the organic content will lose against the invading crawlers, either way. If they build a fort against the bots, they go undergound and remain unseen by their customers. If they don't mind bots, bot summaries would become those forts, with the same effect.

So ultimately the websites have to realize that bots are the only customers they can expect. Humans are long gone, as no human would ever need to click a link, if they are fed with the summary walls.

To realize this, start a blog and see how tough it is to get any meaningful attention to your great content.

illiac786• 1 year ago

The new equilibrium might just be worse than the old one though.

samrus2• 1 year ago

what do we think the post LLM internet will look like?

essentially whats happening here is that LLMs have removed the NLP barrier to automating information flow. thats what agentic systems are under the hood, once you realize alot of the "agentic" stuff boils rule-based engines and hand built state machines. and it will be until the next AI winter gives us proper dynamic modelling of an envirenment, rather than static next token prediction

i think RAG replacing the traditional reverse indexing will mean information wont be tightly linked to webpages anymore. meaning no ad revenue and SEO dying. which i think is good for the internet, because ad revenue just encourages information rent-seeking. this obviously assumes AI RAG is reliable.

i think without the ad revenue profit motive, information needs to be generated some other way. journalists and bloggers need to be supported somehow. crowdsourcing like patreon and stuff could help, but that just shifts the capitalism from optimizing for sensationalist clicks, to optimizing for cult of personality, which i think will become the norm for information generation unfortunately. everyone will have to make content, rather than news to do this for a living.

but i think that wont have the pull that ad revenue does. so i think information generation shifts alot more to amateur bloggers and taxfunded government mouthpeices

for actual services, i think we get more supercharged APIs like MCPs and that makes economic activity much less frictionless. its hard to know if this will just be a quantitative change or an evolutionary one

ones this is the labour displacement. how would all the people who were doing the jobs caused by the impossibility of NLP automation make a living? i think going back to the idea the "agentic" is just rule based engines with automated NLP, which is similar to coding (which is just making rule based engines with automated Formal Language Processing) i think "prompting" (designing, implementing and maintaining rule based engines in natural language) will become the new coding (which, again, was the same thing but making rule based engines in formal languages). essentially LLMs are just the next evolution of compilers, and people will learn to use them as such.

what i have no idea about is what sort of evolutionary change this will bring up, the way compilers brought when they replaced writing machine code. i think formal language programming will still be done by people, like machine code is still written, but the newly unlocked natural language coders will far outnumber them. so then what will be unlocked by this higher level of programming? like what will be the internet of the LLM era?

ianbooker• 1 year ago

Not so sure if Google "still controls 90 of web search" and if this means what it meant three years ago. Many prompts could have been a web search. The metric of interest must evolve here.

The AIpocalypse is here for web sites as search referrals plunge

Related Stories