Google Says 60% of the Internet Is Duplicate

https://www.seroundtable.com/google-60-percent-of-the-internet-is-duplicate-34469.html

by elorant • 4 years ago

20 14 4 years ago

WallyFunk• 4 years ago

Remember running one of those RSS-to-Wordpress plugins that scraped the post from some blog and just reposted it on mine. I monetized it, and it was shocking how much attention it was getting. Those were the days. I imagine this practice has grown even more prevalent.

Eleison23• 4 years ago

Google Says 60% of the Internet Is Duplicate.

arch-ninja• 4 years ago

I say 50% of the Internet is Duplicate. (N=2)

MuffinFlavored• 4 years ago

Google Says 60% of what they scrape needs to be filtered for duplicates?

booboofixer• 4 years ago

Google Says 60% of the Internet is Duplicate.

thinkmcfly• 4 years ago

What does this even mean? If I break up all of the data on the internet up into pieces, wouldn't every chunk be a duplicate of another chunk if we exploded the samples into small enough pieces?

Did anyone else not click the link tho?

hcs• 4 years ago

The light gray text in the slide [1] suggests that this is about multiple URLs returning the same content:

1. remove protocol duplicates - favor HTTPS
2. remove www/non-www
3. remove URLs with useless parameters (sessionID?)
4. remove slash/no-slash variant
5. [...] [che]cksum dups

I haven't seen the talk, but I guess that the 60% is after taking 1-4 into account and directly comparing checksums of the full response (body?)

[1] https://twitter.com/suzukik/status/1595679289873428480

rchaud• 4 years ago

That just means that Google's index of URLs contain duplicates, not the Internet itself.

Google indexes `/page` and `/page.aspx` as two unique pages with duplicate content. All that happened was that the search spider clicked a link to a server-rendered page that can be rendered with a non-canonical link.

Philadelphia• 4 years ago

Thanks mostly to Google themselves, with their algorithms rewarding useless copied content.

cryptos• 4 years ago

Yeah, I don't get it. Google happily shows all the useless Stackoverflow copies in the search result and I guess that Google could easily filter such garbage out. But they do not. Why?

matt_heimer• 4 years ago

Those duplicate sites exist to serve ads, Google profits from selling ad space...

tannhaeuser• 4 years ago

So does Google send users to sites with original content, or to those with Google ads and tracking?

olliej• 4 years ago

I too have seen googles search results for any technical topic.

Everything is afaict generated content that mostly repeats itself enough times to fit in ads.

out-of-ideas• 4 years ago

Don't forget to also thank the idea of "streaming" instead of caching; so they know exactly what, when, and for how long you listen/watch any content.

Google Says 60% of the Internet Is Duplicate

Related Stories