Google Search Results Plagued with spam “.it” domains

https://community.cloudflare.com/t/suffering-from-the-spam-pages-using-cloudflare-on-google-search-result/283152

by pyinstallwoes • 4 years ago

234 206 4 years ago

community.cloudflare.com

pyinstallwoes3• 4 years ago

Also noticed here: https://support.google.com/websearch/thread/118733416/lot-of...

Locked?

rwmj• 4 years ago

I've been gettng these spam .it domains for years and years, this is nothing at all new.

politelemon• 4 years ago

What does Cloudflare normally do with spam sites, is it a hands off approach or do they do some policing?

yellow_lead• 4 years ago

Hands off till it gets upvoted on HN

charcircuit• 4 years ago

They do the bare minimum. You can report sites for abuse and they will take them down. It doesn't appear like they do anything to proactively stop similar sites so the person can just make a new account and domain and be back in business.

miyuru• 4 years ago

cloudflare abuse department is really lacking.

Their abuse form is getting abused too. It sends an email to site operator and the server hosting company in single submit so its getting abused. It not even have a captcha.

https://abuse.cloudflare.com/

breakingcups2• 4 years ago

They will do nothing, and it is a feature.

The only thing they'll do is forward the complaint to the user. Leaving you with no recourse other than to take legal action before Cloudflare will lift a finger.

Unless there's CSAM, of course.

sammy22442• 4 years ago

You need to remember that Cloudflare isnt a host

bad416f1f5a2• 4 years ago

This… seems about right?

A trademark dispute is a civil issue between two parties. We have legal systems to solve these. Cloudflare should ensure that their customers get timely notification of complaints, and that’s pretty much it.

beardyw• 4 years ago

Presumably these folk take advantage of cheap/free domain offers wherever in the world they are.

pyinstallwoes3• 4 years ago

I've actually experienced this and it is not related at all to the device. It was related to the signed in google account across networks and devices.

01acheru2• 4 years ago

I think you are right, https://www.register.it/ is offering free domains for 1 year since some time.

peoplefromibiza7• 4 years ago

it's not that straightforward though.

to register an .it you must prove you are a person or a business working or residing in one of the EU member states and need to provide the ID of a person who's gonna be listed as admin-c of the domain.

mkl95• 4 years ago

.it may be the .tk of the 2020s

toastal• 4 years ago

Shame too. I was hosting my personal site on .tk when I was broke out of college, but often a link too it was automatically flagged as spam.

password1• 4 years ago

Why? .tk was popular because it was free, so it was really useful for teens and young adults in an era when you still had to host things somewhere if you wanted them online. On the other hand .it is the tld of Italy and used legit by all businesses of EU's third largest economy.

ricardobeat• 4 years ago

.it is currently being offered for free / €1

qalmakka4• 4 years ago

Just, no. Plenty of legitimate websites under .it, basically every single Italian company plus all localized versions of international websites (apple.it, google.it, ...)

aaaaaaaaaaab2• 4 years ago

basically every single Italian company

Sounds about as trustworthy to me as a .tk domain.

marginalia_nu• 4 years ago

Whatever domain name is cheap is going to be plagued by spam.

I think in recent time, .icu and .xyz have been the most problematic, to the point where you to this day probably don't want to host a mail server on those domains.

The same with cloud providers. A fairly significant amount of sketchy websites seem to be hosted on cheap cloud providers with weak rules enforcement. I've taken to blocking all of Alibaba's IP ranges from my search engine crawler, the signal to noise from those sites were so bad it just wasn't worth looking for legit content.

jwally• 4 years ago

Anecdotally gmail has been doing a miserable job filtering spam for the last 5 months or so. For me it used to be pretty bulletproof - one of its best features.

Now I get something from McAffee Pratners(sic) every other day warning my computer is about to expire. Back in May I kept winning things from Home Depot and Lowes; and gmail would categorize it as "forums".

No idea if its related, just odd.

sofixa• 4 years ago

I occasionally have spam that slips through Gmail's filters, but when I explicitly mark it as spam it disappears nothing of the same type reappears again.

silversmith• 4 years ago

Add another anecdote to the anecdata pile. Past three months, McAffee and north american shopping chain spam is breaking through Gmail filters. And reporting as spam does not help. I assume they've been somehow building Google reputation for the spam accounts.

edvards• 4 years ago

Been having the same issue on my old email. a LOT of spam going past the filter.

pwiecz• 4 years ago

On the other hand over last few months majority of post from sma few of private Google Groups, I'm member of, keep getting wrongly clsssified as spam.

I don't know if it's related either.

thelollies• 4 years ago

Similarly I've been getting smashed with gmail spam, google calendar spam and google drive spam for around 6 months now after never previously getting any and despite reporting most of it.

tempest_• 4 years ago

Yeah the drive spam started for me last year or the year before. It took a break but in the last couple months has returned with a vengeance.

I had the Slack google drive integration and I needed to mute it because it was couple doc invites every few hours.

pverghese• 4 years ago

I disagree. Prior to Gmail i used to get thousands of spam email everyday Now everything is filtered. Barely get any.

The added benefit is I don't get any tech calls for help from my parents who also don't end up clicking random spam and wondering why bad things are happening

echen• 4 years ago

I've been experiencing the same. Here are a bunch of egregious spam mistakes we collected from different people to illustrate the problem: https://www.surgehq.ai/blog/are-the-spammers-winning-failure...

hn_throwaway_99• 4 years ago

Agreed, same experience, all with the same format, primarily from some sort of outlook.com domain.

hyperdimension• 4 years ago

Ironic, since Microsoft is the worst (in my experience) at being a giant black hole to emails sent from an otherwise well-configured (SPF, DKIM, DMARC, non-SBL-listed IP, &c) but not major SMTP host.

krono5• 4 years ago

Every single one of these results carries the `html` filetype as part of their URL is my experience. This is likely a consequence of the useragent-based switcheroo technique they use to fool Google.

Just blanket block the lot with the following uBlock Origin filter:

    google.*##.g:has(a[href*=".it"][href$=".html"])

Google ain't going to fix itself ;)

permo-w• 4 years ago

I'm sure there are plenty of non-spam html pages based in Italy too

krono5• 4 years ago

Considering the crowd that trade-off that seemed too obvious to mention.

jacooper• 4 years ago

Or use Brave search, which honestly from my experience is much better.

peoplefromibiza7• 4 years ago

cool!

now s/\.it/every TLD/ and you solved domain spam forever.

You might not know that 99.99% of .it domains with urls ending up in .html are completely legit, including some official government one.

nkrisc2• 4 years ago

Since uBlock is run on the client, unless you’re Italian or interested in Italian sites it doesn’t really seem like much of an issue.

I could block all .it sites on my network and I’d likely never even notice.

fijiaarone• 4 years ago

But not many of the official government ones.

qalmakka4• 4 years ago

Blanket banning a whole TLD is stupid. One thing is blocking some obscure stuff like ".su", but .it? It's just too big, and arguably unwise if you are in Europe where having to connect to Italian websites or services isn't a remote possibility.

krono5• 4 years ago

This merely hides Google search results in my browser.

No network connections are blocked...

seumars• 4 years ago

What’s this useragent switcheroo?

krono5• 4 years ago

Browsers and other programs can use the User-Agent[1] header to send along a bit of information about themselves with each request.

This and other information is then used to filter out various types of visitor.

In this case, requests claiming to be a Google Search crawler will receive a boring page with lots of text that it can index and use as search results.

Most browsers' devtools let you change your user-agent string, and a listing of the ones used by Google crawlers is publicly available. Not saying that you should, but you could check this out for yourself... entirely at your own risk of course :)

https://en.wikipedia.org/wiki/User_agent

https://developers.google.com/search/docs/advanced/crawling/...

LinuxBender• 4 years ago

In addition to this if one runs unbound as their DNS on their home router and they block DoH then one could add

    local-zone: "it" always_nxdomain

to NXDOMAIN all requests for the .it TLD and protect non browser devices. I use this method to stay off sanctioned country TLD's and to remove the cheap/free spammy domains and TLD's that often contain more malware than anything useful.

oefrha• 4 years ago

What can search engines do about user-agent based content differentiation? Say my robots.txt allows Googlebot and nothing else. If Google attempts to double-check with a covert user agent, robots.txt is violated. Assign humans to review reported pages? It’s pretty easy to swamp a manual system like that. Just forget about robots.txt?

foobarbecue• 4 years ago

I remember reading on HN years ago that Google bots have never honored robots.txt, but I don't actually know

hombre_fatal2• 4 years ago

robots.txt is just a guideline between well-meaning actors for the majority of their traffic, like helping a bot not waste its time nor your bandwidth by crawling dynamically-generated, endless-scrolling /calendar.php pages. Google does use it to that extent.

It's not a firewall.

Seems like you're describing cloaking (https://developers.google.com/search/docs/advanced/guideline...), one of the oldest SEO tricks, and you can imagine that search engines started defeating it on Day 2 of crawling the web.

fredgrott• 4 years ago

there is also the spam of name.ru.com domains as well

Warning, do not click on those links as you will get your PC infected.

sammy22442• 4 years ago

Wow from clicking on a link on a modern browser? New 0-day?

sub7• 4 years ago

This headline would be accurate without the “.it” domains part

jeroenhd2• 4 years ago

I don't think this problem should be solved by Cloudflare. Cheap domains will always exist and they shouldn't be a problem. The problem lies with Google and its failure to detect these spam sites.

Surely Google can spare an engineer or two to do a deep dive into the way any one of these spam sites manages to get itself to the first page of Google, work out their scheme, and fix the algorithm? This problem isn't exactly hard to reproduce!

Hrundi• 4 years ago

The fact that this has been going on for several years makes me believe Google either doesn't care or the problem is particularly hard to fix (less believable)

frankfrankfrank• 4 years ago

Google has at the very least neglected its search for many years now and recently has also actively made it worse through all the censorship and thought control stuff. I find it rather surprising because essentially all of google’s success is lynchpinned by search. All it would take is for a narrative to dominate that the best results can be found elsewhere, which does not seem particularly remote, considering how much damage google has done to its search.

ehnto2• 4 years ago

It has been an ongoing battle for 10-15 years at this point. Search engines are constantly battling people trying to game their systems. I have to wonder if Google hasn't lost the thread a bit, inside their surely quite complex algorithm black boxes.

For a while now Google has suggested that the best way to rank well is to have human readable content and focus on user experience. At the same time, natural language generation has come leaps and bounds, to the point where sometimes even I, a human, can't tell if an article has been spun by a bot or not.

So if Google starts ranking human readable content, and robots can now produce human readable content, what is the next ranking signal they can use to differentiate spam from humans? Are we going to end up with "Verified Websites" ala verified Twitter handles?

A huge portion of the web at this point is just bots communicating with eachother, and legitimate business systems having to process bots participation on the internet. I imagine the portion of the web that Google crawls that is legitimate versus that which is bot generated would surely be majority bots, just because of how fast they can generate content. One thing they can't do as easily though is register domains, so it may be one of the better points of defense.

jaimex2• 4 years ago

It's incredibly easy to fix. They don't care as they have a monopoly.

worldofmatthew5• 4 years ago

I noticed issues since Matt Cutts left. No one care anymore. There are AI generated website that have been running for years, ranking highly in Google.

hericium2• 4 years ago

I don't think this problem should be solved by Cloudflare. Cheap domains will always exist and they shouldn't be a problem. The problem lies with Google and its failure to detect these spam sites.

The problem exists outside (Google-controlled) web: with (not fully Google-controlled) email, too.

Around 2020 I did a per-tld checks on wanted/unwanted messages (ham and spam). With thousands of messages sent from .xyz domains (envelope sender host or PTR record of sending host; I ignored the From header) there wasn't a single legit message. 100% SPAM.

axsharma• 4 years ago

The irony, Google/Alphabet uses ABC.xyz.

johnklos3• 4 years ago

I really don't get why so many people are willing to give Cloudflare a free pass on stuff like this. Why is it OK for a company to facilitate and host (1) thousands of scam domains, making reporting arduous and ineffective?

Anyone trying to infect others with Trojans and viruses just need to check user agents or use dynamic redirect URLs, and suddenly this clearly illegal activity becomes black magic that is way beyond the comprehension of the folks at Cloudflare.

Cloudflare is basically making the shittiest parts of the Internet safe for scammers and spammers, and this is just one example.

If that's not bad enough, they're trying like crazy to become a monopoly. If this what they do now, imagine how bad it'll be when they control even more and feel even more immune to making money from scammers.

(1) Hosting is providing services on the Internet without which a site would not function. Providing DNS is hosting. Providing proxy is hosting. Providing email is hosting. Don't fall for Cloudflare's "we don't host" bullshit.

Nextgrid6• 4 years ago

I disagree.

For example, SO copycats are legitimate in that they respect the license and otherwise just serve the content to whoever sends them an HTTP request. As far as I know they don't spam links to their domain anywhere. They are low-quality and of dubious utility for sure, but I'd rather not make the Internet a place where you need to prove quality & utility to someone to be able to host an HTTP server.

The real problem is that a dumbass like Google comes along, sees this and decides that it should rank higher than the source content.

jeroenhd2• 4 years ago

Cloudflare should take action on reported domains and their owners, especially if those domains are malicious.

However, I don't want Cloudflare to preventatively police what is and isn't a bad website. When these scam sites go live, they can quite easily contain real content (say, a blog, with articles written by AI good enough not to be immediately obvious) and then change into malware on a schedule.

Cloudflare can't see what code customers run on the backend and that's probably a good thing. They're already holding too much power over the internet and requiring the backend to be transparent would only make them more in control of the web.

Any registrar hosts thousands if not millions of spam sites because every single one of the billion registrars have DNS set up in some way.

Despite being almost exclusively used for spam and amateur projects, the .TK TLD barely shows up in Google. Spam sites are a symptom of other services linking to them and making them worth the investment. If Google, Bing, Qwant and Yandex weren't falling for the SEO scams these scammers use, we wouldn't have this problem.

Hosters have some immunity by design, and that's very much a good thing. They have to respond to abuse complaints, but they're not responsible for filtering out all of their customers. Requiring them to do so is exactly what the EU is trying to force upon the internet, which is terrible for online freedom.

2OEH8eoCRo0• 4 years ago

Cloudflare should worry about what sites Google is choosing to index and show?

This is clearly Google's issue.

nousermane• 4 years ago

"Cheap domains" is not a thing. $25/year domain for a personal website is kinda pricey. But scam/spam operator can pay that and more pretty darn easy.

worldofmatthew5• 4 years ago

Who is charging $25/year for domains?

schroeding2• 4 years ago

There is at least one registrar that gives away .it domains (and apparently .eu domains? WTF?) for free for one year[1], with no major strings attached (as long as you cancel after the first year) as far as I read, correct me if I'm wrong.

Why they decided to ".xyz the TLD", I don't know. ¯\_(ツ)_/¯

[1] https://www.register.it/?lang=en

thejosh3• 4 years ago

There has been a huge influx this year with the amount of sites that simply scrape SO and then have the exact content on their site. It's a pain, and there is no official way to remove them.

I thought that this was a massive nono from Googles side, has something changed?

disruptiveink• 4 years ago

I'd like to get actual confirmation of this, but my vague feeling is that, once upon a time, Google Search would get "updates", as in, actually deployed code that would change the rule of the game and most of the previous dirty tricks would become unusable, leading to people to go out and find out new ones.

This changed with the Google "machine learning" days, where you no longer have humans at the helm laying down explicit rules, so no more "change the world" updates, you can only slightly nudge the parameters towards what you want, meaning the same old tricks keep being effective for far too long.

dewey• 4 years ago

Google Search would get "updates", as in, actually deployed code that would change the rule of the game

That's just what the scheduled "core update" days are now: https://developers.google.com/search/blog/2022/05/may-2022-c...

nerdawson• 4 years ago

The “May Core Update” which recently rolled out impacted every site.

A lot of updates are targeted at specific problems such as low quality product reviews but there are still broader updates taking place.

atwood222• 4 years ago

I have no evidence of this, but the ad load on the returned results has gotten way higher. In theory, ranking sites that display Google ads higher would be a very easy knob for Google to turn to increase profit. The SO scrapers probably have Google ads on them, making them more profitable for Google.

hombre_fatal2• 4 years ago

I ran into so many Stack Overflow "mirrors" yesterday like this: https://www.anycodings.com/1questions/400836/swiftui-update-...

10 years I gave up on a large project where I rehosted and organized dead Usenet forum content because Google's dupe-penalty detector was too good and too aggressive for content that you could barely find beyond a six-year-old cache hit where the origin website was long gone.

Meanwhile these Stack Overflow scrapers are just `<html>{copy-and-paste}</html>` and the same domains are still alive despite years of cloning.

Looks like it's time to boot my project back up.

Nextgrid6• 4 years ago

Turning the knob one way explicitly might raise some anti-trust concerns, however the same motivation can be used to avoid turning the knob the other way and this can be done much more sneakily without leaving clear evidence - simply don't allocate budget/etc to projects that would turn the knob the other way and you're done.

panarky2• 4 years ago

This is a very old conspiracy theory that's been repeatedly debunked.

https://www.searchenginejournal.com/ranking-factors/google-a...

1597• 4 years ago

I've noticed this with youtube. Even though I'm on desktop with an adblocker they repeatedly autoplay the same video with a creator embedded crypto promotion at the beginning (especially when it would be plausible to infer I'm asleep from user interaction and clock/watch time). Must be getting a cut (plus scamming the ad buyer).

Quenhus• 4 years ago

Here is my uBlock filter with hundreds of GitHub/StackOverflow copycats: https://github.com/quenhus/uBlock-Origin-dev-filter

It blocks copycats and hide them from multiple search engines. You may also use the list with uBlacklist.

thejosh3• 4 years ago

This is fantastic! This is exactly what I needed, thanks!

Phlogi• 4 years ago

This even works on Firefox Nightly on Android. Thanks a lot!

colordrops4• 4 years ago

With these two pieces of data:

* the identical text copied from stack overflow should be easily identifiable

* volunteers put together a list of these sites themselves

it should be obvious to Google apoligists that Google is either negligent or intentionally allowing these sites in their search. I'm sick of hearing about how "the world is different" and it's an "arms race" between spam sites and google. Bullshit.

SmellTheGlove• 4 years ago

You rock. Thank you.

maxwelldone• 4 years ago

I've been using this uBO filter since someone recommended on a different thread and it's been great at removing those annoying sites from search results: https://github.com/quenhus/uBlock-Origin-dev-filter

burtekd• 4 years ago

The author acutally posted above your comment ;)

skilled3• 4 years ago

I had this topic brought back to my mind yesterday as I was doing some research using the Ahrefs keyword tool. I do believe it would be possible to create a very large dataset of these copycat sites (using Ahrefs) to be used as a blacklist in various filters/extensions.

But the crazy part is that, for example - Ahrefs says that StackOverflow has "Organic traffic" in the range of 22 million per month. A lot of these copycat sites, at least the ones I saw - have a traffic range anywhere from 10k to 500k per month.

I mean, it's pretty insane just how well such sites can rank in Google, and you bet those copycats are making absolute bank from ads even if the majority of developers immediately close the site.

There's a lot going on with Google Search these days, a lot of people are complaining that sites that scrape content can easily rank really well for long-tail keywords. One case in particular, a site will scrape Google to collect "featured snippets" and "people also ask" - then combined anywhere from 20 to 40 of these answers and publish them as a blog post.

None of the words are changed, all questions/answers worded exactly the same. And Google puts these sites on page 1.

What a joke.

SheetPost• 4 years ago

bet those copycats are making absolute bank from ads even if the majority of developers immediately close the site

I bet the majority of developers block ads

helsinkiandrew• 4 years ago

I do believe it would be possible to create a very large dataset of these copycat sites

Would they just move to creating and using new domains with the same content as soon as traffic to the old becomes drops? (What looks like the spammers in the original post are doing)

But something does need to be done to these sites.

ben_jones• 4 years ago

For all we know the sites have better internationalizations and cater to audiences invisible from a US-based perspective.

lamontcg3• 4 years ago

then combined anywhere from 20 to 40 of these answers and publish them as a blog post.

yeah i've been hitting a ton of those lately.

chewz• 4 years ago

What a joke.

It is simple. Google is making more money from copycat sites then from original content...

cyanydeez• 4 years ago

I'm sorta happy Google is turning to garbage.

Actually forcing a search engine back into the reliable index of valuable sources would be great.

Imagine you to a white list approach to a search engine where a human or AI does an approval first.

pyinstallwoes3• 4 years ago

Similar to YouTube search results. Lots of spam videos. No way to block a creator. Totally ruins it.

Surfactant7• 4 years ago

There's a simple way around that. Nothing to install. Nothing to update.

Just go to SO and use its search bar. It's actually quite good.

I mean, you know that's where you'll want to find the answer anyway - not some random corporate webpage or ad-infested splog. Why not cut out the middle man?

Only if that fails do I bother with Google.

burnished2• 4 years ago

Huh, you know, you're right. I recently did that and it was fine.

I think a lot of others formed their opinion (myself muchly included) about this from sites where the search bar was a joke played on people.

Edit: let me upgrade that 'fine' to 'great', now that I think about it it was actually better than a google search which was not my previous experience.

watchdogtimer• 4 years ago

Or, if DuckDuckGo is your default search engine, you can append ' !so' to your search term.

avereveard3• 4 years ago

Google index used to be fairly more competent at finding relevant issues for a query, especially if some words were synonyms of what found in the snippets at even loosely related

thejosh3• 4 years ago

Because it's not just SO I want answers from. It's blogs people post, social media such as Reddit, HN, etc.

I've found some fantastic articles out there, yes SO is a fantastic resource but there is an entire internet out there :-)

ofou• 4 years ago

just switch to you.com

minutillo• 4 years ago

My theory is that one of the inputs to Google's ranking algorithm is now "how much money would we make from this click?" A click to SO has a small number of ads which are obviously ads and easily ignored. A click to the average scrape-jacked SO page has dozens of ads using every dark pattern in the book to generate accidental clicks.

fluidcruft• 4 years ago

One of the other commenters above made the claim that SO runs yahoo ads. If that's true then from a Google perspective, the click has either zero or negative money-making value.

Maybe that means we should be searching in yahoo rather than google.

xbar• 4 years ago

This reminds me of this one site that simply scraped all the open source code it could see and then produced AI-generated copies.

endofreach• 4 years ago

https://news.ycombinator.com/item?id=29403947

anonred• 4 years ago

Other search engines allow you to block domains from showing up in the results. I’ve switched to Kagi out of frustration and honestly it’s as good or better than Google just because of that one feature.

MaxDPS• 4 years ago

I've been using uBlacklist and it works really well. It even lets me highlight specific websites so I have a better chance of seeing them if they are further down the list. https://iorate.github.io/ublacklist/docs

phreack• 4 years ago

It took me a while for no good reason but I finally got an unofficial extension to add a "block" button to search results. It immediately improved my experience, I can't recommend it enough. No more Pinterest, SO clones, useless Quora spam, with very little work. I can't believe I didn't do it sooner.

aaaaaaaaaaab2• 4 years ago

Some ex-Googlers say that someone ran an AB-test, and it turned out that per-search revenue was decreasing when these sites were blocked.

nottorp• 4 years ago

No, google search is plagued with spam from any domain. And even the non spam results are useless.

H8crilA• 4 years ago

It may be that deep learning is now increasingly used to generate the spam. It either is or will be used for spam generation A LOT. Frankly it seems to be the most promising commercial use-case for the large language models.

mrkramer• 4 years ago

This is 2nd and 3rd search page results spam, I get spam phishing websites on 1st page of search results when I search for certain ecommerce websites. Google is done.

jl6• 4 years ago

I recently cursed google search results when trying to research an actor's birth date. There were two dates given on Wikipedia and I wanted to see which one (if either) was correct. Google returned the actor's IMDB page (which listed a third date, and no source), and then pages upon pages of what appeared to be auto-generated sites that clearly scraped from Wikipedia, repeating one or the other of the Wikipedia dates.

This is not helping to organize the world's knowledge.

hericium2• 4 years ago

This is not helping to organize the world's knowledge.

And Google is not about organizing world's knowledge but creeping on people for YoY financial results.

richbell• 4 years ago

They're quoting Google's own mission statement; though, you are correct.

Google's mission is to organize the world's information and make it universally accessible and useful.

znpy• 4 years ago

This is not helping to organize the world's knowledge.

Oh they stopped doing that long long ago...

dazc• 4 years ago

A lot of actors lie about their age so I wouldn't hold out too much hope on getting an accurate result on that one.

I get your point though about the multiple results for something where there clearly is no authoritative answer.

seydor• 4 years ago

It must be the mafia. No other explanation

tremon3• 4 years ago

Yeah, but which one? The Camorra, Ndrangheta, Stidda, RIAA or FAANG?

randomperson_24• 4 years ago

I think Google can probably not fix it. Users will have to be manually reporting as spam. These websites on seeing traffic from Google's crawler bots show a perfectly legit and highly SEO optimized website, but for anything else show other spam. If Google starts indexing from random IP ranges, most websites would probably block indexing from “unofficial IPs” or some companies (esp in EU) would file some lawsuit against Google. The reason being that some pay-walled news article websites won't be indexed properly, as the “unofficial IP-ed” Googlebot will not get the paywalled content.

If a website lies to Google itself, I believe the only way to solve it is by reporting the search result as spam or Google contracts people to somehow visit all billions of web pages (again the same problem – from different IP ranges) to verify it as a legit page.

I would like to know how Google currently handles it and probably how it could be improved

Nextgrid6• 4 years ago

The practice of "cloaking" has been around for ages and I'm sure Google has (or at least had) solutions against it.

I'm not sure on what grounds could someone sue for crawling from random, unaffiliated addresses as long as the crawling isn't causing a denial of service (they can always check robots.txt using the main IP then use that to throttle crawling from random IPs as to remain compliant).

The reason being that some pay-walled news article websites won't be indexed properly, as the “unofficial IP-ed” Googlebot will not get the paywalled content.

Good riddance? That would be a welcome change.

panarky2• 4 years ago

Google has all the text they've scraped.

They can see all the domains that have served a given snippet.

They also have history to identify where each snippet was first seen.

If SO has a lot of traffic and a good reputation, and if the same snippet is found first at SO and then later at bunch of newly created, low volume, low reputation domains, then show the SO result and not the others.

Thorentis2• 4 years ago

The obsession with "machine learning" is actually making systems dumber. Google Search and Gmail spam filters are getting worse with each passing week, and I am almost certain the increasing reliance on ML is to blame.

patentatt• 4 years ago

I chalk it up to a cost benefit calculation. Google clearly isn't trying to eliminate all spam in search. It's not their goal. They are not trying to optimize for the user experience. They're trying to optimize revenue.

mod2• 4 years ago

They're trying to keep spam out of my inbox, and the spam rate has been increasing for me (and other HN commenters who frequently talk about it)

tremon3• 4 years ago

The competing explanation is that "machine learning" is actually making spam generator systems smarter, so spam gets harder to detect.

Thorentis2• 4 years ago

The type of spam I see in my inbox is anything but smart. But I agree that this has always been a cat and mouse game.

Ueland• 4 years ago

Note that the discussion is a year old. Around one year ago I wrote more about this "phenonomen" here: https://news.ycombinator.com/item?id=27993123

tl;dr: I managed to find the servers behind it, most likely anybody who are still affected can do the same thing I did pretty easily. We also followed the money, which is a tad more work.

kmfrk• 4 years ago

Remember the good old days of talking about a "semantic web"? Now we just get one Google results page of SEO'd garbage with no way to process them.

I can't help plug kagi.com, which has the amazing feature of grouping SEO'd stuff like recommendation lists together, so a thing that's contextually useful is still available but without polluting the other contexts.

midislack2• 4 years ago

Google hasn't given a shit about search since at least a decade ago. It's all about data collection via Android and Chrome OS, and gmail and docs. They don't need search to collect your data any more. Don't people actually know this? LOL

DharmaPolice• 4 years ago

All that data they've collected is only useful if they can sell something (i.e. ads) based on the data. AFAIK the majority of their income they get for ads is from search based ads.

midislack2• 4 years ago

Governments are happy to buy the data actually. In-Q-Tel, aka the CIA, was the first large investor in Google.

worldofmatthew5• 4 years ago

Google appears to stopped caring after Matt Cutts left......

jrockway2• 4 years ago

The Internet is quite a bit bigger than it was in 2016: https://www.internetworldstats.com/emarketing.htm

(Have no idea how reputable that data is, but it seems about right to me. In 2016 there were 3.6 billion Internet users. Now there are 5.3 billion.)

ehnto2• 4 years ago

Also imagine how many bots there are, and how fast they can generate content now.

manquer3• 4 years ago

Most that growth is in non English language users . There are only so many English speakers, we are talking about quality of English search results, they number of users for that has not doubled in 6 years.

meerita• 4 years ago

Dealwith.it

oliwarner• 4 years ago

The real question we should be asking is: Why Google don't care about their index?

Junk copypasta and news-squatting (posting regularly about the same thing with no additional data) is a decade(s) old problem. Crowd sourcing and verifying junk domains could be a weekend project.

But nothing.

Google Search Results Plagued with spam “.it” domains

Related Stories