Spam sites in Google News

by gnewssheriff • 12 years ago

153 43 12 years ago

eli4• 12 years ago

Google News has always given off this vibe like it's being administered by someone as their side project. Very curious for one of the most popular news websites in the world.

tymekpavel• 12 years ago

Without going into too much detail, this is essentially true. The site is built on top of algorithms that were developed years ago, and there are some runbooks to help SREs keep the site running. However, there isn't much investment going on. Occasionally they'll have one engineer or an intern add a feature, but it's not permanently staffed.

Scoundreller• 12 years ago

Would this change if they started running ads on it?

I guess the question is, is the lack of investment because it's a cost-centre, or for some other reason?

r721• 12 years ago

I use the number of blog posts a year in a Google product's blog as an estimate for its importance for Google:

2014: 3

2013: 3

2012: 8

http://googlenewsblog.blogspot.com

So not very important in last two years as it seems.

mnx• 12 years ago

for comparison:

        Gmail:
        2014: 21
        2013: 25
        2012: 35
        2011: 84

        Reader:
        2013: 2
        2012: 0 (!)
        2011: 4
        2010: 16

marincounty• 12 years ago

Yea, I understand what Google is trying to do with their news site, but it'a not working--for me--in the Bay Area.

They have money to burn/waste/spoil Geniuses? Why not hire the best reporters and put together news sites for every country on the planet? Old fashioned reporting? Stop leaching off struggling news papers in order to obtain your content? The internet has destroyed so many good newspapers. Why not reverse the trend and bring back good journalism? Or just, give some of the struggling newspapers cold hard cash--in the form of a grant, before we lose them all? Or, they just become pet/vanity projects for rich tech guys?

kumarski• 12 years ago

Thank you for doing this.

danso• 12 years ago

I used to work on a news site and the process to get listed on Google News was very manual, at least compared to how you think the rest of Google operates. You had to have a physical address and phone number (though I don't think the two attributes are actually tested, i.e. sending a postcard via snailmail). You also had to configure your pages for the spider so that if your site delivers wire content (e.g. Reuters, AP) or does aggregation, the spider would ignore those pages, and only pick up your original news. The first time we submitted, the response we got seemed to indicate that someone had looked over our sitemap and saw aggregated news/wire reports, and so we had to resubmit. The process took a few weeks, IIRC.

That said, bigger news sites were probably whitelisted...it seems like the whitelist could be done algorithmically...but the number of new authoritative news sites, in theory, changes so slowly, that there's probably not really a need to do that.

walshemj• 12 years ago

actually no I used to work at one of the big publishers we did not get our properties listed automatically.

eli4• 12 years ago

You left out the most bizarre part of the process -- the insane constraints on URLs. For example:

"The URL for each article must contain a unique number consisting of at least three digits. For example, we can't crawl an article with this URL: http://www.google.com/news/article23.html. We can, however, crawl an article with this URL: http://www.google.com/news/article234.html. "

(Source: https://support.google.com/news/publisher/answer/68323?hl=en)

declan• 12 years ago

When I worked at CNET, we were (and still are) indexed by Google News despite not having three-digit numbers in URLs.

That's because your excerpt from Google News' guidelines left off a very important addendum: "Please note that this rule is waived with News sitemaps."

franze2• 12 years ago

yeah, and from the paragraph you just cited you left out the last sentence.

"Please note that this rule is waived with News sitemaps."

so basically the >= three digits rule doesn't apply any more (for years now), as a matter of fact i don't know why google still has in in there as it always leads to a lot of confusions.

i once consulted a big newspaper which were in an ongoing project to change their URLs just because one of their (middle)managers has read this spec...

tokenadult• 12 years ago

There is a lot of good information in the post submitted here, which should be actionable for Google if Google cares about the quality of news results. I'll note for the record that I have been using Google News as a news aggregator since the beginning of its existence, and since I almost use Google News in a logged-in condition, Google News responds to the way I have trained it about my news interests by mostly showing me stories from established news organizations all around the world, and not from spammy linkfarms.

My current gripe about Google News is that it pushes far too many low-quality or too local news outlets into the Editor's Picks section of my view of Google News, and even if I click repeatedly as those sources display "Personalize this news source" to display it rarely, the same podunk local TV station or trade magazine website will keep displaying in that section over and over and over. (I have already complained to Google through Google News feedback channels about this behavior. It should be possible to mark a source as NEVER appearing in the "Editor's Picks" section and have that selection be implemented until the user affirmatively turns it off.)

On the whole, I like Google News. But for sure if a site has eyeballs, spammers will try to grab those eyeballs, so the price of freedom from spam is eternal vigilance.

Animats• 12 years ago

One of the sites mentioned, "wtnnews.com" ("West Texas News"), is a straightforward content farm running Google Adsense ads. Here's their home site, "wtnmedia.com", "A global business-to-business media company where you can forge high level client relationships through the web, print, and events." Their advertising kit shows a base ad rate is $1,295 for 7,500 impressions of a banner ad, for a Cost per Thousand of $171. Nobody would pay that; typical CPTs today are around $1. So that's not their revenue source.

The use of the business addresses of other businesses is unusual. That's identity theft. Our SiteTruth system checks out the address of the business found on the web site, and it brings up StreetView pictures of auto repair shops for some of those sites. Lots of sites don't give an address, or use some mail drop, but use of fake street addresses is rare. It tends to draw tax collector and law enforcement attention.

Google's anti-spam efforts have never been very aggressive. They filter out some of the worst offenders, but don't try hard to get rid of bottom-feeder sites like these. If Google didn't put spam sites in news results, how would they get revenue? Google News itself has no ads, and many of the top news outlets have their own advertising systems that don't involve Google.

a3n2• 12 years ago

"... where you can forge high level client relationships ..."

Great double entendre.

graupel• 12 years ago

The funny thing is that as a large publisher, it's not links on news.google.com that we really care about, it's landing in universal Google search one-boxes as a result of being indexed in Google News - that's where the traffic really comes from.

Destitute• 12 years ago

Yea, there's a lot of sites that take advantage of this like The Christian Post who target high search keywords for their "news" that has no relevance to the actual term.

Search for "Stream NFL games" and "Watch NFL football online free" and other variants and see them pop up for everything with no mention of actually streaming games online (obviously).

See: http://www.christianpost.com/news/carolina-panthers-vs-seatt...

notadoc• 12 years ago

Many very top sites do this with any popular news term or keyword, its incentive to create content farms. Madness.

everydaypanos• 12 years ago

+1 for exposing the fake writer profiles. Very amusing!

alternatives• 12 years ago

We need alternatives to Google. We can't have one company deciding what gets seen on the WWW.

It is a great search engine, Google changed the world. But now we need alternatives. We can't have their editorial staff dictate what should and should not be seen on the WWW.

If you look closely at the top search results, they're mostly big spenders in Adwords. Ebay, Amazon, Expedia, TripAdvisor, Yelp, Answers.com and several others. It's blatant conflict of interests. There is no way those search results are "organic". Basically the WWW has become controlled and curated by Google and every site that gets seen must conform or be destroyed.

We need alternatives to Google. We need them urgently.

richthegeek• 12 years ago

It's quite possible that the results are "organic" in the naive definition by following this process:

1. {site} starts at rank 1 due to low visitors/connectivity

2. {site} pays for adwords driving new visitors

3. visitors link to the site

4. rank of the site increases allowing increase adwords spend

5. goto 2

The suggestion that Google artificially inflates the rank of big spenders is maybe conflating correlation and causation. On the other hand, it's not impossible or even that difficult to believe.

username2232• 12 years ago

We need alternatives to Google. We need them urgently.

Thanks to the 80/20 rule, we have them. Google is still unmatched for edge cases in maps and search, but something else works 80% of the time, and Google is spending 80% of the effort for that last 20%.

ooOOoo• 12 years ago

Isn't it the opposite? Google maps works 80% of the time, but for the 20% remaining, you need to look at more specialized maps. For example look at https://www.google.com/maps?q=lamma+island : almost nothing, just of few paths. Now look at http://www.map.gov.hk/ : all paths and houses in Lamma are precisely described!

runn1ng2• 12 years ago

I started to use Bing quite a lot lately.

For a very simple reason - I use Tor browser a lot, Google blocks it and constantly shows me ugly captchas (not even ReCaptcha) and sometimes blocks me entirely, Bing just works. (Startpage - which is a Google skin basically - shows captchas too, and doesn't even redirect me to the actual search after the captcha. DuckDuckGo has unusable results.)

So I am using a Microsoft service to be more anonymous. Which is kind of funny.

andyrj• 12 years ago

It isn't just the tor browser bundle that causes this. I believe it has more to do with either bots using tor to access google or google forcing captcha on access from known tor exit nodes. I have seen the same behaviour from vanilla firefox while trying to use google from tor. I would still rather input captcha than use bing!

a3n2• 12 years ago

Then stop using Google. Don't use their search, news or email, and don't be logged in to Google.

There are a handful of good alternatives. Use one. Use multiple.

Until people stop saying "yeah, but those other sites don't give me the same result as Google" then it's going to always be Google.

It's really up to you.

scrollaway• 12 years ago

(Disclaimer: I'm generally biased in favour of Google)

This logic absolutely doesn't work. The whole "vote with your wallet" attitude only works when a majority of consumers have a concrete choice in the matter. In the current case, even if the entire tech sector decided to "vote with their wallet" and stopped using Google, a phenomenally large amount of users would still be on Google due to partnerships with internet browsers, manufacturers, OEMs and so on.

You need to be honest with yourself. When you say "If you don't like x, don't use it", it doesn't give the person a vote/voice. It's to get them off your back (as if they were on your back in the first place).

_delirium• 12 years ago

There are a handful of good alternatives. Use one. Use multiple.

For search at least, unfortunately, the alternatives are pretty questionable. I've been using DuckDuckGo experimentally for a few months now, which is mostly Bing as the backend, with some DDG-specific add-ons and tweaks. I find myself having to use the !g command to rerun the search in Google somewhere around 40% of the time. Some of this is site owners causing the problem: a surprising number of sites block all crawlers in robots.txt, but then whitelist Googlebot and only Googlebot. But even leaving aside those sites, DDG seems to miss a lot of results, and the quality of the first-page results, at least for how I search, is consistently lower.

If you try searching in a language other than English the differences are even larger. I suspect that's because some of the infrastructure and datasets used for Google Translate are also used by search in some form, while Bing doesn't have a similarly solid multilingual stack.

I'd like to see more competition in search, but the barrier to entry to produce a good full-web search engine seems quite high.

joshu• 12 years ago

what does this have to do with the article?

thezach• 12 years ago

I reached out to Emily Linnert, one of the real journalist who's photo is being used on one of the sites and shes replied. Her (or her station) is taking legal action.

https://twitter.com/TheZach/status/554450915319492608

jmarbach• 12 years ago

I too have found some serious quality control issues with Google News. Some news sites are redirecting their mobile viewers (knowingly or not) to the app store instead of giving them the article that they clicked on. I wrote about this experience here: http://jmarbach.com/google-news-growth-hack-exposed

stevenh• 12 years ago

I noticed that one of the scam sites is currently listed for sale on Flippa. On their listing, they even openly boast that the best way to use the site is to use its Google News status to scam Google's search results:

https://flippa.com/3500230-google-news-approved-site-with-14...

Here's a mirror, in case it gets modified or deleted: https://archive.today/Jzubr

gnewssheriff• 12 years ago

Update: Site has been cataloged and added:

http://googlenewsspam.com/index_.php

Thanks for your help.

zaroth• 12 years ago

Where's Matt Cutts when you need him? [1]

[1] https://www.mattcutts.com/blog/on-leave/

Spam sites in Google News

Related Stories