Jason Calacanis Knows He's Spamming Google, He Just Thinks It's No Big Deal

http://smackdown.blogsblogsblogs.com/2010/02/22/apparently-jason-calacanis-knows-hes-spamming-he-just-thinks-its-no-big-deal/

by mvandemar • 16 years ago

157 76 16 years ago

smackdown.blogsblogsblogs.com

dangrossman2• 16 years ago

While it's always a bit annoying to see a meme take up article slots on Hacker News regularly, I don't mind this one. I see sites like Mahalo as doing massive disservice to huge numbers of people --

* thousands of AdWords advertisers that have paid for their ads to be matched with content sites, not scraper pages with a huge ad to text ratio

* thousands of publishers whose work is being scraped, aggregated and outranked without as much as a backlink

* millions of web searchers that are hitting these pages instead of the real sources of the content they were searching for

And calling out companies that harm the fabric of the web for everyone else is worth doing.

aresant2• 16 years ago

Here's one thing to consider : Google's algorithms for detecting the actual value of clicks for advertisers has improved greatly over the past few years.

If Mahalo’s traffic was utter crap, it would be dropped.

mvandemar7• 16 years ago

"If Mahalo’s traffic was utter crap, it would be dropped."

That's not even close to true. If it were, there would be no AdSense for parked pages program. Their eye is still very much on their own bottom line, not the advertisers.

dangrossman2• 16 years ago

My conversion rates from the content network have never been as high as search. I try new campaigns targeting both networks every once in a while but always end up either opting out or setting a much lower bid on the content side.

mwcremer• 16 years ago

[citation needed]

qeorge• 16 years ago

Its really a shame how low Google's standards are for AdSense. I've tried advertising in the content network several times, but each time I end up wasting so much time blocking MFA sites that I give up.

wheaties• 16 years ago

The rub is, if Google were to block all of Mahalo, a lawsuit would surely be filed.

sjsivak• 16 years ago

I do not think anyone is asking for all of Mahalo to be blocked. The major issue is the insane amount of auto-generated pages that are not given a noindex call and are also added to the site map xml file.

These actions appear to be conscious choices to make sure that google will index these pages, which according to several sources, have nothing but "scraped" content and are therefore spam.

jonknee• 16 years ago

And Mahalo would lose. Google penalizes SEO spammers hundreds of times every day. It's very easy to see that Mahalo does not follow the rules.

seldo• 16 years ago

Google is providing a free service in a field in which there is intense competition. I don't see what basis anyone would have for filing a lawsuit for not being indexed in Google.

w00pla2• 16 years ago

The question is: why doesn't google do anything about this? There are countless such pages. Why not just blacklist domains? Or artificially give them lower pagerank?

Or is it okay as long as they get ad-sense money?

vaksel2• 16 years ago

they do it all the time, as long as it's a small site, Mahalo's existence basically flies in the face of all of Google's rules and regulations

noodle• 16 years ago

they do block domains, at least from their organic search engine results.

i accidentally got one of mine blocked when playing around with some random software that produced a site that was somehow against their TOS. i didn't have it live long, but long enough for the spiders to grab and flag. still waiting on a reconsideration request.

why not mahalo? who knows. if i had to venture a guess, i'd say that the larger the site and the more revenue google sees from that site due to adwords and the like, the higher up the chain the decision to block it has to come from.

JulianMorrison• 16 years ago

I think Google prefers to programmatically detect and blacklist behaviours, without too many innocent casualties - even if working out how to do that takes longer. They aren't interested in playing whack-a-mole.

dlib• 16 years ago

Isn't there a search engine out there that takes into account the advertisements vs. content ratio? It would be trivial to get rid of Made for Adsense sites by scanning for the Adsense code used and seeing how often it's used on a page and the percentage of the page it takes up.

moultano3• 16 years ago

I work for Google, and while I don't know anything about this particular situation, one invariant that I've been consistently impressed with is that the ads side of the company doesn't influence the search side of the company _at all_. I've seen well meaning ads support folks emailing people in search quality about an innocent problem with one of their customer's sites, and getting sternly told that they should never again send an email to those mailing lists.

danik• 16 years ago

Although I understand your points the AdWords and search issues is googles problem. People pay for a product with flaws in it and google is selling, not Mahalo. Mahalo has the freedom to do whatever they want with their pages and they can't be held responsible for what google in it's turn does with them.

Now, stealing other peoples work, that is obviously Mahalos doing and should be a case for the court.

I'm not really sure what the point with this article is. Mahalo spams google - so what? It's up to them to do this and up to google to prevent it. Business is business, even on the internet.

axod2• 16 years ago

>> "Currently when I look, Google tells me that Mahalo has 356,000 pages indexed"

I see 'Results 1 - 10 of about 2,200,000 from mahalo.com'

Have things stepped up a gear or am I misunderstanding?

mvandemar7• 16 years ago

No, different datacenters will show different results... sometimes very different. It also matters if you are visiting Google.com or one of the country variants.

According to what Jason said in another comment, however, all of their pages are listed in their xml sitemap, and all of those are listed in a master xml sitemap index located here (warning! huge files if you follow the links in the first one!):

http://www.mahalo.com/sitemapindex.xml

Based on what I saw 2 million+ looks like a huge overestimate, if what Jason said is true.

Edit: My bad, frederickcook's answer was the right one. I didn't realize you were doing a regular text search.

chronomex• 16 years ago

There are 12 sitemap-mahalo.xml files, each with 50,000 urls in it. The 12th has 48,661 urls by my count. That gives a total size of 598,661 pages.

Methodology:

  wget http://www.mahalo.com/sitemapindex.xml
  cat sitemapindex.xml |sed -e's/[<>]/\n/g'|grep ^http|xargs wget
  for P in sitemap-mahalo.xml* ; do echo $P ; cat $P |sed -e's/[<>]/\n/g'|grep ^http|wc -l ; done

Edit: Formatting

frederickcook• 16 years ago

Search "site:mahalo.com"

Simply "mahalo.com" lists every indexed page with that text on it, such as this one.

axod2• 16 years ago

Yup that's what I did. Maybe the particular google datacenter I hit had a high estimate there.

It also says 59,300,000 for scribd.com which I can't help but draw comparisons with this.

pclark2• 16 years ago

a year on, maybe not so crazy now eh? http://news.ycombinator.com/item?id=485618

mvandemar7• 16 years ago

Heh, got any stock picks you want to share? :)

Nice call.

pclark2• 16 years ago

someones network of contacts > everything

CoachRufus87• 16 years ago

this is getting very, very old. can we move on? please??

prawn2• 16 years ago

It's obviously of interest and importance to a number of people involved in this field or troubled by poor quality material showing up in Google. If you're not one of those people, it's pretty easy to identify these links and not upvote them or visit the articles/comments, etc.

There are countless articles on HN that I have no interest in (e.g., I don't even know what Clojure is), but I just don't click through to them.

moe• 16 years ago

One of the things I really miss in google is a persistent blocking preference a.k.a. site blacklist. Mahalo would go straight in there, along with expert-sexchange, sedo parking and a few others.

ryoshu• 16 years ago

Depending on the browser you use, this is easy to do. I have experts-exchange and other useless sites blocked using greasemonkey/greasemetal.

mvandemar7• 16 years ago

Yeah, sedo parking is a bitch tho, since by it's very nature it's tons of different domains. You would need a way to block by app.

jachee• 16 years ago

SEO is snake oil. This is further proof that the whole "industry" is ruining the integrity and usefulness of the internet.

lmkg• 16 years ago

Not everything that uses the phrase "SEO" should be painted with the same brush. A lot of SEO best practices are making the HTML markup more semantic and human-readable. White-hat SEO dovetails pretty well with a human-readable web and accessibility standards, and is one of the better business cases for kicking the Flash habit. Just because Calacanis makes his name by abusing loopholes in the algorithms doesn't mean that's the only sort of thing that the name "SEO" applies to.

aaronwall5• 16 years ago

That is the angle Jason used when he created his steaming pile. It doesn't mean that anyone in the industry agrees with Jason's strategy.

And if you want to place the blame where it belongs remember that Google is the company funding all this content scraping with their ads programs.

I just tried searching for a recent post from an official Google blog (about AdSense using referral data for more relevant ad targeting) and found a scraper site with their ads outranking them for their own content. Pretty sad.

olefoo• 16 years ago

Actually, pretty savvy.

Google gets paid for the scraped site because of ad impressions and possible clicks.

Google does not get paid for running an informative blog, in fact it's a cost to them.

Google cares about getting paid.

jsz0• 16 years ago

It's really too bad Google doesn't allow you to simply blacklist domains from your search results permanently. The thing that frustrates me the most about these SPAM sites is the fact they're constantly popping up and my down voting of the result seems to do absolutely nothing unless I'm using the exact same search query. Just let me blacklist Mahalo and other sites like it permanently. Better yet make it possible to subscribe to a blocklist so the community can pool it's resources and fight back.

Devilboy• 16 years ago

I request this from Google every couple of months. If I could remove an entire domain from all my personalized search results, I'd be soooo happy.

benatkin2• 16 years ago

I've requested this, too. I even switched to Yahoo! for a few hours because they let me blacklist expertsexchange.com (since changed to redirect to experts-exchange.com). I don't see the option on Yahoo!'s page anymore.

aaronwall5• 16 years ago

With Google custom search engine you can create your own personalized search service...built off the back of Google, but while delisting any sites you do not want results from.

And it only takes about 30 seconds to set one up. Then you embed the search box wherever you like.

http://www.google.com/cse/

jasonmcalacanis3• 16 years ago

this is getting really old and we're not interested in doing anything black hat or even gray hat. as such we're doing the following:

1. we're removing (or building out) any page in our system created by our users with under 200 words of original content. This will take a couple of weeks but it's tarted.

2. we're not letting users create stub pages (short pages) until we can noindex them and put them in a different directory (i.e. /stubs/) so google can easily tell the difference between them.

these pages are < 1% of our revenue and low single digits of our traffic. we don't benefit from them materially, and I think we're being targeted by Aaron Wall and other SEOs for my "seo is bullshit" comment from 2005 or so.

I guess that is fine... I gotta live with the ramifications of what I say. however, for the record I don't believe that SEO is BS any more... when i said that it was when we were building joystiq and autoblog and we spent zero time on SEO.

All that being said, we're being targeted by a small group of folks who want to take us down. we're only going to get stronger from this because our hundreds of contributors are rallying around building out the short pages.

Topix, Kosmix, NYTimes and Zimbio are all making quality topic pages and are not getting attacked over it. not sure why there is some double standard.

regardless.... this is not a material thing for us. we're flushing all these pages and moving them to a different directory going forward so that search engines know where they are located (i.e. /stubs/ ).

thanks for the ass kicking.... having a horrible day today over this.

jcal

http://bit.ly/jasondown

tdm911• 16 years ago

we're removing (or building out) any page in our system created by our users with under 200 words of original content. This will take a couple of weeks but it's tarted.

i'm also getting a list of every page under 300 words and having the page managers build them out in 30 days or deleting them.

from: http://news.ycombinator.com/item?id=1143512

is it under 200 words or under 300? are the goal posts moving already?

jasonmcalacanis3• 16 years ago

300 is the goal right now.... we're making plans right now.

.... keep the attacks up, only makes us more motivated.

mvandemar7• 16 years ago

Jason, did you even read my article? This isn't about the traffic those autogenerated pages get, it's about the fact that through the minuscule amounts of PageRank that they are each capable of grabbing, you are now able to rank your mediocre pages with absolutely zero influence from the rest of the web.

We're not talking about stub pages, it's all the fully automated bullshit that you are generating. They not only need to be deindexed, they need to be nofollowed or removed altogether.

How is it you are out there playing the wounded puppy when apparently you haven't even read the articles or followed the reference links? You can't just skim this one and then craft a rebuttal and think you've addressed the issue. There's a lot of data in those paragraphs you apparently just skimmed over (if that even).

You have over 500,000 pages listed in your xml sitemap, and Google appears to have over 330,000 of them indexed. Click on this link, please, and actually go look at 10-12 of the pages we are talking about here:

http://tinyurl.com/yzmxq7b

Tell me how long it takes you, just by clicking through, to find even 3 pages that have any human interaction in them whatsoever.

Maybe, just maybe, you really don't have a clue what is happening. I personally don't believe that's the case, but if so then whoever it is you have working for you that set this up knows how to spam like a pro.

chintan• 16 years ago

Try site:kosmix.com or site:righthealth.com

Kosmix always had noindex in their "search results" - Now stop whining like a baby and get your ass back to work instead of justifying your mistakes.

Joepuf• 16 years ago

Jason - the "seo is bs" line was heard at every seo conference you headlined when you launched mahalo.

fjabre• 16 years ago

Think this thread has gotten way out of hand.

A lot of the comments in here seem like more of a personal attack than anything else. You might as well change the title of this post to "Jason Calacanis ruined the Internet" or something to that effect..

Can we stop the drama already? I think we're going to need a hose to control this mob.

japherwocky• 16 years ago

Jason Calacanis is the Paris Hilton of the web.

Mahalo is not particularly interesting, not particularly evil, he doesn't really do anything, and yet - we keep talking about him and putting his shit on the front page of hacker news.

It's a mediocre aggregator/linkfarm, with some mechanical turk style incentives for humans to contribute, and a nice chunk of $ in the bank. Just ignore/mock him for another year or two until the funding runs dry.

mvandemar7• 16 years ago

You are absolutely right, it is mediocre crap. The issue is that through the spam techniques described in the article, he can now gain undeserved top 10 rankings for many phrases using pages that have no business being there.

If you are not part of the web development community then these discussions will most likely bore the hell out of you. However, if you are, and if you are aware of how many innocent sites Google bans or penalizes on a daily basis, or AdSense accounts that get canceled with no appeal for offenses much less than his, then this stuff actually matters.

Jason Calacanis Knows He's Spamming Google, He Just Thinks It's No Big Deal

Related Stories