A Duck & a Wiki Team Up Against the Content Farms

http://www.readwriteweb.com/archives/a_duck_a_wiki_team_up_against_the_content_farms.php?sms_ss=hackernews&at_xt=4d4dea696c2807b0%2C0

by JackHerrick • 15 years ago

57 65 15 years ago

readwriteweb.com

estel3• 15 years ago

Sometimes, whilst DDG's partnerships can be great for getting quality content to the top of search results; it's definitely not the sort of thing I'd want to see a dominant search engine doing. DDG obviously needs to do things differently in a way that can drive them traffic, but if Google were to do this, they'd quite rightly be blasted because such partnerships could massively impact the ability of other site's to grow organically without the engine's blessing.

melvinram2• 15 years ago

I agree. Google's prominent ranking of Wikipedia articles usually drives me nuts unless it's the only good article around.

redthrowaway2• 15 years ago

Google's ranking of Wikipedia articles is due in large part to Wikipedia's great PageRank ranking. I don't think Google is giving Wikipedia any sort of singled-out treatment, Wikipedia just ranks highly on Google's relevance metrics.

ddemchuk3• 15 years ago

google's not going to do anything because they're in the business of making money off of search and duckduckgo is not

estel3• 15 years ago

Then what is their business?

JackHerrick• 15 years ago

I actually have a lot of faith in Google to do the right thing for their users. After all they mostly walked away, or at least severely hampered their business in China. They will be more than willing to walk away from sites their users think are low quality.

I'm not saying that this means they will do the exact same thing DDG did. I'm just saying any decisions they make will be primarily based on looking out for their users best interest rather than short term adsense revenue.

ddemchuk3• 15 years ago

what about your statement is supported by their actual actions in the recent years? who are their actual customers, searchers or adwords bidders? They simply need to provide a decent enough search experience to keep users coming back so they can keep adwords bidders participating. It's their entire business model.

redthrowaway2• 15 years ago

In what way is ddg not in the business of making money?

sorbus• 15 years ago

Do you happen to know how DDG is making money, then?

ddemchuk3• 15 years ago

wait, what? Am I wrong in saying that google is not in the business of making money? They're making a damn killing off of search traffic.

Duckduckgo is not. They're getting press for being defiant.

estel3• 15 years ago

No, but you're implying that DDG either has no plans to make money (which is possible, if rather a short-lived plan) or will remain small enough to be reliant entirely on Gabriel's day job.

DDG is a search engine that's using its search to make money. Just like Google.

petervandijck3• 15 years ago

Google has mentioned they're working on the content farm problem. I would expect them to make some moves faster that are smarter than banning content farm X while partnering with content farm Y. Likely algorithmically, coz that's how they roll.

timr• 15 years ago

Google would do well for itself to stop horsing around with attempts to find 'algorithmic solutions' (if that's actually what they're doing), and in the short-term doing what's necessary to make a perceivable improvement in the quality of their product. Even if the content farms were changing domains regularly, Google could probably kill them faster than they'll find a way to algorithmically drop the ranking of their results, in aggregate.

For all of the recent noise that Google has been making about fixing this problem, the number of gamed search results in their index is still insanely high. Take plant diseases for example...every time I've made a search for some specific problem related to my houseplants, the results are so loaded with content-farm crap that I have to go at least a page or two in to get to reliable sources of information.

We're not even talking about relevant webspam from plant stores or online merchants...just eHow garbage, which is poorly regurgitated from the other sources that come lower in the results.

petervandijck3• 15 years ago

Yea, I think you're right there.

Although, perhaps we underestimate the scale they deal with. Plus, manually removing sites that are not clearly evil but on the border may become a PR nightmare.

epi0Bauqu9• 15 years ago

For the record, the vast majority of our content farm stuff is algorithmically based. In addition, we've made some manual blocks and since the beginning we've been promoting good content in our "Zero-click Info" boxes. Recently it has been a lot of programming stuff, e.g. http://duckduckgo.com/?q=python+split

netdog• 15 years ago

> Google has mentioned they're working on the content farm problem.

The irony is that is Google's own AdSense program is the incentive for the creation of airburger web sites, which otherwise have no reason to exist.

Vivtek• 15 years ago

How odd that even a year ago it seemed that the search engine space was closed to new entries, even solved - and now it feels wide open again.

citricsquid4• 15 years ago

I don't think so, the people using DDG are still a tiny tiny minority. If they did what google did with the browser market when they released Chrome then you would be warranted in saying they blew it wide open, but for what DDG has done with search? I don't think so, not yet anyway. Maybe when they've penetrated beyond the tech "elite" they might get somewhere, but right now if I ask any of my friends (who are relatively in the know about technology) they haven't heard of DDG.

Personally I don't think DDG will show wide success because the majority of people don't care about the added extras it provides, it's just other search engines (I think their major source is the bing api? Not 100% sure [1]) data with pretty labels and positioning, which very few care for. I search a relatively large amount and what DDG adds doesn't make it worth switching for me because I've become accustomed to how Google displays search results and adapted my behaviour to that.

[1] http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go-arc... (potentially no longer accurate which is why I'm not sure)

benologist5• 15 years ago

I think the difference is the attitude is different now - when Cuil launched the world laughed and said nothing could be as good as Google. Almost 3 years later a small, insignificant yet maybe not, number of people are starting to use alternatives.

A good parallel could be Firefox - Internet Explorer was the way to browse the internet, now they're just a way.

If Google wants to be the search engine a decade from now they may have to earn that right all over again.

CurrentB• 15 years ago

DDG has a laughably small market share. This doesn't necessarily make them a non-threat, but it's going to take more than this to change users' search behavior (ie convert them from Google to DDG) on even a minute level. I mean Google has the whole page preview thing now anyway, and the content farm thing should be fixed soon, as mentioned. Honestly, Google Instant is the best thing in search for a long time and it's so ingrained in my expectations now that I couldn't see switching to a search engine without it.

So asking what Google will do isn't really an important question.

babeKnuth4• 15 years ago

their market share may be laughable, but that still doesn't eliminate the fact that many knowledgeable people prefer either blekko or duckduckgo over google.

it can definitely be viewed as a "calm before the storm". or if you're familiar with gentrification, the "cool cats" are living on this side of town.

chc3• 15 years ago

Yes, many knowledgeable people prefer Blekko or DuckDuckGo. But that's "many" in the absolute sense, where "many" people are part of any arbitrary group on a planet of 7 billion.

Relative to the size of the group we're talking about, even among geeks, using DuckDuckGo as your main search engine is rare. It certainly doesn't look like DuckDuckGo has a glut of tech industry leaders using it exclusively — mostly just the odd geek here and there who wants to try something new or who has just decided he doesn't like Google.

tectonic3• 15 years ago

Is DDG one person?

citricsquid4• 15 years ago

Yeah, Gabriel Weinberg: http://www.gabrielweinberg.com/

jimboyoungblood• 15 years ago

Don't forget Eli!

tectonic3• 15 years ago

While I'm kind of frustrated by the anti-Google marketing, I'm impressed by how productive Gabriel clearly is.

bryanh2• 15 years ago

DDG is supplanting labor intensive algorithmic improvements with manually manicured search results. Compared to Google's vast network of resources, it is a logical step for DDG to take. It is, essentially, a fast fix to a real problem.

Google has to do things the hard (and dare I say... proper?) way.

InfinityX0• 15 years ago

DDG is supplanting labor intensive algorithmic improvements with manually manicured search results.

This sentence is a paradox.

mkramlich• 15 years ago

To illustrate what he meant, I'll give an example. Assume someone entered a search, "What are the best dog breeds?". Would it be easier to:

1. write an automated system which could dynamically deduce what the answer to this question is and/or what pages to return in the results, based on dynamically crawling the web, indexing, analying link structure, ranking, filtering, etc.

-or-

2. have a human write an answer to this fricking question and enter it into the database as the result for this search

hint: pick 2

it doesn't scale up to handle bazillions of different searches, but on a smaller scale and in tactical cases it's far simpler and easier to implement and can produce better results.

bryanh2• 15 years ago

Of course by manicuring search results I mean blacklisting domains.

zone4112• 15 years ago

I'm sorry but I have big problems with calling DDG a real search engine. Their page states "DuckDuckGo is a search engine like Google." BS. It's a meta search engine that relies on other real search engines, such as Bing, to get the results. If ever becomes a threat to Bing, they'll cut them off in a second. DDG doesn't do the hard and resource-intensive work of crawling and ranking the pages, they just tweak the results of others in ways that gets them publicity on tech blogs.

epi0Bauqu9• 15 years ago

For the record, I've been crawling the Web since the beginning of DDG (and before the BOSS or the Bing APIs existed). Yes, a one FTE, self-funded search engine, does not have the capacity to spend millions of dollars on crawling (does Google spend billions?).

I now focus my crawling efforts where I see they can add significant value, namely on spam removal and zero-click info. Our index actually shows the top result about half the time, which is not from any external API. Given that most people click the top result most of the time, this is not an insignificant addition.

We of course use external sources, but I think your comment is too dismissive of the intelligence we've woven on top and through them. I'm not interested in sharing all of DDG's trade secrets just to prove a point, but needless to say, you can simply compare the external APIs to our results across a swath of different types of queries and see the differences.

Finally, I find the premise of your comment a bit short-sighted, but I'm actually OK with being dismissed as a toy. In the eyes of the average user, they don't care where results came from at all. They just want the right information faster. So quite frankly, I think it is a reasonable move to use external APIs and focus on things like the top result, UI, etc. More recently, Yahoo and Ask seem to agree. Most people are dismissive of them too, but they've been doing a lot of innovative UI things, which matter a lot to end users.

petervandijck3• 15 years ago

Being dismissed as a "toy" is probably the best thing that could happen to DDG right now, and for the foreseeable future. They're searching for an angle against Google, a way in. They may well find one (although it's a tough battle).

greenyoda• 15 years ago

I don't really care whether they're a "real search engine" or not. In my experience, they give better results than Google (less spammy content-farm stuff), and they disclose less of my personal information, so I use them instead of Google.

babeKnuth4• 15 years ago

amen brother.

brianobush• 15 years ago

DDG is a meta search engine now, but this is a step towards being a real search engine someday. Harvesting queries, click through analysis, etc is vital to the development of a real SE. I would call this method SE boot-strapping.

rmundo• 15 years ago

So we have the website equivalent of spam. A simple solution would be to have a secondary page labeled "We think this is spam" where all the banned links would reside. Anyone who needs to see what's there could always dig in.

A Duck & a Wiki Team Up Against the Content Farms

Related Stories