Duck Duck Go Architecture

http://www.gabrielweinberg.com/blog/2009/03/duck-duck-go-architecture.html

by epi0Bauqu • 17 years ago

72 19 17 years ago

jwilliams2• 17 years ago

Cool article. Couple of queries :-)

It mentions that static content is served from nginx, but then mentions S3 later on as well. Why not just Amazon for all the static content?

Mentions nginx being more stable than Apache - you mean no-crash stable, or more consistent under load?

epi0Bauqu8• 17 years ago

Thanks! I use S3 to serve images that are loaded after the initial page load (incidentally, using YUI's ImageLoader: http://developer.yahoo.com/yui/imageloader/). I use nginx to serve static files, e.g. HTML, as well as images that get loaded on page load, e.g. logo. Why not S3 for the latter? Too slow. Amazon's (relatively) new content network may be faster, though I haven't tried it yet.

Wrt nginx being more stable, both. I've crashed Apache as well as pseudo-crashed it (process still running, but not serving right). nginx hasn't crashed yet. It's also more consistent (and faster) under load. However, in Apache's defense, it does a lot more, e.g. mod_perl vs nginx proxy to FastCGI.

jwilliams2• 17 years ago

Cool - well I'm on Apache now, on a very undersized server - so I guess I'll have to give nginx a shot.

Not sure how you'd extract more speed from your current setup. The natural next step would be doing work in parallel - this something you do/have considered?

If you do try the Amazon CDN, let us know how you go :)

almost2• 17 years ago

I've seen Duck Duck Go when you posted before and didn't think much of the idea (hey, I can use Wikipedia!)

Just had a look again and I now think it's pretty awesome. Just for a really speedy "what the hell is this?" type of query it looks absolutely amazing. I love the AJAX extra search links, makes it all feel really snappy.

Works great on the iPhone as well, I've added an icon to my home screen. You might want to consider making an app for the app store (you could pretty much just make an app that loads the site as it is).

Have you considered adding an API? I currently have my Emacs set up to do a dictionary lookup (using the dictd protocol) on a word when I hit C-cl. I'd love to add Duck Duck Go results to the buffer this displays!

epi0Bauqu8• 17 years ago

Thanks! Yeah, we've gotten a lot better since then. On the zero-click info (what is x?) we use lots of sources now beyond Wikipedia (e.g. Crunchbase) as well as have a bunch of new features (http://duckduckgo.com/about.html). We also have both an iPhone app and an API :)

iPhone App: http://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftwa...

API info: http://duckduckgo.com/api.html

almost2• 17 years ago

Whoops, should have spotted both of those :p

Both look good though!

strlen• 17 years ago

Question: why are you going with FastCGI over mod_perl? Issues with mod_perl debugging/memory leaks? Desire to use nginx over Apache for higher throughput? The big advantage -- to me -- of mod_perl is the power of having your code live within Apache's process space.

I wonder if mod_perl works with event_mpm, yet.

epi0Bauqu8• 17 years ago

d) All of the above.

Initially I didn't think anything could beat mod_perl speed, but after benchmarking, this setup is just as fast. It's also more stable, and uses less memory. And yes, also it is easier to debug and I was getting memory leaks on mod_perl.

Don't get me wrong, I love mod_perl. I used it at my last company and on many other projects before this one. And I still use it for other stuff.

Last I looked I think you can run mod_perl with event_mpm but I didn't like this: "The current Apache::DBI should be usable under threaded mpm, though it doesn't share connections across threads. Each Perl interpreter has its own cache, just like in the prefork mpm." (http://perl.apache.org/docs/2.0/user/performance/mpm.html)

globalrev• 17 years ago

do you make money?

epi0Bauqu8• 17 years ago

No ads on the site yet, but in general, search ads make money.

njharman• 17 years ago

I'm not exactly a search connoisseur or even pay much attention to startups but wow. This is one of the first search innovations I've found that was any good, in fact it's great.

I've just replaced my google FF keyword, 'g', with duckduckgo link. I'll try it out for a week or so and see if I still think it's as awesome as I do now.

epi0Bauqu8• 17 years ago

Thanks! Please email me your thoughts after the week--always looking for real feedback from real users. Also note we have a FF extension: https://addons.mozilla.org/en-US/firefox/addon/10536

brandnewlow• 17 years ago

This passed my "does this not suck?" test. Good work. A search engine that does not suck and is not named Google.

kingkongrevenge• 17 years ago

gigablast finds stuff for me that others miss.

tdonia• 17 years ago

complete side note - but your favicon has a lot of aliasing when it's on a dark background (i.e. as it is when you have ddg in an inactive chrome tab). the circle looks nice on white, but you may want to adjust it so it reads a little better against other mattes.

aside from that tiny-minor issue, ddg seems cool & the overview of the architecture is very clear. i'm going to give it a try as my default chrome search engine & figure out just how behaviorally indoctrinated google has me and how your architecture holds up as more people use the service.

epi0Bauqu8• 17 years ago

Thanks! I will look into the favicon issue. Please send me an email after you try it for a while with your thoughts.

rozim• 17 years ago

I must have missed something - what technologies are behind the actual search? Lucene/Nutch/Xapian?? thx, Dave

epi0Bauqu8• 17 years ago

All proprietary Perl.

Duck Duck Go Architecture

Related Stories