Duck Duck Go written in Perl

by fogus • 16 years ago

77 36 16 years ago

draegtun7• 16 years ago

And here are the HN comments on that Duck Duck Go architecture post: http://news.ycombinator.com/item?id=525048

jackowayed• 16 years ago

The perl code isn't a full search engine though. He uses Yahoo! Build Your Own Search Service, so that perl code is just taking the results from there and throwing results out, reordering results, getting the zero-click info, etc. Yahoo does all the crawling.

sp332• 16 years ago

Searching is not the same as indexing. DDG will give different results from Bing search, even though they use the same index.

Edit: DDG does some crawling in addition to using the Bing index.

Keyframe• 16 years ago

I wonder if DDG could benefit from http://www.80legs.com/ ?

epi0Bauqu• 16 years ago

I do my own crawling & indexing in addition to using third-party APIs. It's a hybrid approach where I focus on the places I think I add the most value.

zaphar2• 16 years ago

That's cool. I've used DDG a few times in the past and generally like so far. I had no idea it was done in perl though. It's nice to see new perl apps getting some press.

I wonder if they use any of the newer CPAN libs like Moose and family in the architecture?

draegtun7• 16 years ago

It's nice to see new perl apps getting some press

Some Perl press as appeared on HN before. For eg: http://news.ycombinator.com/item?id=565152 (though sometimes the Perl community have done their best to keep head low on this one :)

mhd3• 16 years ago

I think that's a natural combination. Both p-words showed me that there's more than one way to do things.

sdesol3• 16 years ago

As for Moose, he (DDG author) answered this here

http://www.reddit.com/r/IAmA/comments/bbqw7/i_am_the_founder...

Having written an indexer in Perl for my current startup product, I really can't see the need for using something like Moose. I'm biased though, as I'm not a big fan of OO programming. I gave it a try in Perl years ago but it really sucked. OO programming that is, but I guess Moose was designed to fix this, but it's a little too late for me. I was taught C in school and I've learned to live without OO.

zaphar2• 16 years ago

I can't say I disagree with him on the dislike for OO. However if you are going to do OO in perl Moose is the way to do it.

draegtun7• 16 years ago

I really can't see the need for using something like Moose

Well Moose is certainly getting traction in some big companies like BBC, Cisco, Hearst, Symantec and Yahoo.

ref: http://moose.perl.org/about.html

mkramlich• 16 years ago

i had pictured something elegant like Lisp. but no it was just hacked together in Perl? Somebody needs to link to the famous XKCD strip. :)

piaresquared• 16 years ago

You mean this one? :)

http://xkcd.com/224/

s-phi-nl• 16 years ago

On a similar topic, A God's Lament http://xkcd.com/312/

alexyim• 16 years ago

Interestingly, a search for "ostensibly xkcd" shows the comic in the first link:

https://duckduckgo.com/?q=ostensibly+xkcd&v=

gojomo• 16 years ago

From some early discussion (and their current job listing), search engine startup Blekko also has a significant Perl component:

http://blekko.com/jobs.html

jsrfded• 16 years ago

Yes, blekko has a full web crawler, indexer, and some of the query processing code in perl. Some of the innards and query execution are in C++ for speed.

Generally we tried to delay the conversion of a component to C++ as late as possible, since it makes it so much harder to change and iterate on. We call it "pouring cement on the code". For some things it's absolutely necessary to drop to C for speed, others it doesn't make a lot of difference since you're bound by other constraints like I/O.

draegtun7• 16 years ago

Probably the biggest profile Perl (and Catalyst) public app in last few years would be the BBC iPlayer: http://www.bbc.co.uk/blogs/bbcinternet/2008/12/iplayer_day_p...

dragonquest• 16 years ago

Actually, YouPorn is powered by Perl and Catalyst and regularly makes the top 25 domains. Not sure if it beats the BBC main site though.

deno• 16 years ago

How can he use Bing as a backend? I read their API's terms of service and it forbides you from changing search results in any way.

“(c) modify, filter, obscure, or replace the text, images, or other content of Bing results, including by changing the order in which Bing results appear (but this limitation will not apply to Bing results of type "Web"), intermixing Bing results with search results from other sources, or intermixing with Bing results any other content so that the other content appears to be part of Bing results;”

I was quite interested in using Bing API myself but that does seem too restrictive. I hope you can prove me wrong!

gorm• 16 years ago

> but this limitation will not apply to Bing results of type "Web")

Isn't this what he is using?

irq11• 16 years ago

That parenthetical only applies to the clause about result ordering. if you read the whole TOS, it's quite clear that Bing doesn't want you to do what DDG appears to be doing. In particular, mixing results from different services is a no-no in most search APIs.

sandee• 16 years ago

Thanks. Good Post. This is great example that good application and complex systems can be written in perl as well.

On a side note, i saw the previous post on DDG, where discussion was about how DDG does not store any private information. It makes sense in the context, that since he leverages upon other search engines , his cost of running is low. So he can afford to ignore user information otherwise used for commercial purpose by others.

loewenskind• 16 years ago

>This is great example that good application and complex systems can be written in perl as well.

You could write "good application and complex systems" in machine language if you work hard enough. What are you trying to say?

Duck Duck Go written in Perl

Related Stories