Crawling the web at $2 per million pages

http://www.80legs.com/whatitis.html

by yarapavan • 17 years ago

107 43 17 years ago

ujjwalg• 17 years ago

This seems to be a very neat idea. How about processing of the crawled data, if someone wants to process the data and show it to the end user in a comprehensive way? We are looking for a solution which is affordable and provide both.

jhammerb• 17 years ago

Stuff the results of the crawl into Hadoop, write some MapReduce jobs to process the data, and then serve the structured data back up with HBase?

sachinag• 17 years ago

You know, I'd try this, but the hard part is that the processing is still on me (and I can't code, so I'm SOL). That, frankly, is the hard part, is it not?

tdavis• 17 years ago

You'd be surprised how difficult it is to create an extremely fast, scalable architecture for crawling web sites. Took me a few months, anyway (and would probably take a few more to make it "fill in a web form and go" usable). After that, the processing is pretty straight forward.

You've got a URL, headers, and body content. Just extract what you want and crunch.

jennwilson• 17 years ago

Hi, I work for 80legs and wanted to address this--we will be building an "App Store" over the next couple of months that will allow you to use Apps written by other developers to do some pretty cool stuff. We already have several of these 80Apps (as we call them) available, many of which have been written by the semantic search engine company swingly.com. You can view descriptions for them on the "Create Job" page if you sign up to use our Portal.

jdrock10• 17 years ago

While complex processing can be difficult, having a scalable infrastructure is also unbelievably hard. Sure, you can easily crawl from a single IP address, but if you want to do millions of pages in minutes, you would typically need several servers, IP addresses, and much more.

kbrower• 17 years ago

I would love it if it processed, stored, and indexed the data.

http://parselets.com/ + mysql + sphinx

lsb2• 17 years ago

Try Yahoo Pipes for things like that?

qeorge• 17 years ago

Wow - parselets.com is amazing. Thanks for sharing.

I think 80legs can do some level of parsing now, but Parsely seems like a great way to describe the analysis for targeted crawls on predictable data. If it supported the combination you've described it would really open up some cool possibilities.

jdrock10• 17 years ago

We are looking at storing data, doing pre-done crawls, offering data feeds, etc. It's something on the horizon for us.

Hopefully guys that write parselets will be interested in becoming developers for the upcoming 80legs App Store. They'll be able to sell their parselets and earn 100% recurring revenue from 80legs users using their parselets.

andreyf• 17 years ago

I remember they had an explanation about this, but I don't remember what it was, and it seems to have disappeared: what are these 50,000 computers they're using?

e1ven2• 17 years ago

Plura has a java applet which you can embed into a webpage that gets viewed for a long duration, such as a game or a streaming video. Affiliates embed an iframe which loads their applet as the user plays their game. The user's CPU goes up a bit, and they can help generate revenue for the game makers.

lsb2• 17 years ago

They're targeting desktop apps. The Java app downloads the pages, so it needs high permissions, so you get the default Java unfriendly popup asking you for confirmation.

mikeryan• 17 years ago

I'm actually surprised its not explained on the site (or if it is I can't find it).

If you're going to make a claim of 50,000 computers I'd want to see what you mean by that somewhere.

teeja• 17 years ago

Pricey!

I'd pimp for Open Google instead.

Flemlord• 17 years ago

I wonder how they arrived at their $2 per million pricing model. Why not $10 per million? Either one sounds ridiculously inexpensive.

charlesju• 17 years ago

They're probably using EC2s and then it's a function of how many pages can be scrapped per EC2 compute hour and then taking a premium on top of that.

The reason why they didn't want it to be too expensive (ie. 5 times as expensive) is (1) competitors can easily equilibrate and steal market share if their idea works b/c of the economic inefficiencies in their pricing model and (2) this game plan is more a game of dependence rather than up front profits, so it makes sense to take very little profit up front to get user traction.

jdrock10• 17 years ago

We're not using EC2. AWS doesn't and can't scale to our size, actually :)

Side note: AWS guys have asked if we use them. We said "No, we'd be losing money then and wouldn't be able to scale to our size with you."

fsniper3• 17 years ago

That reminds me about my fools day joke about MassiveClouding. I imagined about a company buying cpu/hours from regular desktop computer users and selling it to anyone interested in this kind of cpu power. Of course any computer software could be run on the MassiveClouding without any changes. My good imagination :) I don't know about the technology behind 80Legs but these 50000 computers might be a bot net.

Evgeny• 17 years ago

Isn't that like what the SETI project does? (Except that they don't buy the cpu power, they ask people to volunteer)

http://en.wikipedia.org/wiki/SETI#SETI.40home

"Any individual can become involved with SETI research by downloading the Berkeley Open Infrastructure for Network Computing (BOINC) software program, attaching to the SETI@home project, and allowing the program to run as a background process that uses idle computer power."

fsniper3• 17 years ago

Yes it's something like it. There is another base difference; SETI@home or other folding projects have a centralized infrastructure for the data to be collected and bound. And all the software to be run on the client side are written for a specific mission. So every software should be rewritten for these platforms. My MassiveClouding dream has a some way of dissecting any software into parts and run on the clients. That's why it's a dream.

skolor2• 17 years ago

There may be a market for this. I know several people who run various types of folding just to be able to brag about the performance they get, and love to show off their stats. Getting paid for it would just be icing on the cake.

Of course, the pricing structure would be difficult, as would setting up the infrastructure for it. My (uniformed, by-the-seat-of-my-pants) best guess would be that subverting the current folding method of jobs would be the best way to do it. Accept jobs, send out to client computers, clients get paid based off of how many they complete. The problem arises from: sandboxing the jobs, to make sure they can't steal client info, and making it relatively easy to program jobs, while still allowing users to do various things to increase their performance (like using GPUs and the like).

fsniper3• 17 years ago

I second you. But this is something needing years and great amount of money to be good enough for everyday use.

It may be a good start up project. But I'm not that determined to do this unfortunately.

sp332• 17 years ago

Sure, computing becomes a commodity and people / businesses will compete on "extras". Compute cycles and storage will be auctioned off in real-time depending on the buyers' need for and sellers' ability to provide data integrity, computational correctness guarantees, cycles/dollar, and a jurisdiction where the computation and data are legal.

keefe4• 17 years ago

They're inflating the cost difference. Cloud : $0.10 / CPU-hour for "large scale" crawling. There's no reason to use a small, unreserved EC2 instance for large scale anything? I have a small reserved instance I use for git and bugzilla and so forth, it's slow as hell. That's why there are bigger ones.

jdrock10• 17 years ago

The cost savings are still there with reserved instances if you do the math.

Actually, the cost is not the biggest issue with the cloud. If you're talking about large-scale crawling, AWS will not adequately scale. You can't get enough nodes or enough bandwidth.

keefe4• 17 years ago

My point was just in questioning a $4/million figure using a 0.10/hr instance. Why don't you run benchmarks against a reserved high CPU instance for 100M pages crawled? I was never contending that you cannot offer cost savings over custom crawling in ec2, just the way the #s on the page were calculated. If you have enough traffic to keep your instances saturated of course it is more economical to buy dedicated servers, but nobody starts out that way. Why in the world do you think you can't get enough nodes or bandwidth to do web crawling off of EC2? I remember seeing Bezos talk about animoto scaling up to 3500 instances instantly for video transcoding, I find it extremely hard to believe such a parallelizable task as web crawling could not be done on EC2. It's one thing to say that you can do it cheaper, it's another thing entirely to say that it can't be done on EC2 at all. How did you come to this conclusion?

anurag• 17 years ago

I tried 80legs a couple of weeks ago when it was still in beta. It works well if you're willing to write Java code to process HTTP responses (see http://80legs.pbworks.com/80Apps). You can issue new requests in your processing code and write processed data to files which you can download once the crawl finishes. Even without Java, you can perform basic keyword/regex matching on crawled data literally by filling out a web form. And they have an API of course. Very impressive.

However, if you want to run crawling on your own infrastructure, I recommend Scrapy (http://scrapy.org/), a python crawling framework introduced on HN last year. Scrapy solves some of the more time-consuming problems involved in writing a crawler from scratch (multiple simultaneous requests, pipelined processing, raw caching, duplicate URL filtering) and comes with nifty development and administration tools. More importantly, it has an active and helpful set of core developers and good documentation. I am comfortable with both Python and Java but I chose Scrapy over 80legs because I can crawl for free on the machines I already have and I can afford to spend more time crawling from a single IP compared to 80legs which will let me crawl much faster but isn't free. Also, with Scrapy my bot can be 'naughty' - 80legs jobs obey robots.txt and limit the crawling rate per domain.

When my crawling needs outgrow my infrastructure, I am going to look at 80legs again.

jdrock10• 17 years ago

Just a quick heads up - we'll be looking into allowing JRuby and Jython fairly soon for when you create 80Apps.

You'll also be able to mashup third-party apps and libraries into your own code. This is something we hope to have in 2-3 months.

coderdude• 17 years ago

How come there isn't a REST API? Java, JRuby, Jython. Ewwww. This isn't how outfits are doing things anymore...

mwexler• 17 years ago

One limitation is the throttling for each domain. If I had a smaller set of pages that I wanted to read frequently, this solution would not work. I understand the need to avoid DOS attacks, but in some cases, it would nice to be able to read a million pages from 3 large domains instead of 1 from a million pages.

bliving• 17 years ago

That and...

"Your parseLinks() and processDocument() methods must complete within a total 10 seconds per document processed"

... as a limit on processing leaves room for competition. One advantage of the BYO-Cloud solution is that you can pay for more intensive processing of the crawl.

tezza• 17 years ago

..:: Loop Detection?? ::..

Are you supposed to implement your own cycle detection?

If not, how deep is the cycle detection that 80legs offers?

There are plenty of HoneyTrap[1] OSS projects which will quickly rack up lots of $$$ if the 80legs spider is backlisted.

For those who don't know, these projects create deeply linked pages, and sometimes create infinite cycles. They are trying to hinder spammers, but may hinder 80legs too.

[1] http://www.davidnaylor.co.uk/stopping-bad-robots-with-honeyt...

jdrock10• 17 years ago

Because of the way 80legs handles crawls, users don't need to worry about loop detection. There are really two issues here:

1. For each user crawl, we only allow the same url once, so any simple loops that involve the same urls are eliminated by this process no matter how large the loop

2. For more sophisticated "spider traps" that work with different urls and domains, they can have a limited effect on your crawls. Because of our per-domain rate throttling, the worst these traps can do is add a few cents per day to someone's crawl.

yarapavan• 17 years ago

MIT Technology Review also has an article on 80legs. Accessible at http://www.technologyreview.com/web/23528/

Crawling the web at $2 per million pages

Related Stories