PiCrawler: A distributed web crawler using PiCloud

https://github.com/studio-ousia/picrawler

by ikuyamada • 13 years ago

24 5 13 years ago

github.com

Fomite2• 13 years ago

I'm rather fond of PiCloud as a platform - it's good to see example projects out there using it.

ikuyamada• 13 years ago

I am also a fan of using PiCloud as a platform.

Using PiCloud for implementing crawler requires some hacks, so I decided to create this as a separate package.

brianobush• 13 years ago

What are the advantages over Rackspace, Amazon, Digital Ocean? Seems they are targeting science and big data apps.

spartango• 13 years ago

PiCloud is a layer of abstraction above the IaaS providers you've listed. That is to say that it gives you software infrastructure to design workflows and distribute them across the machines provided by an IaaS provider. In some ways it's like Heroku, and in others it's another layer beyond Heroku (App Engine-like?). You architect your work around the PiCloud tools. I think PiCloud currently runs on AWS.

Fomite2• 13 years ago

For me? Well, I'm a computational epidemiologist, so targeting science is exactly it's advantage. PiCloud allows the use of AWS resources fairly seamlessly from inside scripts, iPython notebooks, etc.

They're also very responsive to questions, and their package is far, far easier to use for the things I use it for than trying to build something myself via AWS, Rackspace, etc.

Disclosure: They also gave me a $500 research grant.

PiCrawler: A distributed web crawler using PiCloud

Related Stories