Show HN: A Python Spider System with Web UI

https://github.com/binux/pyspider

by binux • 12 years ago

159 40 12 years ago

github.com

bowlofstew• 12 years ago

That is a nice tool....nice work!

meowface• 12 years ago

This looks really nice. The API seems more user-friendly than scrapy's.

Immortalin• 12 years ago

Any plans for a gui based web scraper interface similar to portia?

binux13• 12 years ago

Currently, yes and no.

pyspider is running original python code, something like portia is a code generator (Apologize if I'm wrong, I have not use it). So it can been made as another WebUI module.

But for flexible, I have no idea how to make it right currently. So, We have a css selector helper, but no plan for a complete tool.

prht• 12 years ago

I am not trying to offend you, but I really don't understand when someone says "yes and no". I hear it more and more these days. Is this becoming a cliche? It can be "yes" or "no", not both together. "yes and no" is "no" for me.

zbb• 12 years ago

Take a look at source code. The package hirarchy is not pythonic (use "libs" as top package is not a good idea).

binux13• 12 years ago

agree

redacted• 12 years ago

What is the recommended way? (Serious question, I have larger projects that I would someday like to refactor into proper packages)

paulhauggis• 12 years ago

Why isn't it a good idea? I have plenty of projects setup this way and it works well.

It looks pretty well organized to me.

vertex-four2• 12 years ago

The issue is that if you have two packages installed, and both use "libs" as their top-level package, they'll collide. Use "projectname.common" instead.

huskyr• 12 years ago

Ah damned. the package hierarchy is not pythonic. That renders all the functionality of this package completely unusable.

Come on people, don't be like this. It takes 5 seconds to rephrase a comment like this into something friendlier.

binux13• 12 years ago

Yes, the scheduler, fetcher, processor is stand alone here, they are running in different process. But they are sharing some common libs. I haven't made a decision how to put them into a single package, and running together.

Any advice or project that I can refer to?

mrmondo• 12 years ago

Nice project! I do wish it supported a PostgreSQL backend rather than (or as well as I guess) MySQL.

OedipusRex• 12 years ago

Can someone explain what this is?

bjblazkowicz• 12 years ago

How's the performance compared to scrapy?

binux13• 12 years ago

https://gist.github.com/binux/67b276c51e988f8e2c31

adam-_-• 12 years ago

How does this compare to scrapy? Why would I use one over the other, or is either a fine choice?

skillachie• 12 years ago

+1 Scrapy comparison please

Can you compare to scrapy as requested by other posters. Why could you not build on top of scrapy and leverage celery for scheduling etc (http://www.celeryproject.org/)

What is the immediate value add to using pyspider ?

binux13• 12 years ago

I'm working on a benchmarking suite https://gist.github.com/binux/67b276c51e988f8e2c31 and meet some problem...

pyspider comes from a vertical search engine project. we have two issues:

- 100+ websites, they may change the template or down sometime. We need a dashboard to monitor the changes and the fails.

- update in 5 minutes, when the website updated, we need follow that in 5 minutes. We are using a update time from index(list) page to tell the changed pages. And pages should been updated after about 30 days in case of we missed something. A powerful scheduler is needed.

obviously, I hadn't got the right way to do so with scrapy. I'm not very familiar with scrapy. So I can't say something pyspider can do but scrapy not.

kidsil• 12 years ago

Thanks for making me feel bad about my python-based aggregation solution :)

https://github.com/AZdv/agricatch

erikb• 12 years ago

What is a "spider system"? Never heard that term before.

binux13• 12 years ago

sorry :(

_bitliner3• 12 years ago

I really like the flow/UX. Congratulations! Nice job!

What is the roadmap?

I am really inside scraping, it is one of my daily job. I could consider to integrate it in one of my architectures

_bitliner3• 12 years ago

Furthermore, what you mean with `Javascript pages supported`? Could I just specify where it has to click or do I need to make a reverse engineering of the ajax calls?

binux13• 12 years ago

http://demo.pyspider.org/debug/js_test_sciencedirect is a sample for this.

There is a phantomjs fetcher that can render the page as WebKit did. Furthermore, you can have some JavaScript running before/after page loaded to simulate a mouse click.

binux13• 12 years ago

To make it more flexible and easy to reuse? I have implemented most features I need now.

_bitliner3• 12 years ago

Because I already have a powerful distributed architecture. I was curious about the architecture of pyspider.

For example, how the queue is handled? Is it centralized? Is there a server managing it?

Show HN: A Python Spider System with Web UI

Related Stories