Show HN: Voyager – write a web crawler/scraper as a state machine in Rust

by matsche • 6 years ago

110 11 6 years ago

scudd2• 6 years ago

Cool library, I might use this for a side project I have also parsing HTML data from HackerNews.

What are your impressions of scraper and html5ever? When I initially looked at HTML/XML parsing libraries for Rust, there didn't seem to be a standout library such as serde_json for JSON data. I was also considering using scraper + html5ever. However, I'm curious if scraper adds enough to warrant the additional dependency as opposed to directly using html5ever.

matsche3• 6 years ago

I haven't used scraper too much. I personally find the predicate approach of select.rs [0] easier to use. However in this case the selector approach just made more sense. Standalone html5ever can be a bit cumbersome to work with directly, scraper is basically an implementation of the html5ever's `TreeSink` trait, where as `select.rs` uses the hmtl5ever `RcDom` to parse the document but stores it in a more convenient way. If you look for a minimal approach you should at select.rs which basicially only depends on html5ever

[0] https://github.com/utkarshkukreti/select.rs

scudd2• 6 years ago

Hey just FYI, when you run the hackernews and explore examples without enabling the tokio feature flag, you get compilation errors about undeclared types for all the tokio stuff. I think these examples just need entires in the Cargo.toml requiring the tokio feature like the reddit example.

Once I add the tokio feature, they all run as expected.

singhrac3• 6 years ago

I tried to build a scraper in Rust just a few days ago and got stuck trying to concurrency limit my calls (the website I was scraping appropriately had a rate limit). I couldn’t figure out how to get tokio::stream/tokio_stream to work. Does this fix that problem?

ashish01• 6 years ago

You can limit the numbers pf concurrent calls using a semaphore and acquiring it before making a http request.

matsche3• 6 years ago

Yes, you can either limit how many request can be send concurrently or enforce a fixed or random delay in between consecutive requests.

Basically this a just a futures_timer::Delay [0] that is reset after each request, which is non blocking.

[0] https://docs.rs/futures-timer/3.0.2/futures_timer/

singhrac3• 6 years ago

How do you set the concurrency limit? I'm down to use your framework (thanks!), just curious how you implement it. I couldn't get Stream::buffered to work correctly.

michael_j_ward• 6 years ago

I assume the website has something like "requests per second" quota, which then you'd want the `governor` crate [0].

It was recently used to implement rate limiting middleware for both tide[1] and actix[2]

[0] https://docs.rs/governor/0.3.1/governor/_guide/index.html

[1] https://github.com/ohmree/tide-governor

[2] https://github.com/AaronErhardt/actix-governor

Show HN: Voyager – write a web crawler/scraper as a state machine in Rust

Related Stories