HN user
Maybe. Don't think any of this existed when I need it...
1. The code is tiny with 90% of it dealing with RSS parsing and filtering. Using RX.NET wouldn't really simplify anything.
2. I wanted a library that I can integrate into my apps and run locally to avoid throttling, robots.txt and other BS Yahoo Pipes was suffering from.
I am also not a huge fan of RX... to put it mildly
Can’t really remember, sorry, that code was built for processing RSS feeds and data size was never an issue. I will take a look when get some free time…
But, I see what you mean. I had to deal with similar issues in commercial projects and the "pull" model (generators in Python ~ "yield return" in C#) almost never a good idea, especially when you have to have concurrent consumers. While callbacks are hard to combine, in C# it can be nicely abstracted with “async/await”, not sure how it is handled in Python, I stopped using it around 2.5
Nice project. I wrote something similar in C# long time ago [1]. Mostly to monitor job feeds and craigslist [2] :-) It supports RSS and Atom, async, various filters, deduplication, etc
Yahoo Pipes was a nice project, but as its popularity grew, it started getting blocked more and more. It was also hard to build and maintain pipelines with more than a few steps.
[1] https://github.com/olviko/RssPercolator
[2] https://github.com/olviko/RssPercolator/blob/master/RssPerco...
Just curious, why not Redis?
Same here. InfluxDb has a pretty nice DSL. I wish they switched their backend to something usable and mature instead of re-inventing the bicycle. Operationally, it is expensive to support specialized databases like Influx, unless it is your core business, I guess...
What are the benefits of introducing specialized time-series databases vs using Redis, Casandra, or some Sql database?
Checkout https://github.com/olviko/RssPercolator
Pipes would often get blocked by 3rd party data providers, which motivated me to roll my own library.
No graphical interface. Personally, I found Y! Pipes UI hard to work with when I needed to build large pipelines.
Checkout https://github.com/olviko/RssPercolator.
RssPercolator is a .NET library used for downloading, aggregating, and filtering RSS feeds. Developed out of frustration with Yahoo Pipes.
- RSS and Atom feed formats - Asynchronous multi-feed download - Multiple sources and multiple destinations - Feed filters (string match, wildcards, and Regex)
If you like Yahoo Pipes as a concept this library is for you ;-)
- RSS and Atom feed formats
- Asynchronous multi-feed download
- Multiple sources and multiple destinations
- Feed filters (string match, wildcards, and Regex