Homemade RSS aggregator followup

http://leancrew.com/all-this/2015/12/homemade-rss-aggregator-followup/

by ingve • 11 years ago

70 46 11 years ago

krylon2• 11 years ago

Very interesting!

I am currently building an RSS aggregator, too. Mine is little more complex, though - I wanna be able to rate items as interesting or boring and use some kind of filter (currently, a simple Bayesian classifier, I intend to replace or at least enhance with something more sophisticated over the holidays) to weed out news that I am not interested in.

The biggest problem is that web design is not my strong suite (to put it mildly), so the thing looks pretty ugly. Classification does not work very well, yet, but I am not sure if this is because the classifier sucks in general or if my training set is too small at the moment (I've only been using the thing for a couple of days now).

Anyway, it is quite interesting to see another approach to the problem.

alanpost• 11 years ago

Do you use a single classifier for your entire feed, or do you categorize the feeds and maintain a classifier for each topic? (Or, as always, secret option #3: neither.)

krylon2• 11 years ago

Currently, I use a single classifier for all the news items.

I store the items, along with manual ratings, in a database, from which the classifier is trained. (This also allows me to keep a history of older items, which I can search.)

I hope to make it more sophisticated, eventually.

dubbel• 11 years ago

If you use Bootstrap CSS it looks pretty generic, but it's also nearly impossible to make it look bad.

Sounds like a cool project.

pyre• 11 years ago

There's also stuff like MDL[1] (or Angular Material[2] if you're using AngularJS).

[1] http://www.getmdl.io/

[2] https://material.angularjs.org/latest/#/

abengoam2• 11 years ago

That's awesome! I should know because I also created my own RSS aggregator after the demise of Google Reader.

Here's a screenshot https://imgur.com/YHJOiEX

It fills my needs perfectly because I created it specifically for myself and I control it fully in all aspects.

It's been such a tremendous success for me and so fun to create that I am thinking about replacing other online services with custom-made versions, such as google calendar, google tasks, etc. Something to look forward to in 2016.

Great job, and keep at it!

scrollaway3• 11 years ago

Your reader looks awesome. Did you open source it?

abengoam2• 11 years ago

Thank you! Alas, I did not. I am fully behind the open source/free software movement(s), but right now I am at a point in my life where I can't manage and support an open source project as my availability is super spotty. Also, you don't want to see the css... it makes gotos look good.

m_mozafarian• 11 years ago

After the demise Google reader, pretty much everybody I know started building their very own RSS Reader. We also made one for the Relevant app. It's beautiful card called RSS Reader that you can add it from the library. Then just paste your rss urls in the back of the card and it works. (Currently iOS only). http://relevant.ai/

aw3c22• 11 years ago

My perfect aggregator would also create a WARC archive of the webpage of each post, including all external references, maybe the referenced external websites and their references (with that single depth of recursion). The internet is friggen fragile and I would love to archive what I consume.

vidarh• 11 years ago

I agree with that. I've recently been going through my archived blog posts while revamping my blog design, and so much has just flat out disappeared. And so much of what has disappeared have been things I would have guessed would stay...

rakoo• 11 years ago

I know it's not a feed aggregator, but you could intrumentalize pinboard (pinboard.in), they have the option of retrieving and storing an archive of all your links if you so desire. They even resolve first level dependencies, so external images are also stored (see https://blog.pinboard.in/2010/11/bookmark_archives_that_don_...). See some numbers on link rot here: https://blog.pinboard.in/2011/05/remembrance_of_links_past/

Pinboard is built as a bookmark manager, but if you say that all entries in a feed is a bookmark then it should work for you. Oh and there's full-text search as well.

Gmo• 11 years ago

I think you can replace "they" by "he", since, as far as I know, it's a one-man shop !

epaulson• 11 years ago

Pinboard is fantastic, but it doesn't look like it exposes the archived pages through the API. If you ever wanted to leave Pinboard and do a data export, you might not be able to take everything with you.

zrail• 11 years ago

That's a really great idea. Here's a one-line wget that will grab the provided URL and all of the data necessary to render, to one level of recursion, and dump it to a WARC:

    wget -e robots=off \
    --user-agent="Mozilla" \
    -r 1 -p -E -H -k -K \
    --warc-file=/path/to/your/warc/file/without/warc/extension \
    'http://www.example.com'

I think I might start capturing these. Shouldn't take up too much additional disk space.

edit: previously it was `-r 2` which is two levels of recursion.

scrollaway3• 11 years ago

You can shorten that to `-pEHkKr 1`, so you know.

aw3c22• 11 years ago

You might have more fun with wpull, it has more sensible options to grab resources on referenced urls. https://github.com/chfoo/wpull

sheraz2• 11 years ago

What if, instead of a WARC file, just a high-res screenshot available in desktop, tablet, and mobile mode?

jimmyswimmy• 11 years ago

Why would a binary file be better than its source? Seems like a lot more work (you'd need to format and store for each display size) in exchange for less capability (can't search, probably a larger overall file).

derefr3• 11 years ago

To go further: there's basically no point in the "description" part of an RSS item. RSS is broken in that authors need to lure people onto their sites, so they make the RSS item itself enclose just enough of a preview to make you "click through"—whereupon their site can show you ads and they can make money.

How RSS should work, in an ideal technical sense, is to eschew enclosing any content-body in feed items themselves, and instead just encourage RSS consumers (feed-reader clients; feed-muxer daemons) to scrape the permalinks of the feed items, and then heuristically extract the body-content from the scrape-result, and cache both the resulting page-archive and the resulting cleaned-up text, making both representations available offline.

This, obviously, kills blog ad revenue. But it's better to kill it and replace it with something better (402 micropayment-required errors at point-of-caching, handled automatically by the RSS content-spidering daemon as an HTTP client, with costs passed on to its subscribers?) than to continue on with this semi-braindamaged "I have an offline cache but that doesn't actually mean I can read anything offline" world.

okasaki• 11 years ago

All the blogs I read include the full content in the feed. Some even include top comments.

And if you're parsing sites then you have no use for RSS - you can just parse the index to see what's new. Sounds like an nightmare to me though. Who's going to maintain these parsers for hundreds of sites?

cbr• 11 years ago

Most people I follow on RSS include the full content of posts in their RSS feeds. When someone doesn't, it's generally an oversight and they fix it when I bring it up.

petercooper• 11 years ago

I made a similar script but that also has 'plugins' so 'URLs' like @username, twitter:topic, /r/subreddit, and hn:topic load up the tweets, Twitter search results, sub-Reddit items, or HN search results respectively for certain keywords, using their respective APIs.

oneloop• 11 years ago

Care to share?

fasouto• 11 years ago

I started creating an RSS aggregator some time ago (https://github.com/fasouto/django-feedaggregator) and it was more difficult than expected. There are many broken feeds and different interpretations of the standard.

One day I should finish it...

rcarmo2• 11 years ago

Check my top-level comment.

wanda• 11 years ago

If anybody happens to be looking for an RSS aggregator, I'd like to recommend GoRead.

Obviously I wouldn't pay for it, but self-hosting is pretty straightforward and it has a companion Android app.

Never cared for Feedly and I don't really fancy making my own.

It's the best Google Reader clone I've found.

https://www.goread.io

ents• 11 years ago

I don't like feedly either, but as a backend for apps it works fine, and is free.

oxplot• 11 years ago

Given that email clients are fairly mature and advanced already (especially Gmail's), it seemed logical to go the unix way and use it as the UI to stream of feeds sent as email. I wrote a bit of python [1] and stuck on it a free openshift cartridge and it sends me one email per feed item. It's been up since April this year shortly after I abandoned feedly. I like it more than Google Reader now.

[1] https://github.com/oxplot/lapafeed

pmoriarty• 11 years ago

I've recently come back to using newsbeuter[1] and have been quite impressed. It's really feature rich and very customizable. It's a terminal app, which some might not like, but for me that's a plus.

[1] http://www.newsbeuter.org/index.html

younata• 11 years ago

Been working on my own RSS reader for iOS. https://github.com/younata/RSSClient/

Pretty much all the internal logic (parsing feeds/opml files) are also written from scratch, which was interesting to do.

rcarmo2• 11 years ago

I have a fair amount of code that people can re-use to build their own aggregators, since I did a number of experiments when Google Reader died.

One was a Fever clone that had a number of strategies for doing parallel fetching:

- https://github.com/rcarmo/bottle-fever

Andrea Peltrin took that and evolved it into Coldsweat: https://github.com/passiomatic/coldsweat - which I recommend if you want a web UI.

I did a number of other things, but eventually went back to what I used _before_ Google Reader: e-mail.

I was one of the contributors for http://newspipe.sourceforge.net/, and after getting bottle-fever going I decided to investigate the state of the art and did a quick fork of rss2email that injected messages into an IMAP store instead of sending them via SMTP, to avoid spam traps.

It was a quick hack, but it allowed me to read feeds using any mobile IMAP client, and a friend eventually did a Go version, which I've also tweaked to my liking:

- https://github.com/rcarmo/rss2imap (Python) - https://github.com/rcarmo/go-rss2imap (Go)

Any of the above are likely to save people a fair amount of time (do bear in mind that the Python version was a hack atop code that was written by Aaron Swartz a decade ago, and it shows its age).

These days I ended up going back to Feedly, simply because I have to use Windows, the Web UI is good enough and there are lots of good clients for the platforms I use (NextGen, Reeder, etc.)

Plus I realised that trying to archive stuff from hundreds of feeds was somewhat pointless -- the stuff I really want to keep around goes into Pocket or OneNote, and that's that.

Edit: Also, here are some notes from 2008 on Bayesian classification and its effectiveness: http://taoofmac.com/space/blog/2008/01/27/2203#an-update-on-...

voltagex_• 11 years ago

I like the idea of (ab)using protocols to do not-quite what they were intended to do.

Pushing RSS feeds into IMAP is a great idea - I wonder how much work it'd take to make NewsBeuter to that, then expose it somewhere and have FastMail pull it into a folder for me.

These hacks eventually start looking like Rube-Goldberg machines, but they've got a certain charm.

Offtopic: I wrote a Wake-on-Lan server that allows me to turn on VMs as if they were physical machines - https://github.com/voltagex/junkcode/tree/master/CSharp/Virt...

The next one for me is probably going to be a DNS server that resolves the name and IPs of VMs.

newtang• 11 years ago

I built a similarly simple feed aggregator for anyone to use at https://plumfeed.com Shows the most recent post of each feed. I've found it particularly nice for blogs and comics that update once a day or less.

mmb• 11 years ago

The bigger problem is that the amount of information you can consume using RSS feeds is declining. Most sites don't publish RSS feeds of their content any more.

Sadly RSS is left over from a time when things were more open. Now everything is an app and everyone wants you to stay in their walled garden.

anotherevan• 11 years ago

Yeah, there's a couple of interesting sites I would like to follow, but they don't provide RSS feeds, so I don't bother.

l1n• 11 years ago

I personally use http://changedetection.com/ to create RSS feeds for sites that don't have them (though I may move to Huginn agents in future).

fulafel• 11 years ago

There seem to be many services for addressing this. I wonder if anyone has recommendations about which one to use? Searching for "auto generate rss" returns at least a screenful of these.

ojiikun• 11 years ago

Surprised there has been no mention of NewsBlur. Though you can pay the author for use of the central instance, it is also fully open sourced on github. Saying it is more feature-rich than most examples would be an understatement.

gerty• 11 years ago

I guess Tiny Tiny RSS hasn't been mentioned yet. FOSS, self-hosted with multiple Android clients. I had been using Feedly since Google Reader went down but should have actually been using TTRSS since the beginning. I ain't no power user but it has definitely more than I ever would ask for.

hasteur• 11 years ago

Adding annother "Google Reader" replacement for TinyTiny-RSS. There is some jiggery-pokery" that needs to be done, but it does nearly everything you could want (I wouldn't mind getting the behind the cut reveals)

axx• 11 years ago

I'm also working on an open source RSS Reader called HappyFeed. It's compatible with Fever RSS API, so you can use it with Reeder, ReadKit, Press and so on.

I work on this project mainly for myself, but if you're interested and want to contribute, feel free to get in touch!

Screenshots and development blog: https://need.computer/happyfeed/2015/12/20/happyfeed-drag-an...

GitHub: https://github.com/aleks/HappyFeed

Homemade RSS aggregator followup

Related Stories