Escape the walled garden and algorithm black boxes with RSS feeds

https://www.johnwalker.nl/posts/escape-the-walled-garden-with-rss

by rekl • 2 years ago

297 106 2 years ago

vaylian• 2 years ago

The article also mentions atom feeds. I've seen plenty of RSS feeds but I have never encountered an atom feed. Are atom feeds still relevant?

giantrobot• 2 years ago

Atom and RSS are functionally equivalent despite their technical differences. Effectively no one has ever implemented Atom push capabilities so Atom is primarily only ever used for syndication. The term "RSS" is just a generic term for "XML syndication feed".

progval• 2 years ago

Atom push is actually implemented. SWORDv2 (but not v3) is based on Atom Push, and https://en.wikipedia.org/wiki/SWORD_(protocol) lists a few implementations.

My employer maintains a SWORDv2 server, and we have a handful of clients pushing to us; most of them are not listed on that page.

crtasm• 2 years ago

As an example, feeds available on every github repo:

https://github.com/EFForg/privacybadger/commits.atom

https://github.com/EFForg/privacybadger/tags.atom

https://github.com/EFForg/privacybadger/releases.atom

superkuh• 2 years ago

Google (Blogger, etc) pushes Atom feeds. Weather blogger is still relevant is up to perspective.

cadamsdotcom• 2 years ago

That helps find content to read.

I wish it was easier to find out what my friends have been up to without getting them to sign up for some platform they’ve never heard of, then post in multiple places in perpetuity, and move on again when that platform also goes to shit.

A hard problem but surely not unsolvable. It belongs in a pg “please solve these big problems” essay.

meiraleal2• 2 years ago

That's an interesting problem. I'm thinking about creating a local-first RSS Reader that syncs using github. It seems doable to create a personal feed based on my feed and publish it also to github.

kcplate4• 2 years ago

Def how I consume HN. It’s even nice to see flagged and dead which I wouldn’t see easily if I came via the HN homepage.

xrd5• 2 years ago

How do you get this, may I ask?

I asked because I didn't know how to get exactly that, so I'm building a site that finds RSS feeds from sites on the front page of HN. Then, at the end of the day it emails me with a summary if I miss one.

https://RSS.surf

If anyone wants to try the daily summary email would love feedback.

kcplate4• 2 years ago

I use Feedly, which as a RSS reader is ok but probably one of the better ones available on iOS/iPadOS platform. I loved Google Reader back in the day, but Google loves to kill every product of theirs that I actually like and used.

I believe the HN feed is available here: https://news.ycombinator.com/rss

ashryan• 2 years ago

This is so pushing happy buttons for me.

Not that this is novel in any way, but I just started a repo call Subcurrent yesterday for the Astoria Tech Meetup in NYC at our Saturday hack session. Subcurrent aims to provide a feed aggregator page made of our community members' feeds. https://github.com/astoria-tech/subcurrent

I did not know that Meetup.com exposes RSS feeds at all, so I will be adding that to our Subcurrent instance since our group keeps events on Meetup.com.

I had never heard of Kill the Newsletter, but I'm a fan sight-unseen. Substack at least has feeds. You can append `/feed` to the newsletter's URL.

Thanks for writing this!

soupfordummies• 2 years ago

So many RSS stories this week! I'm sensing a trend (hopefully)

Pooge• 2 years ago

I'm sorry to be the pessimist here, but I doubt it. HN users are the most likely to use RSS in the first place. I sincerely doubt that RSS is going to make a comeback this year with all my non-IT friends using it.

meiraleal2• 2 years ago

Don't underestimate what programmers with too much free time can achieve.

azemetre• 2 years ago

Why is the goal to appeal to as many people as possible?

bluebarbet2• 2 years ago

Realism shared. I work with journalists, exactly the type of normies who would most benefit from this technology. Years of evangelism has had no effect whatsoever. Nobody they know is using it, they don't see the logo anywhere, or a big friendly "Get started" button. It's all so unfamiliar and technical-sounding. Even the name itself was a disaster, IMO: a hopelessly geeky and opaque acronym. It should have been called Webfeeds! It's all such a wasted opportunity.

Eji17002• 2 years ago

Honestly that might be for the best in some ways. I see rss as "allowed to exist" because not enough people use it. There's plenty of ways to subscribe to things that I enjoy right now that could be nuked if suddenly enough people were using it that they weren't seeing the conversion rates they wanted.

fgeiger• 2 years ago

I for one am consuming HN through RSS. I find it is incredible.

coffeefirst• 2 years ago

I think there’s a genuine desire amongst people who think about these things to take back control from the algorithms. That’s real.

Scaling it to a wider audience than HN is a longshot, as much as I love the idea.

jopsen2• 2 years ago

This is missing planets.

A good way to find interesting blogs is to subscribe to a few planets.

These are essentially aggregations of blog related to some project/topic.

https://planet.gnome.org/ https://planet.kde.org/ https://planet.mozilla.org/ https://planet.documentfoundation.org/

PS. If you know any good planets worth skimming, please add to below :)

That said, I don't really have a good RSS reader that syncs across devices. I currently use Feedly, but it tries to be too smart.

gvurrdon• 2 years ago

I tried Feedly after Google Reader shut down but eventually settled on Feedbin - that might be worth a look.

Gualdrapo• 2 years ago

I'm on the Feedly train boat too since the sinking of Google Reader. Besides the ai bs it doesn't really let you search something in your feeds without a subscription, so once I tried Feedbin.

But I went back because third world issues (and not a fan of microtransactions either) I could not find a way (if any) to change the layout on desktop. I really like the minimal list view at Feedly (and even on mobile too).

jopsen2• 2 years ago

If I read feeds daily I might spend 5 bucks per month, but I don't.

It'll just be a subscription I forget that I have. I don't need more of those :)

ognarb• 2 years ago

Since planet.kde.org is mentioned, it's super easy to create something similar with a bit of python and a static site generator like Hugo :)

https://invent.kde.org/websites/planet-kde-org/-/blob/master...

mamidon2• 2 years ago

What are planets, in this context?

tempest_• 2 years ago

https://en.wikipedia.org/wiki/Planet_(software)

https://web.archive.org/web/20051029095046/http://www.planet...

https://intertwingly.net/code/venus/docs/index.html

https://github.com/kgaughan/mercury

flobosg• 2 years ago

Feed aggregators: https://en.wikipedia.org/wiki/Planet_(software)

daveguy• 2 years ago

They are just feed aggregators composed of multiple feeds. Based on this software:

https://en.m.wikipedia.org/wiki/Planet_(software)

ssttoo• 2 years ago

(Disclaimer: my project)

https://feed.perfplanet.com for web performance

Also please open a GH issue if I’m missing a blog or 5

PaulHoule8• 2 years ago

I think it's little appreciated that planets solve many of the practical problems of feed crawling.

If you wanted to follow 2000 blogs yourself you'd find it is really a hassle. You can follow one planet and its easy.

For that matter, if 2000 people want to follow your blog (and many other blogs) they are going to generate 2000 requests per polling period. It is not wonder why people like [1] get so exasperated. There are three kinds of polling periods: (1) too fast, (2) too slow, (3) both at the same time. Instead of having 2000 people poll your blog too often, one planet can poll your blog. It improves the scalability and economics of the system dramatically.

(e.g. the difficulty of finding a good polling regime is one of 10 or 20 or so unappreciated reasons why RSS has remained nerdcore)

[1] https://rachelbythebay.com/w/2024/05/27/feed/

scarface_742• 2 years ago

And her RSS feed doesn’t work at all

Unearned5161• 2 years ago

big fan of feedmail [1] for a "dumb" reader. It just delivers it to your inbox. I have a folder set up called "feed" and I can do all the email things to the items landing in that folder. Every email costs a credit, you can buy 10 000 credits for $10, which works out to $0.001 per article.

The only thing it misses is any sort of highlight saving deal, but for those I just save the article to zotero and annotate it there.

[1] https://feedmail.org

twapi• 2 years ago

Thank you for reminding me about planets! I had forgotten about them, and your post has inspired me to explore them again. I appreciate your insights.

8organicbits• 2 years ago

I have a project that is collecting blogrolls and planets, especially those that publish OPML files. Here's my list of around one hundred planets!

https://github.com/robalexdev/rss-blogroll-network/blob/385d...

These are aggregated and enriched to build this site: https://alexsci.com/rss-blogroll-network/blogrolls/

darthrupert• 2 years ago

Perhaps you'd like Miniflux, it seems too simple to me.

qudat• 2 years ago

If anyone needs an rss-to-email service, check out https://pico.sh/feeds

vishnu_ks• 2 years ago

Do try https://diff.blog which is an aggregator of developer blogs which I built around 5 years back.

diff.blog tracks over 2000 dev blogs at the moment.

And you can also follow blogs and topics.

delduca• 2 years ago

+ for Reeder for macOS and iOS

I follow several feeds and Youtube channels.

Reeder have a very smooth experience.

terminaltrove• 2 years ago

Try adding our feeds if you're looking for curated terminal tools every week, we have a blog as well with ATOM.

https://terminaltrove.com/feeds/

All our feeds have an easy to see preview of the feed instead of unstyled XML so you know what the feed looks like.

https://terminaltrove.com/new.xml

fevangelou• 2 years ago

A handy bookmarklet to find & preview any site's feed (before subscribing):

https://feedreader.xyz

domysee2• 2 years ago

Similar tool here: https://lighthouseapp.io/tools/feed-finder

Finds feeds even if they're not RSS autodiscoverable. Everything it does is described here: https://lighthouseapp.io/blog/deep-dive-finding-rss-feeds

frobnic• 2 years ago

Try Rss-Bridge [1] when the website does not have any feed, it might have an integration already. It also supports custom CSS-selectors to create feeds, or even use SEO-Sitemaps for your advantage to generate a feed from it.

[1] https://github.com/RSS-Bridge/rss-bridge

janice1999• 2 years ago

I see PolitePol mentioned. What do people use to run scrapers to generate RSS feeds themselves for sites without them? Do you self host scrapers?

dagurp2• 2 years ago

I do. Just a raspberry pi with nginx and a cron job that runs python scripts.

I'm not into AI but writing those scripts is boring so chatgpt is perfect for this.

Havoc2• 2 years ago

There is one called rssbridge or something that can do that

PaulHoule8• 2 years ago

The hate for algorithms boggles my mind.

Chronological feeds are awful. You'll never see anything from the people who post occasionally because they get drowned out by the people who are posting all the time.

There may be some algorithms that deliberately magnify hate because that's a way to increase engagement, but if you want to create one of those algorithms you can make a training set based on chronological feed + boosting/retweets/reposts.

I'm amazed at how people keep making failing RSS readers that keep failing with the same failing user interfaces that have been failing since 1999; everybody knows RSS has been failing but they never ask why or if we have a choice.

We still see the readers that make you mark things as read, that take their cues from email and newsreaders, that, when you subscribe to N feeds show you N boxes with a list of items, etc.

My RSS reader works like TikTok because I'm not afraid of algorithms.

beej712• 2 years ago

I prefer time-based feeds, but that's just me.

I don't think people are objecting to suggestive feeds in general; they're objecting to suggestive feeds whose primary objective is to keep you scrolling for as long as humanly possible to maximize company revenue.

That's simply not what I want to do with my day.

I'd train my own if I wanted to go that route.

PaulHoule8• 2 years ago

The problem with time based feeds is this.

You can read x articles a day; your system ingests y articles a day.

x=y is perfect but requires close-to-perfect balance (if x=0.9y to 1.1y maybe you can adjust your reading habit to your your feed)

if x>y then your system isn't showing you enough, if y<x you are going to miss things you subscribe to based on some arbitrary or random characteristic.

With an algorithmic feed of some kind you choose to read x items a day, your system shows you the best x items a day out of y based on some set of criteria and constraints.

These things are common sense but seemingly nonsensical to a lot of people. For instance our impoverished rights-based discourse (see [1]) about "free speech" presupposes that 100% of people can read 100% of what everybody else posts, realistically platforms can only show people some fraction of what gets posted so one thing is going to get more visibility and other things get less and that's a choice -- it could be random but it's still a choice. (As Rush would put it, "if you choose not to decide, you still have made a choice")

I think the discussion is so impoverished that we never hear that an algorithm could choose to do anything other than maximize profits for a platform, when in fact that is just one thing an algorithm could do out of countless options.

[1] https://www.amazon.com/Rights-Talk-Impoverishment-Political-...

ahalimah• 2 years ago

Which RSS reader do you use in that case?

And is it fair to say there's a middle ground between purely chronological feeds vs algorithms that reward engagement/time spent on the app?

PaulHoule8• 2 years ago

An algorithm doesn't have to reward engagement or time spent on the app. It can reward or punish anything at all -- and that's what it is about algorithms.

My feed reader (that I wrote myself) rewards things that I thumbs up and punishes things that I thumbs down.

openrisk5• 2 years ago

You'll never see anything from the people who post occasionally because they get drowned out by the people who are posting all the time.

If you control the algorithm there is nothing to prevent you from sorting feeds by volume and adjusting their presentation accordingly.

Actually you can imagine countless other UI adaptations depending on preferences, usage patterns etc.

Ideally RSS readers should offer flexible customizations, e.g., with plugins or even some low-code environemnt.

PaulHoule8• 2 years ago

It's one thing to sort a list of things by one criteria, say "a". If you have two criteria, say "a" and "b" you can sort by "a" and break ties by "b" or you can sort by a+b or a-b or something like that... But that's not the same as optimizing both things. Maybe you can say "there are these 5 people who are special to me who I never want to miss their posts" but I think people would struggle to maintain rulesets and might not really be happy with the results they get.

In search rankings for instance you night have a document score (like Pagerank) that tries to identify the quality of a document, and you might have a query-document score that identifies the relevance of a document to a query. It's not trivial at all to find a way to blend those that gives you queries that are both relevant and quality as opposed to just one or the other.

The greatest weakness of my current RSS reader is that it's slow, depending on how much I am using it, articles could be delayed anywhere from a day to a week. For certain kinds of articles [1] recency doesn't matter, but other articles [2] have a definite shelf-life and if you repost them too late you look like a total dope.

I'd like to let articles about sports "cut the line" in front of higher quality articles about other topics but it's really hard to find a balance that's right because I don't want to get flooded with lower quality sports articles. It's one thing to say "let people make up their mind about how to balance these things" but when you really try it you find it's pretty hard. Not only do I have to change the whole way my pipeline works (can't be a batch job anymore) but it's not clear to me how to tune up the selection criteria.

[1] https://www.sciencedirect.com/science/article/pii/S294979062... [2] https://www.usatoday.com/story/sports/nfl/2025/01/19/eagles-...

nonrandomstring• 2 years ago

Let it go Paul. It's like the word "hacker". We ain't getting it back.

It's sad the popular perception of a fundamental computer science concept, the great progeny of Al-Khwarizmi Musa, the life work of heroes like Donald Knuth, is sullied and stained with dire B-movie comic-book Bond villainy. Thank the hooligan broligarchs and their manchild ambitions for trashing that bit of computing culture.

Even my mum spits in the dust like a cowboy at the sound of "Algorithm".

PaulHoule8• 2 years ago

It might be easier to convince the people at the CS department to name the "algorithms" class something else.

josephcsible• 2 years ago

Chronological feeds are awful. You'll never see anything from the people who post occasionally because they get drowned out by the people who are posting all the time.

My preference would be chronological feeds, but following few enough accounts that I can see every single post in it. Then that's not a problem.

Eji17002• 2 years ago

You'll never see anything from the people who post occasionally because they get drowned out by the people who are posting all the time.

I use inoreader and track basically everything I want in a huge list of feeds.

It's pretty trivial to mark stuff as "always flag this", and then leave the rest of the pile as "scan through and manually tag anything else"

Anything low volume goes on the "always" flag list as it takes one review a day of the new content to decide if it's something I care about.

Ultimately what I want, and what most people want, is the ability to just hook up to various data streams and apply rules to it. From there filtering as desired comes pretty easily.

Havoc2• 2 years ago

People aren’t afraid of algorithms…just ones that are Blackbox and controlled by someone else who’s motivations don’t align with yours.

Being concerned about that is downright logical.

edoceo• 2 years ago

I love RSS but how can we get more of it? Walled-ish gardens seem to dominate. Many good producers are on platforms that just don't syndicate. What kind of pressure can we, consumers, put on?

tap-snap-or-nap• 2 years ago

Request good producers to use platforms that syndicate.

jefurii• 2 years ago

If they're not using an RSS-capable platform maybe they're not so great?

ColinWright2• 2 years ago

Every Mastodon account automatically has an RSS feed ... just append ".rss" to the account name:

https://mathstodon.xyz/@ColinTheMathmo.rss

openrisk5• 2 years ago

An advanced RSS reader/browser could bring back the magic of the web.

xrd5• 2 years ago

What does an advanced reader look like in your mind?

openrisk5• 2 years ago

1. Convergent desktop/mobile app that syncs between devices (open source and local)

2. Performant for large numbers of feeds

3. Integrated browser and automagical discovery and organization of feeds while you browse

4. Multiple taxonomies and viewing layout options, chronological, by subject etc

5. Advanced filtering by keywords

6. Transparent and pluggable locally runing algorithms to track usage and inform the user of patterns and if desired adjust presentation

In a sense an advanced RSS 'reader' is what the web "browser" should have evolved to. There is really no real boundary between these two clients.

An advanced RSS reader is essentially a more dynamic browser that queries the internet in more ways that the user-initiated visit of a bookmarked url or typing something into a search form.

uzerfcwn• 2 years ago

For me, that advanced reader is Thunderbird. It has amazing customizability with userchrome.css, web extensions, about:config and developer tools from Firefox.

quinto_quarto• 2 years ago

i built this site (https://fanzine.world) to help people discover micro-creators and share their top 5 faves. not sure what to do with it. maybe someone here has a suggestion?

geor9e• 2 years ago

Feedbro is my favorite reader.

pyromaker• 2 years ago

Is RSS really coming back? :) (you'll say it was never gone!) Lots of RSS related posts and comments recently. I've released Mashups a few weeks ago.

https://www.mashups.io

It's yahoo pipes clone - so you can mix and filter RSS feeds that you want.

RamblingCTO• 2 years ago

awesome! good luck

pigeons2• 2 years ago

I need to find some way to see a chronological feed of what people I am friends with on facebook post. I literally get randomly suggested posts from unrelated accounts, and scrolling through them all will start to repeat them, never showing me content from some people who are actual friend connections.

BugsHack• 2 years ago

Try this: https://www.facebook.com/?filter=friends&sk=h_chr

pigeons2• 2 years ago

Thanks!

Beijinger• 2 years ago

You need this: https://socialfixer.com/

At socialfixer, options, hide post put everything that should disappear follow reals etc..

I am not on my laptop otherwise I would post my block list here.

ColinWright2• 2 years ago

Every Mastodon account automatically has an RSS feed ... just append ".rss" to the account name:

https://mathstodon.xyz/@ColinTheMathmo.rss

xyst• 2 years ago

RSS feeds, so underrated. Rediscovering RSS and adding my YT subs as RSS feeds feels so liberating.

Plus, RSS feeds combines the reg videos + short videos into single feed.

rubicks• 2 years ago

Relevant: https://newsboat.org/

If you like your email in mutt, then you'll probably like your feeds in newsboat.

OisinMoran• 2 years ago

I love this!

I'm trying to build a new corner of the old web with my social link sharing site https://lynkmi.com, and every tag automatically has an RSS feed so you don't need to know anything about them to set one up, or even need an account to follow one.

The main idea is instead of following everything a person posts you can just follow a subset of their interests. So if I post about Irish Dairy Innovations [0] and also about Advice [1] you can follow whichever combination of those you like.

If you'd like to sign up, my email is in my bio. And if you don't want to sign up, my email is still in my bio.

[0] https://lynkmi.com/oisin/Irish%2520dairy%2520innovation [1] https://lynkmi.com/oisin/advice

toofy• 2 years ago

i would love to see Bluesky add RSS to user feeds the same way Mastodon does by just adding /rss to the end of the url.

it really is rad to add my most read users to the feed.

xavdid• 2 years ago

Bluesky supports them! If you append `/rss` to any profile, it redirects to a feed. e.g.:

https://bsky.app/profile/xavd.id -> https://bsky.app/profile/did:plc:fp2izjrrcsbsqphlu7f5sixa/rs...

More info: https://openrss.org/blog/bluesky-has-launched-rss-feeds

aboardRat4• 2 years ago

I've been trying to make it work for ages, but it's not going to work.

"News" are not actually that easy to automate as in "serialising posts".

Big companies are experimenting with UI all the time for a reason.

Especially because it would involve a lot of heuristics.

renegat0x0• 2 years ago

ooooh I so escaped walled gardens.

I use my own set of domains to find places on the internet [0].

A working example how it could be used is at [1]. It is a domain viewer in javascript.

I also use my own RSS client [2] that stores all links in [3], but don't get me wrong, I also have my storage for bookmarks [4]

Links:

[0] https://github.com/rumca-js/Internet-Places-Database

[1] https://rumca-js.github.io/quickstart/public/static_lists/vi...

[2] https://github.com/rumca-js/Django-link-archive

[3] https://github.com/rumca-js/RSS-Link-Database-2025

[4] https://github.com/rumca-js/RSS-Link-Database

domysee2• 2 years ago

Really great to see so many RSS stories recently.

I'm working on a feed reader, called Lighthouse (https://lighthouseapp.io/). It combines RSS feeds with read-it-later, by putting new content into Inbox, where you can either archive or bookmark. Bookmarked content shows up in Library.

It's fantastic for content curation.

sys_64738• 2 years ago

There's also https://brutalist.report

Escape the walled garden and algorithm black boxes with RSS feeds

Related Stories