So many feed readers, so many behaviors

https://rachelbythebay.com/w/2024/05/27/feed/

by zoidb • 2 years ago

130 98 2 years ago

A fair number of people are sending conditional requests, but are doing it every 5 or 10 minutes. This is ridiculous. I don't write that often, and never have.

This objection might make more sense if the purpose of a feed reader was to do nothing except check rachelbythebay.com for updates.

komadori• 2 years ago

At least at the point you receive a 429, the Retry-After header provides some guidance as to how frequently the resource should be polled.

supriyo-biswas• 2 years ago

There’s websub/PSHB[1], though I don’t know how widely it’s implemented.

[1] https://en.m.wikipedia.org/wiki/WebSub

onli• 2 years ago

I think that's the right answer. At least for server based readers it seemed to be supported last I checked, and it's the most efficient way to solve the feed deliver with minimal resources on the server.

veeti2• 2 years ago

I requested the feed in my browser _twice_ and already received a 429 with a Retry-After of 24 hours! Not only is this easy to trip by accident, the problem for a feed reader is that it can't know when the author publishes new posts. If a reader happens to scrape the blog 5 minutes before a new entry is posted, it would then take over 23 hours for it to update.

There are surely a lot of badly behaved clients out there and I sympathize with that, but the author's policies seem rather heavy handed and counterproductive to providing an open feed. God forbid their most dedicated readers receive the most timely updates.

masklinn2• 2 years ago

If a reader happens to scrape the blog 5 minutes before a new entry is posted, it would then take over 23 hours for it to update.

Given the update frequency and kind of content it does not seem much of an issue does it?

the author's policies seem rather heavy handed and counterproductive to providing an open feed

It’s not though? “Open feed” is just that, it’s not a license to be a resource hog because you refuse to be a good citizen.

God forbid their most dedicated readers receive the most timely updates.

Rachel does not post day to day life updates. “Most timely updates” is not a thing, it’s not a storm reporting feed. Latency is not a concern.

quectophoton2• 2 years ago

I requested the feed in my browser _twice_ and already received a 429 with a Retry-After of 24 hours!

My guess: No `Cache-Control` response header.

checks

Yup, no `Cache-Control` response header. So she doesn't want more than one request per day, but also doesn't want to add this header to the responses:

    Cache-Control: public, max-age=86400, stale-if-error=86400

pavel_lishin• 2 years ago

the problem for a feed reader is that it can't know when the author publishes new posts.

It can certainly guess, based on that retry-after header, that she doesn't post more than once a day, and should respect that.

noncoml• 2 years ago

In general it’s not a good argument, as the 10 min polling interval technically controls the latency and has very little to do with the posting frequency or interval between posts.

masklinn2• 2 years ago

As far as I’m concerned it makes more sense if it’s checking even more stuff.

If I have 500 feeds in my reader, most of whom update weekly at most, I would assume the reader does something smarter that mindlessly launch 500 requests every 5 minutes.

vsnf• 2 years ago

I would assume the reader does something smarter that mindlessly launch 500 requests every 5 minutes.

Aside from respecting any 429 headers, what is a reader supposed to do without custom update heuristics for each feed that would be considered more than mindless?

boricj6• 2 years ago

On my blog, my feed.xml is currently 62654 bytes compressed, 307726 bytes long uncompressed. The compressed version is served with gzip_static, so if the client sets the HTTP header "Accept-Encoding: gzip" then it's sent at no additional CPU cost to me.

Looking at my access logs, some RSS readers are well-behaved and get served 304s. Some of them however not only request feed.xml without caching, but they don't even set the HTTP header "Accept-Encoding: gzip". They download feed.xml every time and they download it uncompressed.

I'm hosting my blog on a feeble 5-year old Synology NAS located in my apartment. It's not seeing a lot of traffic so it's currently not a problem, but these RSS readers are wasting several orders of magnitude more of network bandwidth and CPU resources (gotta encrypt it for HTTPS) on my end than well-behaved RSS readers.

It's just plain embarrassing.

kevincox3• 2 years ago

Exactly, unless she expects every reader to configure the polling interval for her blog this isn't a good solution.

I would highly recommend she sets `Cache-Control: max-age=3600` or something similar. That is at least a method that can automatically be applied. Sure, right now most readers just use fixed intervals, but further support for this is a standard that can be pushed slowly. Plus some readers probably use HTTP libraries that would automatically cache the response so some clients will accidentally have support.

vallode2• 2 years ago

I'm not sure how I feel about the content of this entire post. RSS feeds are a more or less stagnant technology, mostly adopted "for fun" or by a niche of people who find them useful. The only way I can see to move forward is to make them as easy and painless to use as possible, the onus falls on both the creator and consumer of feeds (the websites and the clients... the user is completely out of the equation here in my opinion).

This kind of attitude reeks of the "you're using it wrong!" of the Linux world. Are you really telling me you are getting enough RSS feed requests to put a dent on your tech stack? Is your bandwidth overhead suffering that much (are you not caching?)? Make it painless and let's be thankful it isn't all web scrapers masquerading as users.

Mind-boggling problem to be angry about.

cess113• 2 years ago

It's like pissing in the street. No one gets hurt and the street isn't going to break, but there will be a smell and it's perceived as a rather rude behavior except in the case of animals and small children.

cess113• 2 years ago

The downvotes remind me of a thing at a job. We had an API for programmatic access to our customer's data, and one customer had bought some expensive BI-solution that they wanted to feed with data from the system we provided. For months they came to us and complained that the API was broken and asked us to fix it.

When I looked in the logs I could see that they hit the API with a lot of requests in short succession that generated 403 responses, so we said that they need to look at how they authorise. After a while they returned and claimed our API was broken. Eventually I offered to look at their code and they were like 'yeah sure but your silly little PHP dev won't understand our perfect C# application'.

So I looked at it and it was a mess of auto-concurrent, nested, for-loops. If they had gotten any data out it would have been an explosion of tens of thousands of requests within a few seconds. They also didn't understand the Bearer-scheme and just threw the string they got from the auth-endpoint straight into the Authorization-header without any prefix. Maybe we should have answered with 400 instead of 403, but yeah, that would have been a breaking change and we didn't want or have time to do a new API version because of this.

Anyway, their tech-manager got really mad that I found the issue within an hour that they had struggled with for months, and also had mentioned that the API-adapter was designed for DoS rather than a polite API-consumer and maybe they should rewrite it to be less greedy and maybe also use ranges instead of crapping out a new request per row in in a response and stitching it together again on their end.

A few weeks later they got it running and it was brutal, but our machines could take the load so we didn't think more about it. Later I heard they got performance issues on their end and had to do as I had suggested anyway.

Be polite and pay attention to detail when you integrate with protocols and API:s. At best you'll be a nuisance if you don't, but many will just block you permanently.

troupo2• 2 years ago

It's a bigger issue than RSS, really. Fetching RSS is a simple GET request. It requires the most basic understanding of HTTP, and people still can't do it right: they don't know how to deal with standard headers, how to deal with standard response codes, how to send proper requests etc.

Do you think regular REST API calls to any other service are any different?

vallode2• 2 years ago

Very valid point, the frustrations here do share commonalities with the overall HTTP ecosystem.

mimischi• 2 years ago

Any pointers for good resources to grok best practices?

hennell• 2 years ago

So you could be angry at web scrapers, but not at RSS readers making badly formed requests every 5 minutes?

Making it painless is letting readers use the options available, supporting the commonly used standards and maybe fixing small problems (I'd probably space trim that URL for example). It doesn't mean supporting systems that are sending poorly formed or badly controlled requests, regardless of the impact it has on your tech stack.

michaelt• 2 years ago

Are you really telling me you are getting enough RSS feed requests to put a dent on your tech stack?

Both RSS feed providers and reader maintainers believe that RSS isn't dead, it's just pining for the fjords.

They've got to keep their readers efficient ready for RSS to rise again - much like Christians have to resist the temptation of sin ready for the second coming of Christ.

boricj6• 2 years ago

Some of us are not serving our blogs through Cloudfare. In fact, I'm using a 5 year old entry-level Synology NAS located in my apartment to serve mine. I do that because it's already always-on by policy and therefore doesn't cost me anything extra to serve my blog from there, besides the DNS domain name.

Badly behaved RSS readers that download feeds uncached are wasting orders of magnitude more of bandwidth and CPU (gotta encrypt it for HTTPS) on my end than well-behaved clients that get served 304s. Some of them don't even set "Accept-Encoding: gzip" and download it uncompressed. That's hundreds of kilobytes per request wasted for nothing.

My blog doesn't see enough traffic to make this an issue for me, but I can see why this could be a real problem for popular blogs with loads of traffic.

madeofpalk4• 2 years ago

I think it's generally a poor idea to serve a blog, especially a popular one with loads of traffic, from an entry-level Synology NAS.

mdp20213• 2 years ago

RSS feeds [would be] a more or less stagnant technology, mostly adopted "for fun" or by a niche of people who find them useful

Pray tell, what would be the good alternative, for this «niche of people» who collect the news? We use RSS because news are collected through RSS.

And clients have to be properly, wisely configured: some publish monthly some much more than hourly; some are a feed per domain, others are thousands of feeds per domain... This constitutes a set-up problem.

8organicbits• 2 years ago

RSS is the best option if you are looking to avoid walled gardens. It's a great way to find content without search ads or social media ads. RSS readers put you in control, instead of algorithmic social network feeds that manipulate you into doom scrolling. I think RSS has been growing as the social networks enshittify.

cqqxo4zV46cp4• 2 years ago

This is delusional. Are you not able to identify when a thing that you do isn’t very widely done?

mariusor2• 2 years ago

I see very little in the article that's actually targeted at people that hold it wrong. My interpretation is that the rant is targeted at RSS service developers who should know better, and for whom you are inventing excuses to justify laziness or incompetence.

geor9e• 2 years ago

Conspiracy theory (that I genuinely believe): Consumers don't know what they want, and in reality they would love RSS if it was allowed to blossom. But, RSS readers make the web unmonetizable, so they've been actively destroyed from every angle, in favor of the enshittified feeds the world is now addicted to. There are so many ways to quietly bury a certain technology if the market incentives are stong enough. You can't algorithmically manipulate and addict someone who just follows accounts chronologically without ads. So silicon valley market forces tend to discourage RSS. Google killed their RSS reader once they realized browser-based feed browsing generated way more adsense profits. Other readers get VC investment and mysteriously their free version becomes unusable garbage and adoption plateaus. Not every controlling interest in a VCs portfolio is benevolent. Facebook, twitter, etc all make it against their terms of service to "scrape" your own friends privately shared posts - you can only see them via the walled garden of ads. Apples app store fees would drop if consumers understood the utopia an RSS based web has the potential of being, instead of a dozen addictive apps. But it was too free. RSS derails trillion dollar roadmaps. The incentives are clear, and silicon valley knows it.

Zecc• 2 years ago

The irony of the situation is this post might incentivize new people to subscribe to her feed. I know I'm one of them.

codetrotter• 2 years ago

Hopefully taking the time to investigate how the reader works and pick one that behaves nicely.

FinnKuhn2• 2 years ago

Is there a easy way to see how your reader works?

ghusbands• 2 years ago

There's another one which hits every 2 minutes without fail [...] it's the one feed user-agent I've had to block outright. It doesn't stop requesting things [...] Clearly, I need to use a bigger hammer.

I'd be tempted to go the opposite direction and give it articles that seem new, every time, called "Your feed reader is very broken and you should use a different one".

boricj6• 2 years ago

Just give it a custom feed with only one article titled "Your feed reader sucks" explaining why they get this. Maybe even generate the contents of this article dynamically to provide diagnostic messages.

That way, badly-behaved clients can't use their broken RSS to receive actual articles until they fix it and they get to know why exactly.

wolpoli• 2 years ago

I was surprised to hear that they got authors of RSS readers to get in touch and fix issues. Last time I looked for an RSS reader for Windows, it was a graveyard of abandoned software.

manuelmoreale• 2 years ago

Plenty of very actively maintained projects at least on the Apple side of things.

People are absolutely still using RSS, the tech is very much not dead.

orf3• 2 years ago

So much effort for such specific pedantry.

I just added her feed to a third party proxy service. If the owner of the site doesn’t want to serve a very small and basically static text file in a way people expect then that is their prerogative.

But having to run their feed through a proxy to work around this doesn’t seem to be something they would like to encourage.

simoncion4• 2 years ago

If the owner of the site doesn’t want to serve a very small and basically static text file in a way people expect...

I'm an old programmer and sysadmin. Programmers and sysadmins are definitely the "target audience" for this blog.

I expect that if I request a resource on the web, and I get back a 429 with a "Retry-After" then:

- I WILL NOT get a new version of that resource until after that time has elapsed

- I MAY get banned for an unknowable-by-me period of time if I continue to attempt to request that resource, despite getting a 429.

orf3• 2 years ago

Unfortunately with stuff like this Hyrum’s Law happens at a distance and in aggregate. And 429’s are clearly not part of that living contract.

The reality is that it’s not much effort to serve a small bit of plaintext data, and nobody really cares that much to even notice.

It’s unclear if the effort is worth it, for anything other than what amounts to “well ackchully the spec says I can do it so I’ll do it” pedantry.

oliwarner6• 2 years ago

I love RSS but this approach worries too much.

It's static content, at least as static as the rest of your content, so write once and let one of a dozen CDNs cache it for you for free. They will do a good job of setting cache headers for you. If people want to ignore them and honk on it from their crappy readers, that's on them.

Even if you're personally serving it from your kitchen toaster, it's static content. You need a significant number of bad actors for this to be a problem.

Moreover, if this post is anything to go by, treating a bad reader with a logical hurdle and expecting a sane result is baffling.

It's not worth this much headspace.

kkfx• 2 years ago

Are you sure it's static? How many regularly update the same article after having published it, just for instance?

soapdog• 2 years ago

Exactly. Mine is not static for example, but I’m a low traffic blog and it is not a problem yet.

oliwarner6• 2 years ago

I do. But that still falls well into the umbrella of WORM for me.

In practice, my SSG generates my RSS at the same time as my website and it's all running under Cloudflare Pages so it's just deployed and I don't have to think about any of it past the git commit.

I think that's what I'm trying to get at here. If you're dynamically generating RSS for a dynamic blog today, you've made some significant design errors.

kevincox3• 2 years ago

It's "mostly static". I just tell my CDN to cache HTML for 10min. This makes the origin server load trivial but still makes updates go live fairly quickly.

rakoo4• 2 years ago

I don't like this approach: it means that instead of fixing the problem, we just put the problem under the rug and let a third-party centralize always more of our communications, giving them absurd power over our infrastructure. That's not an engineering solution, that's a politician solution.

oliwarner6• 2 years ago

The reason I tolerate the power over our infrastructure is that it's completely escapable.

Building and caching my blog can be done well and freely by a dozen services, and it can easily fall back to me on my server if all else fails.

If we were talking about edge functions and proprietary integrations, that might be a more significant issue, but static websites are insanely easy to push off to CDN services.

NoboruWataya2• 2 years ago

Granted I do not have any blog feed, much less a reasonably popular one like Rachel by the Bay, but I would have thought that serving simple static content like an RSS feed is quite low-cost such that it almost doesn't matter how much people are querying, no? I'd be quite surprised if the author's server was genuinely struggling under the weight of these requests but happy to be corrected.

boricj6• 2 years ago

Rachel by the Bay's atom.xml file is about 544 KiB (which for some reason seems to be always sent uncompressed regardless of the "Accept-Encoding" header, but I digress). It's by far the most requested file, with every subscriber's feed reader polling it periodically. For example, Feeder's default polling frequency is ten minutes, but some might poll more frequently.

That file is served over HTTPS, so every response must be encrypted individually. If it's cached by the client, that's a 304 and a couple hundred bytes of response headers to encrypt. If it's not, that's a 200 and a couple hundred thousand bytes to encrypt.

Now imagine if a garbage reader doesn't cache the feed and requests the full file every time, like Tiny Tiny RSS tends to do in my access logs. Multiply that by the number of subscribers using these busted readers. Multiply that by the number of requests they do each day.

That's a whole lot of load on the server's CPU wasted for nothing.

rfl890• 2 years ago

Serving a kilobytes static file over HTTPS at around ~1k rps shouldn't be too hard for a modern server in the year of our lord 2024 right?

renegat0x02• 2 years ago

The Internet is dying. Everything shifts from standards toward managed corporate walled gardens. There is no place for RSS in the future. This is my personal opinion.

I use only RSS. That is how I obtain new information. I do not know personally anyone else doing that. I ping sources every hour, but I ping at least 400 sources.

From my sources none provided last-modified in headers: reddit, youtube, personal sites (In ff f12, network). My site supports it. It is nice also that you support it, but I doubt there is any impact of that in real world. Most of the attention goes through tiktok,youtube videos, through chrome browser. Either it is supported now, or it doesn't really matter if 40 dudes makes request every minute or so.

We should also provide clean title, description, in open graph protocol meta data, and yet not everybody does that.

We should also return correct HTTP status codes, and yet not everybody does that.

I am disenchanted with current state of the Internet, or maybe it was always a little bit pile of various things/garbage.

0x4454423• 2 years ago

+1 for shared sentiments. Do you have a blog post(s) that explains your setup with the processes you’ve described?

renegat0x02• 2 years ago

I do not have any blog entry worth sharing. I am running an "ethic web scraper". I think I cannot speak about processes. I just may be lacking knowledge. I think it is more about "experience" rather than process.

Web crawling core is in file: https://github.com/rumca-js/Django-link-archive/blob/main/rs...

Some things more project specific are in https://github.com/rumca-js/Django-link-archive/blob/main/rs...

I know that there already are spiders, metadata processing packages for python, but I like having control over the process.

Old man yelling at the cloud. I hate also:

- blocking me with 403 because my user agent is not "mainstream". Why do I have to use chrome undetected to read some RSS feeds? Why can't I use third party clients? Contents can have adverts. I just want my own layout, buttons

- RSS feeds protected with cloudflare, so tools cannot read feeds easily

- not using, or outright blocking RSS functionality in wordpress. Some sites could be more open that way, but no. RSS feeds are closed/removed

- some sites have "/blog" location, but the main domain is empty, or nearly empty, or returns 404. Can I trust such location?

- when HTML meta data are not available. I like YouTube. It allows me to scrape metadata, but it protects video contents, and that is good

- weird redirects. Domain does not have any contents. Does not describe what it is. It just have javascript redirects. From main domain to some weird locations within the domain

- url shorteners, vanity links. You do not know where you will be transported. I understand they are counting sheep, but they sacrifice my security

- google returning links with syntax "https://www.google.com/url", not directly. Youtube does the same with syntax "https://www.youtube.com/redirect". For me again this is vulnerability

My ethic web scraper results are placed in: https://github.com/rumca-js/Internet-Places-Database.

mrighele• 2 years ago

You cannot hope that consumers are well-behaved on the Internet, it has never been the case unfortunately, you have to act defensively.

If the issue is the processing power because the content served is dynamically generated, cache the content. Even with a server running on ESP32, just a few second should be enough (if the rest of the website to be statically generated)

If the issue is the bandwidth you may focus on big content (i.e. media) and return a 429 or use a CDN just for that (not worth for a feed in my opinion)

If the issue is neither, why bother ? Your time is precious, and you should spend it on more enjoyable things (that is, unless you find doing this enjoyable of course).

rasz• 2 years ago

I did that way back then because browsers used to care about RSS and Atom, and they'd put that little yellow feed icon somewhere in the top bar when they spotted this sort of thing in a page. At least in the case of Firefox, you could click on it, and it would throw the target URL to a helper of your choice.

Still here in Vivaldi.

daft_pink• 2 years ago

It’s cause Aaron Swartz isn’t around to fix RSS.

xyst• 2 years ago

On the topic of RSS feeds. I really do miss this way of consuming news.

I tried using a few RSS feeds from major news organizations (WaPo, NYT). All I get are the headlines and a link to the article. I am not sure if it has always been this way or not.

On the other hand, “rachelbythebay” blog posts can be viewed entirely in the RSS reader.

I am using “feedly” on iOS. Downside I see is that it does not properly display all of the formatting elements. Some examples:

- Bullets represented as just *.

- The “code” sections sort of sit awkwardly in the reader. No formatting applied.

- ~~In some of the code sections, it’s just awkwardly replaced by ellipses (…)~~ nvm, this is the authors writing style :)

Maybe that’s just a specific RSS reader issue?

Rudism• 2 years ago

I think the answer to your first point is easily explained by the fact that the major news orgs typically have paywalls that check the user's IP, cookies, and so forth to decide whether they're allowed to see the full article for free or if they have to pony up first. There's no way to really incorporate that functionality into RSS feeds so they're just pushing out the bare minimum as a Hail Mary to drive their dwindling RSS user base back to the website. Smaller personal blogs that don't rely on advertising or paid membership revenue to stay afloat are far more likely to provide full article content on their RSS feeds (since there's no reason not to).

Some feed readers have "scraping" support where they request each article's URL internally and figure out how (or can be configured with CSS selectors) to extract the article's text content and display it in the reader, though it can be pretty hit or miss whether that can work for a given site.

As for formatting, there are a couple possible explanations--the reader could be ignoring or mangling HTML tags in the content when it's displaying it, or the site itself may be generating its RSS feeds with mangled or missing formatting elements. In my experience both possibilities are equally likely.

kevincox3• 2 years ago

I checked what my feed reader does. It seems that it typically gets 304s (I send both If-None-Match and If-Modified-Since) but still about half of the requests get 429s.

So the request pattern basically looks like:

1. Get 304.

2. Wait 5min.

3. Get 429.

4. Error backoff evenly distributed between 30-60min

5. Go back to step 1.

So I guess 5min is "too fast" even for conditional requests. However the reason my reader picks 5min is:

1. There is no caching headers to suggest the author's preference.

2. Conditional requests are supported.

3. The feed is fast.

4. The feed is popular.

5. The feed is active (posts every week or so)

Sure, for her feed it could probably be lower but how is a robot supposed to know that? I would highly recommend setting a cache header. That gives an automatic signal of when the last fetch should be considered stale. I'll admit that many feed readers just use a fixed schedule, but many will still use a library that accidentally caches requests (maybe the browser extension readers that she was complaining about would?). For my feed reader we won't poll more often than your cache header (Cache-Control or Expires will both work) unless it is more than 24h in which case we will poll no more frequently than daily. If you just post blog posts about it it is never going to change, it is better to start pushing for broader support of an actual protocol that could be implemented.

Another great option is supporting WebSub. It is easy to get set up with a public hub and then readers that support it will poll very rarely. (Mine will poll weekly.)

So many feed readers, so many behaviors

Related Stories