Feedmaker: URL + CSS selectors = RSS feed

https://feedmaker.fly.dev

by mustaphah • 10 months ago

173 30 10 months ago

feedmaker.fly.dev

mustaphah3• 10 months ago

The good news: made it to the front page.

The bad news: so did the 503 page.

benbristow• 10 months ago

In some ways a good thing, no? Shows you've got work to do on optimisation for large audiences. A free stress test (unless you're on a host that charges per hit or bandwidth excess), as you will.

Did load eventually for me, thought it was broken as no styles but looks like it's intentional.

uyzstvqs• 10 months ago

Seems to be hosted using fly.io

kschaul• 10 months ago

Glad you’re find the tool interesting! A short blog post behind it: https://kschaul.com/post/2023/04/16/feedmaker-quickly-genera...

And the GitHub url (hopefully easy to host your own instance): https://github.com/kevinschaul/feedmaker

mustaphah3• 10 months ago

Looks like you're hosting this on fly.io - PAYG model. You could probably host this for free on Cloudflare Workers; 100k requests/day on the free tier; static content (the homepage) is free & unlimited.

Edit: The catch is the 10ms CPU cap per request - you'd need a super lean implementation. Django's too heavy for that.

0cf8612b2e1e• 10 months ago

Python alone is many milliseconds to start. Unless they give you some allowances for interpreter overhead.

mustaphah3• 10 months ago

Well, someone already did with JS: https://github.com/ProfessorManhattan/rss-worker

bradbeattie• 10 months ago

https://github.com/RSS-Bridge/rss-bridge is what I've been using for the same purpose.

zekenie• 10 months ago

Not the same but this gives me an idea… what if there was a map reduce for doms as a web primitive. Like imagine if I could make a dom (or feed) that was some selection and transformation of another dom

onedognight• 10 months ago

You have just re-invented XLST.

pimlottc• 10 months ago

*XSLT

Pfeil• 10 months ago

Related discussion to remove XSLT from the web platform: https://github.com/whatwg/html/issues/11523

1-more• 10 months ago

https://www.w3schools.com/xml/tryxslt.asp?xmlfile=cdcatalog&... give it a whirl!

jeroenhd• 10 months ago

In the days of XHTML, XSLT would be exactly what. The modern web equivalent would require XSLT but for HTML/SGML (SGSLT?).

int0x29• 10 months ago

I made a CGI program that ran CSS selectors against URLs and returned the output. I debated making it public and then realized I probably didn't want to run an open proxy. I'm curious how long this will last.

oneeyedpigeon• 10 months ago

You could just have made it public and self-hosted.

crazygringo2• 10 months ago

I love this.

Has anyone tested to see if it works with Blogtrottr which will email you whenever there's a new item in an RSS feed?

Just since this doesn't seem like it even includes a date field in the RSS? And of course no guid. So I'm wondering how compatible it winds up being.

kevincox2• 10 months ago

Dates shouldn't matter. The feed has ID elements which is what identify entries. Atom has no guid element. So I would expect this to work with any reader.

edoceo• 10 months ago

I wish they had concrete, accurate id and created_at. IIRC these attributes are fixed in AT.

crazygringo2• 10 months ago

But is this producing ID elements? And if so, based on what, since they don't seem to be coming from any CSS selectors? That's my question.

ZYbCRq22HbJ2y7• 10 months ago

Should be able to achieve this without selectors with HTML to Markdownish (something like Firefox's Reader mode).

cleartext412• 10 months ago

Oh, so this is what Reader mode does.

The few times I actually tried it, it worked badly, with huge chunks of text content missing from the page. Makes me wonder if with modern web the task has became so difficult even a browser couldn't pull it off, or if they just wasn't trying to do a good job with the feature.

mg• 10 months ago

That is a good idea.

59 requirements, including Django, seems pretty heavy though?

For my own RSS feed, I use this 48 line Python file with no dependencies outside the standard library:

https://github.com/no-gravity/atomfeed.py

It takes an array with the entries as input, not a web page. But I guess the HTML parsing should take no more than another few lines? For HTML parsing, I have good experiences with the lxml module which is in the Debian repos. It is fast and works pretty well.

oxalorg• 10 months ago

I recently added the python-feedgen module for creating feeds in my blog generator: https://github.com/oxalorg/genox/commit/3a73013ffe82930b1a7e...

I always love removing dependencies and simplifying software. I will try and switch to a simpler implementation like yours, thanks for sharing!

ulrischa• 10 months ago

Same can be done wirh freshrss

msephton• 10 months ago

If you're serious about RSS curation and reading, FreshRSS really is the Swiss Army Knife. It does so much, including this“any site/page to RSS”. My favorite feature is that it makes refreshes in your feed reader client pretty much instant, which is such a huge quality of life improvement.

gottlobflegel• 10 months ago

You can just use an XSLT stylesheet like this: https://wwwcip.cs.fau.de/~oc45ujef/misc/src/atom.xsl xsltproc includes a handy --html flag that lets you just process the source file directly.

pkal• 10 months ago

Can you also generate+use the XSLT stylesheet dynamically from a form input so that you can use a single meta-stylesheet for multiple sites?

Oh, and is you brother coming to the party?

danielheath• 10 months ago

I wrote a similar thing in go (using chromedriver, so it could handle things that need JS).

Handled most things nicely, but I found a few sites where I wanted multiple selections to be combined into one document.

I emailed the result to myself, turning any images into attachments; this meant my “feed reader” had read/unread tracking that synced across devices, some html support, folders, offline viewing, etc.

Feedmaker: URL + CSS selectors = RSS feed

Related Stories