Readability.js

by stefankuehnel • 2 years ago

164 23 2 years ago

zerojames• 2 years ago

I have used and love readability.js. I used it in an application that lets you run various NLP analyses over a web page (surprisals, reading time, word counts, etc.). For that, I needed only the main page content. readability.js retrieves main page content well, consistently.

The Alan Turing Institute maintains a Python wrapper around readability.js, too: https://github.com/alan-turing-institute/ReadabiliPy.

crabmusket• 2 years ago

I had not heard the term "surprisal" before. It's a delightful word. https://en.wikipedia.org/wiki/Information_content

simonw2• 2 years ago

I like using Readability.js as a demo for my shot-scraper CLI utility: https://shot-scraper.datasette.io/en/stable/javascript.html#...

Running this in a terminal (after installing shot-scraper):

    shot-scraper javascript \
      https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/ "
    async () => {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }"

Outputs this:

    {
        "title": "Scraping web pages from the command line with shot-scraper",
        "byline": null,
        "dir": null,
        "lang": "en-gb",
        "content": "... long string of HTML ...",
        "length": 7104,
        "excerpt": "I\u2019ve added a powerful new capability to my shot-scraper command line browser automation tool: you can now use it to load a web page in a headless browser, execute JavaScript \u2026",
        "siteName": null,
        "publishedTime": null
    }

nozzlegear• 2 years ago

This is really cool, thanks for sharing!

stefankuehnel• 2 years ago

Wow, that's really cool! Thanks for sharing. I knew about "shot-scraper" before, but I didn't know you could do something so cool with it.

Zhyl• 2 years ago

Is there a way of using this in conjunction with, say, curl? I'd love to be able to grab clean web pages for offline use and printing.

simonw2• 2 years ago

See my comment here! https://news.ycombinator.com/item?id=39504105

input_sh• 2 years ago

If you run it inside a container, it's fairly simple: https://github.com/phpdocker-io/readability-js-server

oulipo• 2 years ago

Is there a way to use it as a copy-paste in the console, or link to paste in the URL bar, in order to convert a poorly formatted webpage to a nicer one?

johnchristopher• 2 years ago

minScore (number, default 20): the minimum cumulated 'score' used to determine if the document is readerable;

Ahaaa, so that's why sometimes I get the reader icon and sometimes not (especially on mobile).

sabr• 2 years ago

Readability is awesome! I used it to build Smort.io [1] to easily read articles & ArXiv papers.

[1] https://smort.io

harrisonjackson• 2 years ago

This is pretty good for stuffing web scrapings into an LLM context.

Can also use `JSDOM ---> element.textContent` depending on your needs. Useful for snagging all the text content or a specific element's.

flir• 2 years ago

I wrote a browser plugin that does this. Could never figure out how to deal with paged articles though.

shiomiru• 2 years ago

See also the C port here: https://github.com/eafer/rdrview/

It works well with text-mode browsers like w3m.

zyang• 2 years ago

This is the type of work ML should excel at.

noman-land• 2 years ago

Has anyone been able to get this to work on Cloudflare Workers?

tuananh2• 2 years ago

I did this few years back https://github.com/tuananh/reader

also 2nd version use golang and copmile to wasm https://github.com/tuananh/reader2

iansinnott2• 2 years ago

I seem to remember it can be run with JSDom rather than a real DOM. I've never tried with CF workers specifically though.

tuananh2• 2 years ago

I love this feature so much, I built a website to let me share clean text URL with others.

The first one use js lib but it's kinda limited. the second one use Go, compiled to wasm. Both deployed to Cloudflare workers

- https://github.com/tuananh/reader

- https://github.com/tuananh/reader2

iansinnott2• 2 years ago

Excellent library. This is what i'm using to build a full text index of every website I visit [1]. It doesn't work perfectly, but well enough.

[1] https://github.com/iansinnott/full-text-tabs-forever

r-k-jo• 2 years ago

Clipper.js is built on top of Mozilla's Readability library, Turndown to convert HTML to Markdown https://github.com/philschmid/clipper.js

bluish29• 2 years ago

That would be a good chance to mention postlight reader [1] which is a browser extension (both Firefox and chrome versions available) that uses Readability to give you a better reading experience. I recommend it

[1] https://reader.postlight.com/

nicman23• 2 years ago

huh that is probably good for ereaders

Readability.js

Related Stories