Ask HN: How does archive.today bypass paywalls?

https://news.ycombinator.com/item?id=35842575

by puttycat • 3 years ago

39 14 3 years ago

news.ycombinator.com

I can understand how it does it for simple paywalls that can be bypassed using a user agent pretending to be Google.

But how does it work for firmer paywalls with, e.g. NYT?

m348e912• 3 years ago

I wondered the same thing, I just figured they were whitelisted by the paywalled sites but I don't know exactly why. Perhaps archive.today has accounts to these paywalled sites although I am not sure that's the case.

There is another site, 12ft.io, that used to bypass many of these site's paywalls but they got pushback and now longer offer bypass for sites like WSJ and NYT.

tpmx2• 3 years ago

My guess: stolen usernames/passwords. Or actual paid subs.

Bender• 3 years ago

Does that mean they would have to rewrite each site to remove the username specific links and content out of the mirror to keep the origin sites from banning their accounts?

puttycat• 3 years ago

My thoughts exactly. That would require a lot of HTML parsing, plus the risk that any website might watermark the page.

ipaddr• 3 years ago

Identify as google-bot

simpli• 3 years ago

Identifying as Googlebot no longer gives access to most paywalled news.

navjack27• 3 years ago

I mean you could kind of do the same thing locally if you use archivebox. I think it's just the way it scrapes the web page.

stonogo• 3 years ago

I believe the question here is, what is "the way it scrapes"?

guilhas• 3 years ago

I would guess most paywalled sites will allow Bots to scrape content to improve site discoverability. Whether it is Archive.org, Archive.today, Google, Google news...

chmaynard• 3 years ago

Perhaps they subscribe (gasp!) to sites with paywalled content, sign in, and begin scraping.

gsora• 3 years ago

Sometimes the paywall is nothing more than a couple HTML div’s.

You can programmatically remove them and then return the page.

throwaway67743• 3 years ago

Mostly used agents, sites want to be crawled to appear in search results, but from time to time it does actively login to sites (linkedin was one that stopped working, presumably suspended as it likely violates t&c)

nora-puchreiner• 3 years ago

NYT has one of those simple paywalls indeed: https://gitlab.com/magnolia1234/bypass-paywalls-chrome-clean...

Ask HN: How does archive.today bypass paywalls?

Related Stories