Show HN: I added 12ft.io style unpaywalling to my DNS based unpaywaller

https://github.com/fipso/freenews

by fipso • 3 years ago

3 3 3 years ago

github.com

1vuio0pswjnm7• 3 years ago

The methods used are

// pull page from google cache

//spoof twitter referer

//spoof google bot ua

//spoof google bot datacenter ip (in x-forwarded-for header)

//disable cookies

All this, and more, can be done in haproxy. Smaller, faster than golang. Nice to know that Google cache works for www.ft.com. Another method that worked, e.g., for www.wsj.com, for years was using AMP URL. But WSJ has recently ditched AMP. As a poor substitute I have been using WSJ URLs from podcasts. The advantage of using haproxy versus golang is there's no need to edit and recompile the program when one wants to add a new method. It's possible to set it up so that one only has to add a line to the config file, or add a hostname to a map file. Even more, haproxy lets one reload the configuration without restarting the server.

simonblack• 3 years ago

12ft.io seems to be down for me. Is it still functional?

KomoD• 3 years ago

No, Vercel took it down and stole the domain last I checked.

Show HN: I added 12ft.io style unpaywalling to my DNS based unpaywaller

Related Stories