Shutting down our search proxy Leta

https://mullvad.net/en/blog/shutting-down-our-search-proxy-leta

by coldblues • 9 months ago

7 3 9 months ago

n1xis10t2• 9 months ago

Sounds to me like they were having trouble scraping Google, and didn’t want to allocate the man power to stay on top of it.

The main problem is that Google desperately wants you to run their JavaScript. They implemented a JS requirement a few months ago, but you could still pretend to be the Lynx web browser and then not be served any JS. Then a little bit ago they removed Lynx support, so that stopped working. There were a few other things you could pretend to be that still worked though, like an old iPod I believe. Then those were blocked as well, but currently there is still at least one user agent string that works.

Some of this I learned about from a proxy called 4get.ca. The guy who runs it is clever and has the energy (for now) to keep the Google scraper working. The caveat though is that he has a somewhat crude and demented sense of humor.

Even if all the user agents are blocked, I worked with him a bit on an alternative bypass which is a bit slow, and is based on my article in the most recent issue of 2600 Magazine (Autumn 2025 if you want to read it). If need be I can probably make it faster. He also has a bunch of Google CSE api keys and makes that available as a user selectable option.

It isn’t a meta-search engine because it doesn’t mix results together from different sources and re-rank them, it’s just a proxy, but as a proxy it’s pretty good.

r721• 9 months ago

Launch thread: https://news.ycombinator.com/item?id=44116503

n1xis10t2• 9 months ago

Thanks!

Shutting down our search proxy Leta

Related Stories