HNGlance

Top Best New Show

OpenAI scraping Reddit through redlib instances

https://hcrypt.net/2025/06/08/scrapers.html

by udev4096 • 1 year ago

4 3 1 year ago

HC

Related Stories

Loading related stories...

Comments

3

unstablediffusi• 1 year ago

that's not scraping, that's web search.

their scrapers wouldn't identify themselves

gkbrk• 1 year ago

The author used to run a public Arch mirror under mirror.ext4.xyz, so it's not exactly an unknown domain.

Combined with the fact that a lot of their self-hosted stuff, including the Reddit front-ends, are in the Certificate Transparency logs [1], it's not hugely surprising that web crawlers would run into them.

[1] https://crt.sh/?q=ext4.xyz

udev4096• 1 year ago

I am definitely not surprised. It's quite normal, as I stated. What I was trying to say was they are abusing the private frontends to get around legal restrictions of aggressively scraping the web

Source preview

hcrypt.net

HNGlance

© 2024-2026 HN Glance. Story content belongs to its original sources.

Privacy
Terms