Cloudproxy – hide your scrapers IP behind the cloud

by todsacerdoti • 5 years ago

192 111 5 years ago

pacifika• 5 years ago

Maybe it’s a better idea to start or subscribe to an ethical scraping initiative with some user agent header that references some kind of policy and verifiable membership id

intev• 5 years ago

I think it's a pretty safe assumption that if everyone started scraping with certain headers almost every website will blanket block those headers.

weird-eye-issue• 5 years ago

Naive

meltedcapacitor3• 5 years ago

That's the existing default: if a service doesn't mind scraping, they need not take any countermeasure (other than simple rate limiting with a clear status code). Any site that does go beyond that is telling data thieves that they're not welcome.

dzhiurgis• 5 years ago

Hiding behind any larger cloud or VPN looks futile. If there's one place to hide scraper nowadays it's Apple's iCloud+ or whatever stupid name they've picked - two proxies to mimic tor.

ameshkov• 5 years ago

It’d be easier to use Cloudflare WARP command line client for the same purpose.

Apple uses Cloudflare for their private relay so there’s a chance that these IPs won’t be blocked.

fxtentacle• 5 years ago

Ironically, Cloudflare is selling both protection against scraping and tools to facilitate it.

EveYoung• 5 years ago

If I remember correctly, Cloudflare WARP forwards your client IP in the header. So it probably won't be useful for scraping.

gibson12• 5 years ago

i tend to use tor network as a proxy which enables me to request a new ip whenever i need

dorianmariefr• 5 years ago

quite slow and often blacklisted (by Cloudflare for example)

gibson12• 5 years ago

with parallelization you can come up to speed - on big machine with several cores may be still cheaper than many cloud instances

of course no option if your 'target' blocks tor ips

aeyes• 5 years ago

Tor has a couple of hundred exit nodes and they are all on public lists, the Tor network is easy to identify and block.

If you are in a position were you need a rotating list of proxy servers in the cloud to scrape, Tor is probably way too slow.

tushar-r• 5 years ago

This is relatively easy to block, given that these companies publish their IP ranges. Similar to the Tor block, blocking "datacenter IP's" simply becomes the norm. This is why you have companies offering "resi's" or residential proxies to bypass such blocks for some years now.

TheFreim3• 5 years ago

I heard a while back, on HN 8 I think, that a lot of residential proxies are computers that have been infected with malware. How true is that?

robjan• 5 years ago

There have been a number of Chrome extensions which monetize by selling your bandwidth to provide this kind of service. Not ethical but probably not a compromise.

0xy• 5 years ago

A significant portion are grey market "ISPs" that purport to sell residential services but actually never do.

They sell these "residential" IPs to Amazon, other ecommerce retailers and shady people for an extreme price.

In the e-commerce world, scraping is necessary to stay in business. Amazon has armies of scrapers constantly monitoring their competitors, and in some cases automatically undercutting price updates.

samjmck• 5 years ago

Are there even ethical or “official” ways to source residential IPs? I thought most of them were from botnet infected IoT devices.

pseudo02• 5 years ago

I've heard of services that offer free proxy bandwidth in exchange for the user acting as a residential proxy node for other users and/or paying clients. It's generally marketed as a way for the user to avoid geoblocks and such. If this is clearly stated to the user upfront rather than buried behind half a dozen dark patterns and fine print, it seems like this business model could be conducted ethically. When it comes to whether the companies in this space are currently acting ethically though, I have serious doubts.

devit• 5 years ago

You can buy a bunch of mobile SIMs and use them.

alephu5• 5 years ago

I worked on a fairly large web-scraping project (around 2 million pages per day) and we used luminati. Amongst other things, they offer genuine residential proxies with user consent.

https://brightdata.com/proxy-types/rotating-residential-ips

abc03• 5 years ago

Packetstream.io buys bandwidth from users and is cheap. I‘m a customer.

_whm1• 5 years ago

This is easy to block since all IPs of these cloud machines will belong to the same ASN.

Advanced scrapers mostly use residential IPs nowadays, but there are some services to detect those too, e.g. https://focsec.com/

holstvoogd2• 5 years ago

If your scraper behaves, perhaps it would not be blocked…

r_singh• 5 years ago

Not really, most popular websites block any requests coming from curl

TheFreim3• 5 years ago

Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?

holstvoogd2• 5 years ago

As someone who has to deal with a lot of bots, bot networks and other weird scraper apps people use: The biggest issue is that most of these tools are not behaved very well. This tool is clearly designed to circumvent protections, rate limits mostly, against scraping that might be essential to keep things running.

They follow links that are explicitly marked as do not follow, they do not even try to limit their rate, they spoof their user agent strings etc etc. These bots cause real problems and cost real money. I do not think that kind of misuse is ethical. In fact, using this tool to circumvent protections can turn your scraping into a DDOS attack, which I do not feel are ethical.

If your bot behaves itself though, public information is public imo. Just don't take down websites, respect rate limits and do not follow 'no-follow' links.

To give an idea of the size of the issue, we have websites for customers that have maybe 5 hits per minute from actual users. Then _suddenly_ you go to 500 hits/minute for a couple of hours, because some bot is trying to scrape a calendar and is now looking for events in 1850 or whatever. (Not the greatest software that these links are still there tbh, but that is out of my control.)

Or another situation, not entirely related, but interesting i think: A few years back for days on end 80% of our total traffic came from random IPs across china & request could be traced through HTTP referrers where the 'user' had apparently opened a page in one province, then traveled to the other side of China and clicked a link 2 hours later.

All these things are relatively easy to mitigate, but that doesn't make it ethical.

wu_187• 5 years ago

Exactly. We'd get customers thats sites would drown in bot traffic because they pulled the same shit, changing user agents, different ips, etc. I had to build custom mod security rules to block the patterns these boys would pull. What's funny is that the bots would have a site where you could control the crawl rates but it's just a placebo. They would crawl even if you requested them to stop.

The issue with bots hosted on AWS or any cloud for that matter is that as a web host you can't just block the IPs because legitimate traffic comes from them in the form of CMS plugins, backups, etc.

thekyle• 5 years ago

Just so you know, adding rel="nofollow" has never been intended to prevent bots from following those links. Even famous crawlers like Bingbot will sometimes follow, and index pages linked to by "nofollow" links.

The only thing that rel="nofollow" does is tell search engines not to use that link in their PageRank computation.

If you do want to block well-behaved crawlers from crawling parts of your site, the proper way to do that is to use robots.txt rules.

twothamendment• 5 years ago

"... because some bot is trying to scrape a calendar and is now looking for events in 1850 or whatever."

Been there, done that - at least on the side of fixing it. Anyone who implements a calendar, don't make pervious and next links that let someone travel time forever.

I've always wondered how much bit traffic costs us - but never actually tried to figure it out. It is a good portion of our traffic - even when we block a lot of it.

redis_mlc• 5 years ago

Is it ethical to scrape things, trying to bypass prevention methods, against the expressed wishes of a site?

No, but many sites only allow Google to index them now.

I don't think that was the original plan for the web.

deanclatworthy• 5 years ago

It doesn't really matter what's ethical and not, or what the site's wish - what matters is what the law says [1]. I don't want my neighbours smoking on the balcony below me, as I can smell their smoke but the law doesn't allow my building to ban it without consensus. Alas...

[1] https://techcrunch.com/2021/06/14/supreme-court-revives-link...

TheFreim3• 5 years ago

I care since I try to be ethical in my actions.

judge20205• 5 years ago

I don't want my neighbours smoking on the balcony below me, as I can smell their smoke but the law doesn't allow my building to ban it without consensus.

That's simply because you're in multi-family housing, it would be a different story if the neighbors smoked and put up a fan to blow the smoke into your yard/window/etc, and that's probably a more apt comparison to bots that literally fork bomb themselves to crawl your site at 500 requests per second.

cortesoft2• 5 years ago

Why can't we talk about ethics? There are a lot of things in this world that are legal, yet I choose not to do them because they are not ethical in my opinion.

We are allowed to talk about ethics separate from the law.

Fiahil• 5 years ago

Absolutely, especially if you want to keep an eye on specific behavior, -for example- sellers on Amazon.

sschueller• 5 years ago

It's public not behind a paywall or login. Just don't hammer the network, that is what most are concerned about as it costs money and time.

thespacecowboy• 5 years ago

I think it depends on the industry. I've worked at a few ecommerce companies and they all either bought data collected with scrapers or had a scraper team. They also paid for bot protection on their website to stop other companies from scraping their data. I have no problem with this, as ultimately consumers get more competitive and usually lower prices. That is assuming that they are scraping a reasonable rate and not sending 500 requests a second.

What does concern me is the other uses that scraping tools have. For example, what's to stop me from writing a bot specifically to fuck with a competitor's analytics and a/b testing?

Havoc• 5 years ago

Not following how this is helpful?

From what I've seen the cloud IPs are the first to get blocked by anti-scraping tech.

e.g. Built an amazon scraping thing at home. Worked fine. Deployed to cloud. Literally first request Amazon goes nope you're a bot

popotamonga• 5 years ago

I have my crawlers running on a 24/7 notebook at my house because of that. No blocks whatever rate i use. Deployed to digital ocean runs for 5 minutes then blocked forever.

trutannus• 5 years ago

Essentially. All you're doing here is making it easy for the target sites to mass block a bunch of scrapers. Running it off your device is generally better.

cpach• 5 years ago

Another idea: Buying an LTE modem/router. That IP address would probably be harder to block.

nullify88• 5 years ago

I agree, it would be impossible for sites to block individual customers behind carrier grade NAT without blocking other customers sharing that NAT endpoint.

Of course, the websites can raise an abuse complaint with the ISP, and they may take further action.

jimmySixDOF2• 5 years ago

In a related note the US Supreme Court just 10 days ago vacated a previous ruling against LinkedIn blocking a Scraper service. So the issue is back to the 9th circuit for a new determination that might, if reversed, change the landscape for Cloudflare and anyone else these operators try to sit behind.

https://www.scotusblog.com/case-files/cases/linkedin-corp-v-...

cortesoft2• 5 years ago

I don’t think the Supreme Court ruled they can’t block them with technical means, just that they can’t sue them if they get around the block.

jimmySixDOF2• 5 years ago

You are basically correct although it seems to hinge on the definition of an authorized use(r) and also they only vacated the appeals court decision so Microsoft will continue with the same case now in the lower court.

There is a good discussion of the more nuanced situation here:

https://news.ycombinator.com/item?id=27510103

superasn• 5 years ago

Even though I only do it for hobby projects, crawling pages is becoming increasingly difficult unless you are a big player like Google or Microsoft with a whitelisted IP range.

I've had some success in scraping lately with a similar project called FlareSolverr(1).

It's purpose it to get you access to sites which won't let you crawl unless you are using a real browser (e.g amazon, instagram). It doesn't hide your IP but uses puppeteer with stealth mode to get you access to otherwise restricted urls.

(1) https://github.com/FlareSolverr/FlareSolverr

temp89645• 5 years ago

Doesn't Amazon have an API? Why everyone wants to scrape Amazon? Honest question.

helsinkiandrew• 5 years ago

Yes, all their product data is available for affiliates: https://webservices.amazon.com/paapi5/documentation/

mkl953• 5 years ago

Amazon's marketplace APIs are available to developers who register through a seller account. Pricing is $40 per month.

The main benefit of using the API is that you can request a LOT of data without hitting their rate limit. Unless you need to get dozens of results per second, you are usually better off with a spider (or just use Huginn). And if you are hit with a 503 and a captcha, gluing a free captcha solver with some middleware is a trivial task.

avipars• 5 years ago

Amazon has one for their affiliates, you need to be approved and get 3 sales attributed to you in order to apply.

Dunedan• 5 years ago

For one pet project I had to crawl a rather popular site. While that worked in general I would frequently get internal server errors. Turns out that this was the response of their CDN when it detected bot-like behavior. That left me wondering how they prevent search engine crawlers from being detected as bots and getting throttled this way as well. Turns out they just check the user agent for that. As soon as I put "Googlebot" in my user agent the frequent errors vanished. So sometimes it's not about using the right IP addresses, but just the right keywords in your user agent. ;-)

beebeepka• 5 years ago

Changing user agent to Google it or whatever also lets you past quite a few paywalls too. Or at least it used to. These days I don't even bother

judge20205• 5 years ago

This probably means incompetency on the CDN’s part - Cloudflare has a detection rule for fake googlebots and it checks by doing reverse DNS to see if it’s really Google. Doing this trick is more likely to get your IP marked for spam, at least if you crawl CF sites with it.

https://developers.google.com/search/docs/advanced/crawling/...

iforgotpassword• 5 years ago

There's a bot in an IRC channel I've been on for over a decade that announces the <title> of any link being mentioned in the chan. It's becoming less and less useful as it's running on someone's vps, and a lot of sites behind cloudflare don't yield anything as they're returning the "checking your browser" page to the bot. Then there are pages that are pure javascript a d don't even deliver a title tag, and others try to show a GDPR banner or paywall and thus yield some generic title and not whatever article the link is supposed to show.

Scoundreller5• 5 years ago

I guess it's time for a user-side script that sends the http request through their daily-driver browser to see what it is, but then they're getting their home computer to visit any and every link... Maybe only when it sees cloudflare DNS...

MuffinFlavored• 5 years ago

puppeteer with stealth mode

I think sites like Zillow can detect after your third or fourth interaction that your actions aren't very human like and will prompt you a captcha.

blantonl3• 5 years ago

Unpopular opinion around here, but scraping Web pages is often unethical, and the scourge of many Web developers and infrastructure managers.

For Christ sakes people, develop your own products!

If you have to develop and use a cloud based tool to "get around" businesses blocking your scraping app, maybe reevaluate your business and life choices.

arsome3• 5 years ago

Sure, let me just develop my own monopoly on all real estate information.

blantonl3• 5 years ago

no business is under any obligation to provide you access to their data and services for you to build a business on top of.

If you feel that the owner of data or services maintains an illegal monopoly over that data, work to lobby government to resolve that issue.

It's not your individual place to unilaterally decide that a business owes you anything other than what is described in their terms and conditions of use. Period.

hakube• 5 years ago

Most use cases for web scraping is usually related to data analysis, creating machine learning dataset, etc. Google and other search engine services are scraping the web too.

meltedcapacitor3• 5 years ago

Users are also victims: captcha walls are basically an anti-scrapper countermeasure, which everybody pays for in reduced web usability.

"Proof of work" style countermeasures (where data has to be decoded browser-side in expensive ways that are bearable for a regular user but onerous for a mass scraper) is another externality everybody pays for (the total amount of CPU wasted by regular users is at the system level a total waste).

dmw_ng• 5 years ago

I have a webshare.io subscription for stuff like this, it's something extremely negligible like $3/mo. for 100 addresses and some liberal bandwidth cap. Always happy to self-build, but sometimes it just doesn't make any sense

ericcholis• 5 years ago

https://www.scrapingbee.com/ does this as a service + a headless browser.

stephen123• 5 years ago

I proxy through an aws lambda. Works well for my lowish rate project. Does anyone else do this?

whalesalad• 5 years ago

I’ve wanted to build something similar with Lambda. I ran an experiment recently and there is pretty good distribution of IP space on AWS Lambda… you could fan out to 20+ pretty easily.

dawnerd• 5 years ago

I’ve been using lambda to scrape an api that technically is only available in their app. The api uses aws so running the scraper on lamda seems to have hidden it well enough. Doubt they want to start blocking amazon ips.

banana_giraffe• 5 years ago

I indirectly worked on a project for a company trying to create an in house web indexer to track user generated content to find trends. Like 80% of the work was making it look like the web scrapers weren't coming from an IP in the cloud.

The IPs for the different cloud providers are super well known, and the big guys either put you behind captcha hell, or just flat out block you if you're coming from one of them.

raxi• 5 years ago

When you order/cancel DigitalOcean instances, the IPs of new instances will be reused from recently cancelled.

So, you will not get many IPs this way.

judge20205• 5 years ago

I wonder if there's some internal APIs the cloud providers share to let them know when customers release ownership of an IP so that its reputation can be reset.

gizdan• 5 years ago

Seems more economical to attach multiple IPs to a single instance and route traffic accordingly as opposed to spinning up multiple instances.

aaadult• 5 years ago

A similar approach I use is google cloud sdk + ssh socks5 proxy.

Create some preemptible instances on google cloud first, then connect them with commands like "gcloud compute ssh instances-name -- -D localhost:port".

And the last step is to connect scrapper to those proxy ports over localhost.

claffin• 5 years ago

Hi all - I'm the creator of CloudProxy and only just came across this. Appreciate all the comments and stars (over 600 in less than two days). All the points shared are valid, you won't get the same effectiveness as residential IPs and may face issues with the proxies being blocked. That being said this solution, if you find they're not being blocked, is a lot cheaper and very quick. I've used it to scrape some major websites extensively without issue. I created it for my own use but then decided to open source it. Just hope you guys find it useful!

Cloudproxy – hide your scrapers IP behind the cloud

Related Stories