A Facebook crawler was making 7M requests per day to my stupid website

https://coding.napolux.com/a-facebook-crawler-was-making-7m-requests-per-day-to-my-stupid-website/

by napolux • 6 years ago

1074 403 6 years ago

coding.napolux.com

spiderfarmer2• 6 years ago

Was it a real FB crawler or a random buggy homemade bot masquerading as one?

Check here: https://developers.facebook.com/docs/sharing/webmasters/craw...

hinkley5• 6 years ago

Someone up to no good using a false flag would not surprise me in the slightest.

"who do people already like to hate? I'll pretend to be them."

napolux30• 6 years ago

Doesn't seem the case https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...

draw_down• 6 years ago

Why is it so hard for people here to believe that Facebook could have a crawler misbehaving? It is all over this thread, even though multiple people are saying they saw the same thing and confirmed the IP belonged to FB.

What gives?

napolux30• 6 years ago

One IP (ipv6) I just got from CloudFlare.

https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a...

namanaggarwal• 6 years ago

Did you check the ip and not just the UA? Does it matches with the one from fb?

napolux30• 6 years ago

Yes, one random ip from cloudflare: https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a...

ablanco• 6 years ago

It would be nice to understand how you came to the conclusion that this was a Facebook bot.

akerro• 6 years ago

You can check list of IP addresses?

huac• 6 years ago

the OP did not (explicitly) check, but you can check if the IP falls into a range allocated to Facebook e.g. https://ipinfo.io/AS32934

napolux30• 6 years ago

Ip addresses.

for example https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a...

k2xl• 6 years ago

This is actually a common tactic from malicious actors - pretend to be other bots to get through to websites. If you reverse lookup the IP address you can see if it part of the facebook network.

UA is very untrustworthy

napolux30• 6 years ago

This is one of the IPs

https://www.ultratools.com/tools/ipv6InfoResult?ipAddress=2a...

napolux30• 6 years ago

Here is another one https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...

dylan604• 6 years ago

Why does FB need a crawler? Their users provide the content for their site. Is there an FB web search engine?

youeseh• 6 years ago

When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.

toomuchtodo• 6 years ago

What happens if that link performs an action upon a GET request?

Edit: Folks, I agree with you all, but I've seen a lot of garbage out there. Just asking the question for the discussion.

systemvoltage3• 6 years ago

This pattern needs to die.

Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise.

Messaging apps have no business in looking up the URL. Just let it pass as a link.

Fuck everything about this and we need to push back on this nonsense.

slig• 6 years ago

And to prevent some URLs to be posted.

laurentdc• 6 years ago

Unrelated, but has anyone written a Chrome/Firefox extension to browse the web sending out Googlebot or Facebook user agent?

I wonder if you can bypass paywalls or see things that aren't generally presented to regular users

spiderfarmer2• 6 years ago

Safari has this built in: https://osxdaily.com/2013/01/16/change-user-agent-chrome-saf...

monkpit• 6 years ago

You chose to mention Safari, but link to an article about Chrome, Firefox, and Safari? Looks like all major browsers (probably Edge too) have this built in.

danemacaulay• 6 years ago

https://github.com/iamadamdev/bypass-paywalls-chrome

Sohcahtoa82• 6 years ago

You wouldn't even need an extension. Both browsers allow you to change your User-agent. An extension just makes it a little easier.

I have a UA switcher on Firefox so I can change my UA in two clicks. One of the built-in UA options is indeed the Googlebot.

Ambroos• 6 years ago

I surfed through a Google network for a bit a few years ago, and it does reveal things. The most common was Drupal and WordPress sites with some malicious code running that would link to online pharmacies on every page, but only to Google user agents / networks, I assume to boost their ranking. I let a bunch of webmasters know, most of them didn't believe me because they didn't see it themselves.

leafo• 6 years ago

Same thing has happened to me: https://twitter.com/moonscript/status/1124888489298808834

The network address range falls under Facebook's ownership, so I don't think it's someone spoofing. I do think it's very possible someone found a way to trigger crawl requests in large quantity. Alternatively, I would not be surprised it's just a bug on facebook's end.

_lqaf• 6 years ago

They've done this before to me, too. First I tried `iptables -j DROP`, which made the machine somewhat usable, but didn't help with the traffic. After trying a few things, I tried `-j TARPIT`, and that appeared to make them back off.

Of course, sample size of 1, etc. It could have been coincidental.

hinkley5• 6 years ago

Tarpits are an underappreciated solution to a pool of bad actors.

You can add artificial wait times to responses, or you can just route all of the 'bad' traffic to one machine, which becomes oversubscribed (be sure to segregate your stats!). All bad actors fighting over the same scraps creates proportional backpressure. Just adding 2 second delays to each request won't necessarily achieve that if multiple user agents are hitting you at once.

justinclift• 6 years ago

Interesting. Implemented DROP for some Cloudflare net blocks a few weeks ago due to lots of weird interrupted TCP connections from them.

That's seemed to be good enough, but will consider the TARPIT option if it turns out to be needed. :)

fomine3• 6 years ago

TIL TARIPT!

zackb• 6 years ago

This was happening to us > 5 years ago. The FB crawlers were taking out our image serving system as we used the og:image thing. What we did was route FB crawler traffic to a separate Auto Scaling Group to keep our users happy while also getting the nice preview image on FB when our content was shared. I can't understate the volume of the FB requests, I can't remember the exact numbers now but it was insane.

avian• 6 years ago

This reminds me of a case years ago when I had a small-ish video file on my server that would cause Google Bot to just keep fetching it in a loop. It wasted a huge amount of bandwidth for me since I only noticed it in a report at the end of the month. I had no way of figuring out what was wrong with it, so I just deleted the file and the bot went away.

crazygringo5• 6 years ago

If you can verify it's actually coming from Facebook's IP address (as many other comments here suggest), you should absolutely get in touch with them, though I'm not sure how. Perhaps their security team. That's a very serious bug that they'd be glad to catch -- if it's affecting you it's surely affecting others.

Otherwise it's a bot/malware/etc. spoofing Facebook and gone wrong, which sucks. And yeah just block it by UA, and hopefully eventually it goes away.

mikece2• 6 years ago

From the sounds of it the difference between Facebook's actual crawler and malware could be hard to define or differentiate, given the circumstances.

napolux30• 6 years ago

It was from their IPs tried to figure out how to get in touch, without success

lun4r6• 6 years ago

We've had the same issue. They were doing huge bursts of tens of thousands of requests in very short time several times a day. The bots didn't identify as FB (used "spoofed" UAs) but were all coming from FB owned netblocks. I've contacted FB about it, but they couldn't figure out why this was happening and didn't solve the problem. I found out that there is an option in the FB Catalog manager that lets FB auto-remove items from the catalog when their destination page is gone. Disabling this option solved the issue.

napolux30• 6 years ago

Thanks man! I'll have a look.

cdown2• 6 years ago

Hey! Facebook engineer here. If you have it, can you send me the User-Agent for these requests? That would definitely help speed up narrowing down what's happening here. If you can provide me the hostname being requested in the Host header, that would be great too.

I just sent you an e-mail, you can also reply to that instead if you prefer not to share those details here. :-)

pera• 6 years ago

I just had an idea: if you control your own name server I believe you could use a BIND view to send all their own traffic to themselves based on the source address.

By the way, if someone discovers how to trigger this issue it would be easy to use it as a DOS pseudo-botnet.

gerdesj2• 6 years ago

Nice idea, I like the thinking. I'll tuck that away for use later. PowerDNS has LUA built in amongst a few other things.

My stack of projects to do is growing at a hell of a rate and I'm not popping them off the stack fast enough.

foobiekr• 6 years ago

if only there was a way to accidentally amplify that.

gorgoiler• 6 years ago

Something about this idea sits uncomfortably with me. I also just had an idea / thought experiment based on your idea.

We think of net neutrality as being for carriers and ISPs, but you could see it applied to a publicly accessible DNS service too. These DNS service providers are just as much part of the core service of the Internet as anyone else. It’s not a huge leap to require that those who operate a publicly accessible DNS service are bound by the same spirit of the regulations: that the infrastructure must not discriminate based on who is using it.

It’s different to operating a discriminatory firewall. DNS is a cacheable public service with bad consequences if poisonous data ends up in the system. Fiddling with DNS like this doesn’t seem like a good idea. Too much weird and bad stuff could go wrong.

Another analogy would be to the use of encryption on amateur radio. It seems like an innocuously good idea, but the radio waves were held open in public trust for public use. If you let them be used for a different (though arguably a more useful purpose) then the resource ends up being degraded.

Also along these lines of thought [begin irony mode]: FCC fines for DNS wildcard abuse / usage.

raverbashing• 6 years ago

You could just probably send an HTTP redirect to their own site, no need to play with DNS for that

mszcz2• 6 years ago

I had a different idea. Maybe you could craft a zip-bomb response. The bot would fetch the small gzipped content and upon extraction discover it was GBs of data? Not sure that's possible here, when responding to a request, but that would surely turn the admins attention to it.

smallgovt• 6 years ago

> The bots didn't identify as FB (used "spoofed" UAs)

That's surprising. What were the spoofed user agents that they used?

We've run into this issue also, but all Facebook bot activity had user agents that contained the string "facebookexternalhit".

rob-olmos• 6 years ago

I've seen these these two user-agents from FB IPs, maybe others:

Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53

Mozilla/5.0 (iPhone; CPU iPhone OS 13_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1

Which also execute javascript in some modified sandbox or something, causing errors, and executing error handlers. Interesting attempt to analyze the crawler here: https://github.com/aFarkas/lazysizes/issues/520#issuecomment...

paulpauper• 6 years ago

twitter does the same thing. it sends a bunch of spoofed US visitors from Korea and Germany and US. The bots are spoofed to make it harder to filter them.

rapind2• 6 years ago

I wonder what the (legitimate?) reason is for them to spoof. Seems intentionally shady. Maybe there's a legit reason we're missing?

stiray• 6 years ago

I dont really understand what is the issue. On my welcome page (while all other urls are impossible to guess) i give browser something that requires a few seconds of cpu at 100% to crunch. And tracking some user action in between, visting tarpitted urls etc. In last few years no bot came through. Why bother with robots.txt, just give them something to break their teeths...

(I would give you the url, but I just dont want ti be visited)

cblades• 6 years ago

I don't think most users would appreciate having a site spike their CPU for a few seconds when they visit...at least I wouldn't.

aetherspawn• 6 years ago

I don’t think the solution is flatlining everyone’s CPUs.

trashburger• 6 years ago

Sounds like terrible UX! Won't browsers give you a "Stop script" prompt if you do that?

xg152• 6 years ago

but I just dont want ti be visited

Maybe you should just take your website offline?

quickthrower22• 6 years ago

How much have you mined?

hombre_fatal2• 6 years ago

The far-fetched "solution" along with the "I don't understand everyone doesn't do <thing that is easy to understand why everyone wouldn't do it>" make it hard for me to believe you're actually doing this. Sounds more like a shower thought, maybe fun weekend project, than something in production.

wheresmycraisin4• 6 years ago

Side question: how did you get in contact with facebook? I've an ad account that was suspended last year and gave up trying to contact them.

missedthecue• 6 years ago

Really? I've never had problems contacting them by email. They're one of the easiest tech companies to talk to.

justusw• 6 years ago

Similar experience, closed ad account because of “suspicious activity”, at least ten support tickets (half closed automatically), four lame apologies (our system says no, sry) and then finally, “there was an error in our system, you’re good to go”

lun4r6• 6 years ago

I have an option to email them, live chat, or have our account manager contact us. I guess if you spend enough on ads per month you are entitled to more support...?

bishalb2• 6 years ago

Try their live chat.

danmurphy• 6 years ago

try contacting their NOC, they _may_ give you a human that can help.

adrianN2• 6 years ago

Set up a robots.txt that disallows Facebook crawlers, sue Facebook if the crawling continues for unauthorized access to computer systems, profit.

albatross• 6 years ago

I believe this only works if it is coming from a corporation and targeted at an individual -_-

justinph• 6 years ago

robots.txt is not a legal document. It is asking nicely, and plenty of crawlers purposefully ignore it.

6c696e7578• 6 years ago

Thank goodness for mod_rewrite, which makes blocking/redirecting traffic on basic things like headers pretty easy.

https://www.usenix.org.uk/content/rewritemap.html

You could of course block upstream by IP, but if you want to send the traffic away from a CPU heavy dynamic page to something static that 2xx's or 301's to https://developers.facebook.com/docs/sharing/webmasters/craw... then this could be the answer.

lun4r6• 6 years ago

Here are some more details from my report to FB:

"My webserver is getting hit with bursts of hundreds of requests from Facebook's IP ranges. Google Analytics also reports these hits and shows them as coming from (mostly) Philippines and Peru, however, IP lookup shows that these IPs belong to Facebook (TFBNET3). The number of these hits during a burst typically exceeds my normal traffic by 200%, putting a lot of stress at our infrastructure, putting our business at risk.

This started happening after the Facebook Support team resolved a problem I reported earlier regarding connecting my Facebook Pixel as a data source to my Catalog. It seems Facebook is sending a bot to fetch information from the page, but does so very aggressively and apparently call other trackers on the page (such as Google Analytics)"

69.171.240.19 - - [13/Aug/2018:11:09:52 +0200] "GET /items/ley3xk/ford-dohc-20-sierra-mondeo-scorpio-luk-set.html HTTP/1.1" 200 15181 "https://www.facebook.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134"

"e.g. IP addresses 173.252.87.* performed 15,211 hits between Aug 14 12:00 and 12:59, followed by 13,946 hits from 31.13.115.*"

"What is also interesting is that the user agents are very diverse. I would expect a Facebook crawler to identify itself with a unique User-Agent header (as suggested by the documentation page mentioned earlier), but instead I see User-Agent strings that belong to many different browsers. E.g. this file contains 53,240 hits from Facebook's IP addresses with User-Agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134"

There are a few Facebook useragents in there, but far less than browser useragents: 7,310 hits: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) 2,869 hits: facebookexternalhit/1.1 1,439 hits: facebookcatalog/1.0 120 hits: facebookexternalua

Surprisingly, there is even a useragent string that mentions Bing: 6,280 hits: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

These IPs don't only fetch the HTML page, but load all the page's resources (images, css, ..) including all third-party trackers (such as Google Analytics). Not only does this put unnecessary stress at our infrastructure, it drives up the usage costs of 3rd party tracking services and renders some of our reports unreliable."

final response from FB: "Thanks for your patience while our team looked into this. They've added measures to reduce the amount of crawler calls made. Further optimizations are being worked on as well, but for now, this issue should be resolved." <- NOT.

liveoneggs• 6 years ago

check these hits for an X-Purpose header or similar

forgotmypw17• 6 years ago

Maybe someone at Facebook is testing a project that didn't work as expected?

hinkley5• 6 years ago

I wish we had better tools for things like this.

What I've heard happen is someone puts a governor on the system, and much later your new big customer starts having errors because your aggregate traffic exceeds capacity planning you did 2 years ago and forgot about.

president• 6 years ago

I just thought of a malicious idea. If I were Amazon or some other cloud provider, I could hammer my customers' web sites with tons of requests to make them pay more for resource usage (network bandwidth, s3 calls, misc per/unit usage etc). It would be hard to trace as well. Wonder if people are already doing that today.

eloff• 6 years ago

The amount of bad will that would generate when, not if, it came to light would dwarf the possible gains. So ethical concerns aside, it would be idiotic.

Sometimes that doesn't stop the pointy haired bosses, because sometimes they are the perfect storm of unethical and moronic.

lun4r6• 6 years ago

And FB bug report: https://developers.facebook.com/support/bugs/189402442061080...

nickysielicki• 6 years ago

It bugs me that they require a login to see a bug report, especially in the case of this bug where those affected aren't necessarily facebook users.

napolux30• 6 years ago

I've stumbled in that too.

sradman• 6 years ago

Interesting. According to the Facebook Crawler link provided by the OP [1], it makes a range request of compressed resources. I wonder if the response from the OP’s PHP/SQLite app isn’t triggering a crawler bug.

Maybe Cloudflare can step up and act as an intermediary to isolate the problem. Isolating DoS attacks is one of their comparative advantages.

[1] https://developers.facebook.com/docs/sharing/webmasters/craw...

JoshMcguigan• 6 years ago

Thanks for sharing. I’d like to hear more about the website described here, it sounds very interesting.

napolux30• 6 years ago

Cant' share, sorry :)

tejas53• 6 years ago

Dude something is super sketchy

napolux30• 6 years ago

I can add I work in SEO (tech side), so the website is "super optimized", but it's not rocket science.

Accacin3• 6 years ago

Super optimized? The site takes longer to load than Hacker News for me.

bberenberg• 6 years ago

Doesn't ignoring headers and taking someones site down fall afoul of the CFAA? Especially given how comments here are showing that this is a recurring issue? Depending on which side of the fence you're on, there could be standing to go after FB for this either for money, or for their lawyers to help set a precedent to limit CFAA further.

meritt• 6 years ago

No. The owner needs send Facebook a cease and desist and block their IPs in order to revoke authorization under the CFAA, and then Facebook would need to continue accessing his website despite explicitly being told not to. That's the precedent set by Craigslist v. 3Taps.

Whether or not the above even holds today (hiQ Labs v. LinkedIn argues it does not) remains to be decided in a current appeal to the Supreme Court.

yupyup541335• 6 years ago

When someone does this to Facebook its malicious and they go to jail. WHen Facebook does it to someone else... "oops".

yupyup541335• 6 years ago

Never forget Aaron Swartz. All he did was send download requests to JSTOR. https://www.youtube.com/watch?v=9vz06QO3UkQ

occamrazor2• 6 years ago

From the point of view of the law, the intent (malicious or benevolent) is often as important as the action itself.

alpb• 6 years ago

Hate to say but this is probably a FB engineer running a test/experiment and not a production crawler taking robots.txt etc into account.

mcguire• 6 years ago

Or malware infecting some FB-internal machine?

x02• 6 years ago

Doubt it. If you're a bad enough dude to get control over Facebook's internal infrastructure, I doubt you'd blow that access by using it to spam random sites.

6510• 6 years ago

How do you monetize a robot?

napolux30• 6 years ago

That's a good question. Even better if Facebook can pay me some money :P

machbio• 6 years ago

dont know whether the facebook bots execute javascript - https://www.rdegges.com/2017/how-to-monetize-your-website-wi...

thephyber4• 6 years ago

Establish contractual damages in a ToS for the site. Prove violation and offender. Take to court and collect damages.

Converting the effort into cash is tough, but the strategy exists.

Project HoneyPot is an API which allows any website to do this for honeypot email addresses which are injected the website, along with a ToS which says:

By continuing to access the Website, You acknowledge and agree that each email address the Website contains has a value not less than US $50 derived from their relative secrecy.[1]

[1] https://www.projecthoneypot.org/terms_of_use.php

the_jeremy2• 6 years ago

Has that ever worked? I can't find any record of judgments one way or the other, on their website or elsewhere.

8organicbits• 6 years ago

If it runs javascript, get it to mine bitcoins for you?

amelius• 6 years ago

What would happen if you created a "rabbithole" for webcrawlers? Would they fall for it?

sgift• 6 years ago

Modern web crawlers usually have mechanisms against rabbit holes, e.g. for Apache Nutch you can configure it should only follow to a certain depth of links from a root, it should only follow a redirect a few times and so on. It's always a trade-off though. If you cut off too aggressive you don't get everything, if you cut off too late you waste resources.

danschumann• 6 years ago

It's an interesting thing to think about, that someone can make a request to your website and you pay for it.

This is not how the mail works, where someone needs to buy a stamp to spam you.

thephyber4• 6 years ago

This is a common pattern in (blocking i/o versus non-blocking i/o) and (sync task versus async task).

The typical analogy I hear for DDoS attacks is if you own a business and a giant group of protesters drives traffic to your store, but no one is interested in buying. They create a backlog to enter the store, they crowd the isles, they pick up your inventory, and the lines and crowds scare off your legitimate visitors/customers.

ortusdux• 6 years ago

My office still gets spam faxes to this day. The paper and toner only add up to a few cents a month, so it's not worth doing anything about.

I knew a realtor that had a sheet of black paper with a few choice expletives written on it that they would send back to spammers. There was an art to taping it into a loop so it would continuously feed. This was a few decades ago when a the spam faxes could cost more than a stamp.

mulmen6• 6 years ago

My desk phone at an old job used to get dialed by a fax machine. Not fun picking that up. I redirected it to a virtual fax line and it turns out it was a local clinic faxing medical records. I faxed them back with some message about you have the wrong number but they never stopped.

mulmen6• 6 years ago

Most of my junk mail does not have stamps because the spammers drop it off in pallets at the post office and just have it delivered to everyone.

The USPS is like a reverse garbage service.

Nextgrid4• 6 years ago

It’s because we as an industry decided to give in and submit to the cloud providers’ bullshit model of paying overpriced amounts for bandwidth while good old bare-metal providers still offer unmetered bandwidth for very reasonable prices.

xmprt2• 6 years ago

The question isn't about pricing. The fact that you pay for your own bandwidth and metal is a problem.

lclarkmichalek• 6 years ago

Make sure you file a bug - there are a myriad of sources internally, but we can often hunt it down easily enough (assuming it gets triaged to eng). Important info is the host of the urls being crawled, and the User Agent attached to the requests (headers are also good too). A timeseries graph of the hits (with date & timezone specified) can also help.

napolux30• 6 years ago

Where? Here? https://developers.facebook.com/support/bugs/

falcolas• 6 years ago

Why is this the owner's problem? Someone at Facebook should be filing the bug, or better yet instrumenting their systems so that incidents like this issue a wake-up-an-engineer alert.

Fuck this culture of "it's up to the victim of our fuckup to file a bug report with us".

nhooyr• 6 years ago

100%

mulmen6• 6 years ago

Not just “file a bug report” but also compile a time series graph (lol) and then pray that Facebook triages it correctly, which they have no incentive to do.

shultays3• 6 years ago

isn't this what robots.txt for? can't you block facebook?

detaro5• 6 years ago

As the article clearly states: no

shultays3• 6 years ago

whoops, my laziness. sorry

DreadY2K• 6 years ago

The article mentions that they tried that and the crawler ignored it.

jokoon• 6 years ago

That's 81 request per second on average.

Shouldn't anybody doing such thing be liable, and be sued for negligence and required to pay damages?

That sounds like a lot of bandwidth (and server stress).

napolux30• 6 years ago

I saw also of 300reqs per second.

Traster• 6 years ago

Yes, this was my first thought too - Facebook is essentially denial of servicing people's sites and it sounds like they're aware of it. Move fast and break your things. If this were your average Joe I'm fairly certain something like the DMCA could be used.

sprayk2• 6 years ago

What? How does the DCMA apply here at all?

jrockway4• 6 years ago

Shouldn't any public facing website have a rate limit? If someone attempts to circumvent simple rate limits (randomizing the source IP or header content), then that could demonstrate intent to cause damage, and you'd have a better case. But if you don't set a limit, how can you be mad that someone exceeded it?

(I know they're ignoring robots.txt, but robots.txt is not a law. And, it doesn't apply to user-generated requests, for things like "link unfurling" in things like Slack. I am guessing the crawler ignores robots.txt because it is doing the request on behalf of a human user, not to create some sort of index. Google is attempting to standardize this widely-understood convention: https://developers.google.com/search/reference/robots_txt)

ricardobeat2• 6 years ago

There is a cost to apply throttling, you still have to receive those requests and filter/block them, plus incoming bandwidth.

The post mentions setting `og:ttl` and replying with HTTP 429 - Too Many Requests, and both being ignored.

dgellow• 6 years ago

The author specified that the crawler ignores 429 status code. So they do have some rate limit.

sammitch• 6 years ago

Plenty of crawlers, including Facebook's, are operating out of huge pools of IPs usually smeared across multiple blocks. If you have an idea on how to ratelimit a crawler doing 2 req/s from 300+ distinct IPs with random spoofed UAs let me know when your startup launches.

toast0• 6 years ago

Rate limiting websites is a lot of work. You have to figure out what kind of limits you should set, you have to figure out if that should be on an IP or a /24 or an ASN, if it's an ASN, you have to figure out how to turn an IP into an ASN.

If your website is pretty fast, you probably won't necessarily care about a lot of hits from one source, but if it's supposed to be a small website on inexpensive hosting, and all those hits add up to real transfer numbers, maybe it's an issue.

quickthrower22• 6 years ago

In the ToS, add a charge of 2 guineas, 5 shilling, a sixpence and a peppercorn per request.

FridgeSeal• 6 years ago

Nah make it a strand of saffron per request and get your moneys' worth out of them.

rcfox• 6 years ago

Did anyone notice the branded IP address? 2a03:2880:20ff:d::face:b00c

"face:b00c"

sprayk2• 6 years ago

if only they could have convinced IANA to give them a vanity prefix.

napolux30• 6 years ago

The mighty power of IPV6 :P

tombrossman• 6 years ago

My "404-pests" fail2ban-client status, which drops everything making *.php requests (This machine has never had PHP installed...):

2a03:2880:10ff:14::face:b00c 2a03:2880:10ff:21::face:b00c 2a03:2880:11ff:1a::face:b00c 2a03:2880:11ff:1f::face:b00c 2a03:2880:11ff:2::face:b00c 2a03:2880:12ff:10::face:b00c 2a03:2880:12ff:1::face:b00c 2a03:2880:12ff:9::face:b00c 2a03:2880:12ff:d::face:b00c 2a03:2880:13ff:3::face:b00c 2a03:2880:13ff:4::face:b00c 2a03:2880:20ff:12::face:b00c 2a03:2880:20ff:1e::face:b00c 2a03:2880:20ff:4::face:b00c 2a03:2880:20ff:5::face:b00c 2a03:2880:20ff:75::face:b00c 2a03:2880:20ff:77::face:b00c 2a03:2880:20ff:e::face:b00c 2a03:2880:21ff:30::face:b00c 2a03:2880:22ff:11::face:b00c 2a03:2880:22ff:12::face:b00c 2a03:2880:22ff:14::face:b00c 2a03:2880:23ff:5::face:b00c 2a03:2880:23ff:b::face:b00c 2a03:2880:23ff:c::face:b00c 2a03:2880:30ff:10::face:b00c 2a03:2880:30ff:11::face:b00c 2a03:2880:30ff:17::face:b00c 2a03:2880:30ff:1::face:b00c 2a03:2880:30ff:71::face:b00c 2a03:2880:30ff:a::face:b00c 2a03:2880:30ff:b::face:b00c 2a03:2880:30ff:c::face:b00c 2a03:2880:30ff:d::face:b00c 2a03:2880:30ff:f::face:b00c 2a03:2880:31ff:10::face:b00c 2a03:2880:31ff:11::face:b00c 2a03:2880:31ff:12::face:b00c 2a03:2880:31ff:13::face:b00c 2a03:2880:31ff:17::face:b00c 2a03:2880:31ff:1::face:b00c 2a03:2880:31ff:2::face:b00c 2a03:2880:31ff:3::face:b00c 2a03:2880:31ff:4::face:b00c 2a03:2880:31ff:5::face:b00c 2a03:2880:31ff:6::face:b00c 2a03:2880:31ff:71::face:b00c 2a03:2880:31ff:7::face:b00c 2a03:2880:31ff:8::face:b00c 2a03:2880:31ff:c::face:b00c 2a03:2880:31ff:d::face:b00c 2a03:2880:31ff:e::face:b00c 2a03:2880:31ff:f::face:b00c 2a03:2880:32ff:4::face:b00c 2a03:2880:32ff:5::face:b00c 2a03:2880:32ff:70::face:b00c 2a03:2880:32ff:d::face:b00c 2a03:2880:ff:16::face:b00c 2a03:2880:ff:17::face:b00c 2a03:2880:ff:1a::face:b00c 2a03:2880:ff:1c::face:b00c 2a03:2880:ff:1d::face:b00c 2a03:2880:ff:25::face:b00c 2a03:2880:ff::face:b00c 2a03:2880:ff:b::face:b00c 2a03:2880:ff:c::face:b00c 2a03:2880:ff:d::face:b00c

napolux30• 6 years ago

I think we're together in this, my friend

cranekam3• 6 years ago

FB has 2a00::/12 (and probably other blocks?) and can make something that looks like its company name from [0-9a-f]. Why wouldn't it do something harmless and fun like this? It's not as if it requires special dispensation or is breaking any rules.

gk1• 6 years ago

I don't think the person was implying it's bad. Just pointing out an easter egg.

detaro5• 6 years ago

A single company does not get a /12 prefix. 2a00::/12 is almost half of the space currently allocated to all of RIPE NCC. Facebook seems to have 2a03:2880::/29 out of that /12, and a /40 through ARIN (2620:0:1c00::/40)

chirau• 6 years ago

I think sending back a `429 Too Many Requests` can solve this. Unless you want to block their IPs completely, which I doubt you'd want to do.

gramakri• 6 years ago

It says in the post he did this already

napolux30• 6 years ago

I tried. It was ignored.

klohto• 6 years ago

HTTP 429 -> ignored

It's in the article

jandrese• 6 years ago

A well made crawler that would support 429 codes isn't likely to have this problem in the first place.

tlrobinson• 6 years ago

And then they’ll probably ask you to pay the bill

Can anyone comment on whether there would be any legal basis for this?

ruuda5• 6 years ago

Hi Napolux,

It looks like your site is using a theme based on my website (https://ruudvanasseldonk.com/, source at https://github.com/ruuda/blog). That is fine — it is open source after all, licensed under the GPLv3. But I can’t find the source code for your site, and I can’t find any prominent notices saying that you modified my source. Could you please add those?

RhodesianHunter• 6 years ago

Despite how it sounds, I ask this with zero judgment and pure curiosity.

Why do you care?

kungato• 6 years ago

Well I respect the question of GP if nothing then for principle. People should get their shit together and start respecting the license. The state today is horrendous and the culture is evolving into less and less respect of the license. If people don't follow them personally they wont care to follow them professionally either

timy2shoes• 6 years ago

Please read https://sfconservancy.org/copyleft-compliance/principles.htm.... My understanding is that if you do not enforce your copyright (or copyleft in this case) you can lose the copyright.

john_alan• 6 years ago

Cause it’s his special.

mpoteat• 6 years ago

I can only guess at motivations, but holistically people need to enforce their license terms or they lose their teeth.

mastermojo• 6 years ago

The author probably cares about either copy-left or the right to tinker. Some people believe it's important and others don't.

Basically any derivative product out of GPL3 code needs to either be open-source/copy-left, or if it uses the code as a library needs to let end users substitute that library for their own version.

ruuda5• 6 years ago

I made my site open source so others who like it can take a look at how it works, or give it their own twist. Visitors of my site can see that they have that freedom, but visitors of an adaptation might not know if it’s not stated anywhere.

playingchanges• 6 years ago

Kind of an odd request for an open source wordpress blog theme imo

ricardobeat2• 6 years ago

They look similar at a glance (border-top + Calluna font), so he might have taken inspiration from yours, but doesn't seem to have used any of your assets - the styles are clearly different and based on the WP 'BlankSlate' theme.

(my personal blog had a top border like that a decade ago, when styles on the body were a novelty :))

napolux30• 6 years ago

I took inspiration but I didn’t know he was sharing the code for is blog

https://github.com/napolux/coding.napolux.com/

ruuda5• 6 years ago

I checked the source and it does indeed mention https://wordpress.org/themes/blankslate/, but judging from the screenshot there, that theme is just really a blank theme with no style at all, and it was used to include a different stylesheet.

The real style is at https://coding.napolux.com/wp-content/themes/coding.napolux...., which looks like normalize.css followed by a Wordpress adaptation of my stylesheet.

The similarity is more than superficial, the footer headers match exactly.

napolux30• 6 years ago

Sure man no problem. The code for my theme is here BTW with credits To your original blog

https://github.com/napolux/coding.napolux.com/

ruuda5• 6 years ago

Thanks, I’m flattered to see it be used as inspiration :)

I searched quickly but I didn’t find that repository. You might want to link it somewhere in your footer or from a comment in the html.

not_a_moth• 6 years ago

Am inexperienced in the technicals of this, is "taking inspiration" from someone else's website grounds for a copyright or gpl violation?

When I've worked with designers it usually starts "pick a site that inspires you", where the deliverable has resemblance.

dubcanada• 6 years ago

It appears to be a serif font and a border top. That's all it is? Am I missing something.

ganstyles• 6 years ago

Design layman here, switching between the two sites the top bar looks similar (it's just a solid line) except a different color and slightly different height. I don't see other similarities or assets coming from your site? Can you explain this request; I'm not sure two sites both using a blank horizontal rule across the top is something I would personally call out as similar.

PetahNZ• 6 years ago

My understanding is unless he's distributing it (not just serving it from a website) he doesnt need to release his changes. That's what the AGPL is for.

ThrustVectoring• 6 years ago

It's not the copyright on the server code that is at issue, but the copyright on the HTML and CSS files (and portions thereof) that get distributed by the server.

dahfizz3• 6 years ago

The actual GPL uses the word "convey", not distribute or serve. [1](section 5). Merriam-Webster defined "convey" as "to transfer or deliver". Pretty hard to argue that a web server does not transfer or deliver HTML + CSS.

[1] https://www.gnu.org/licenses/gpl-3.0.html

johnwheeler2• 6 years ago

Next time, have some courtesy for the author and the rest of us by requesting via personal exchange over email instead of hijacking the thread and distracting from the conversation.

mhh__• 6 years ago

If it were a financial thing I'd agree but if you aren't complying with the GPL I think it's good to set an example (but not shaming)

TheSpiceIsLife4• 6 years ago

Boo hiss.

There’s plenty of room here for a polite exchange between professionals.

Collapse the thread and move on.

Dahoon2• 6 years ago

... and the rest of us ...

Please don't try to police the thread and speak for yourself only.

maxk42• 6 years ago

GPL license states you must provide source code to the end-user. In this case, the end-user is legally Napolux, not the visitors to his site.

userbinator• 6 years ago

In any case, View -> Source

The source code of a public website is automatically provided to everyone who visits it. This fact is unfortunately not well-known.

napolux30• 6 years ago

Hello there. I've added a link to your website in the footer! :)

https://coding.napolux.com/

Thanks for reaching out!

ruuda5• 6 years ago

Awesome, thank you! Sorry for derailing your thread, I did not mean for my comment to escalate like it did.

laughinghan2• 6 years ago

Your blog is awesome! I'm really glad you commented here, or I wouldn't have found it. I even forwarded your most recent post to some friends.

staycoolboy2• 6 years ago

I'm new to webadmining in the cloud and my website is getting hammered by baidu, google, fb, et others causing traffic I/O costs to increase.

What's an AWS LoadBalancer way of blocking this traffic? Again, noob here. THanks.

namibj2• 6 years ago

Don't host this stuff on AWS if you care about cost.

staycoolboy2• 6 years ago

So you don't know how to do this in AWS is what I'm hearing?

pmlnr• 6 years ago

Are you sure it's a crawler and not a proxy of some sort? Eg. one of your links is on something high traffic in facebook, and all requests are human, running through fb machines.

napolux30• 6 years ago

Nah, it's a specific user-agent for their crawler

pier25• 6 years ago

Shouldn't have Cloudflare considered this a DDOS attack?

TheChaplain• 6 years ago

Reading through the article and comments here I find it odd that a company like FB can't get a crawler to respect robots.txt or 429 status codes.

Even I would stop and think "maybe I should put in some guards against large amounts of traffic" when writing a crawler, and I'm certainly not one of those brilliant minds who manage to pass their interview process.

callalex2• 6 years ago

If the dev respected others’ boundaries and norms of expected behavior, they would probably work somewhere else.

tejas53• 6 years ago

Did you reach out to the fb contact?

tejas53• 6 years ago

Do you have some sort of fb plugin in your SEO site?

snorrah• 6 years ago

Surprise, Facebook not caring about playing nice with others!

Anyway, what reason would they have for so many requests? Why would they need to do 300 per second?

nraynaud• 6 years ago

We had that with google in the past. We put a kml database on a web server and made the google maps API read them. Google went to get them without any throttling (my guess is that they primed their edge servers straight from us), the server went down so fast we sought there had been a hardware failure. We ended up putting a cdn just for google servers.

_drimzy• 6 years ago

Why would a crawler want to crawl the same page every second (let alone 71 times a second). Even if I were Google, wouldn't crawling any page every few mins (or say every minute) be way more than enough for freshness?

cascader• 6 years ago

Is it worth duplicating the fb specific parts of your website at another web address as a test site, so that If it was similarly affected, you could provide the address info to help troubleshoot?

itsjloh• 6 years ago

When I enabled IPv6 traffic at home and logged default denies I constantly see Facebook owned IPv6 blocks trying to reach addresses browsing Facebook.

I have no idea what its trying to do but its legitimately the only IPv6 inbound traffic that isn't related to normal browsing.

hank_z2• 6 years ago

I’m a bit confused. Facebook is social media company, why do they send out crawlers?

thenickdude• 6 years ago

Facebook shows previews of pretty much everything you share a link to, so at a minimum they have to fetch the page/image to build those previews.

hank_z2• 6 years ago

I see, thanks for the explanation.

shadykiller• 6 years ago

Then your website isn't stupid but rather the FB crawler

chrismarlow9• 6 years ago

bit rot? 1 off anywhere in a DNS request could slam someone

dan153• 6 years ago

Facebook caches URLs pretty aggressively (often you need to explicitly purge the cache to pick up updates) so I don't quite understand exactly what happened here. The article is very light on details. Do you have 7 million unique URLs that have been shared on Facebook? Or is it the same URL being scraped over and over?

Are you correctly sending caching headers?

drewmol• 6 years ago

Title change? A stupid Facebook crawler was making 7M requests per day to my website

napolux30• 6 years ago

Nice one. ;)

accurrent• 6 years ago

Why does facebook have crawlers in the first place?

omershapira• 6 years ago

I once made a PHP page which fetches a random wikipedia article, but changes the title to "What the fuck is [TOPIC]?". It was incredibly funny, until I started getting angry emails from people, mostly threatening a form of libel lawsuit.

Turns out, since it was a front for all of wikipedia[1], google was agressively indexing it, but the results rarely made it to the first search page. And since this isn't exactly an important site, old results would stick around.

Hence a pattern:

1. Some Rando creates a page about themselves

2. Wikipedia editors, being holy and good, extinguish that nonsense

1.5. GOOGLE INDEXES IT

3. Rando, by nature of being a rando, googles themselves, doesn't find their wikipedia page anymore (that's gone), but does find a link to my site in the first page of google results with their name.

4. Lawyers get involved, somehow

Details: http://omershapira.com/blog/2013/03/randomfax-net/

[1] Hebrew wikipedia. I can't imagine what would've happened on the English version.

smileybarry• 6 years ago

I understand why people got angry, though. The title was rougly "What is this '<article name>' shit?"[1], with "shit" in this Hebrew context also able to be interpreted as calling the subject "shit".

Saying "מה זה לעזאזל X?"[2] is closer to "what the fuck is X?" (except it's more "what the hell" than "what the fuck").

[1] I wasn't sure how to portray this to non-Hebrew speakers but, surprisingly, Google Translate actually nailed it: https://translate.google.com/#view=home&op=translate&sl=auto...

[2] Google Translate got this example right, too: https://translate.google.com/#view=home&op=translate&sl=auto...

Too• 6 years ago

I own a little website I use for some SEO experiments [...] can generate thousands of different pages.

Do you have more details on this? One could read that as if the page is designed to attract crawlers.

keevitaja• 6 years ago

we had a similar experience with yandex. had to block it. crawler started to crawl on a larger e-shop by trying out every single possible filter combination. in other words we got ddos attack from yandex.

kuu• 6 years ago

I wonder what the crawler does to it requires so many requests per day to the same site...

magwa101• 6 years ago

Their 2FA page was broken yesterday...it goes on

historyremade• 6 years ago

Someone could use this for DDOS or Coinhive

zippy786• 6 years ago

There was a very old post: https://news.ycombinator.com/item?id=7649025, lots of ways to abuse Facebook network.

hodgesrm• 6 years ago

This is one of the better HN threads in a while. Starting from an easily described problem, it goes into a number of interesting directions including how FB seems to DDOS certain sites for reasons unknown, defenses against such attacks and DDOS in general, GPL licensing, how to collapse HN conversations, etc.

Not a profound comment. But it was really fun to follow the path down the various rabbit holes. This is why I like HN.

mesozoic• 6 years ago

Detect the crawler and delay the returns as long as you can. Have it use as many resources as you can that should be interesting.

A Facebook crawler was making 7M requests per day to my stupid website

Related Stories