GitHub: About This Week's Availability

https://github.com/blog/1036-about-this-week-s-availability

by remi • 14 years ago

182 54 14 years ago

robbiet480• 14 years ago

Who would DDOS GitHub and for that matter, why?

11235813212• 14 years ago

I'm not saying 37signals did it, but I bet someone who read this post did: http://37signals.com/svn/posts/3099-benchmarking-basecamps-u...

I'm sure it was tempting more than a few people with the resources.

Edit: just to be clear I meant any reader of signal vs. noise could have seen that (and Jesse's #1 comment and the jokes about DDOS in the comments) and decided to pull a prank. 37signals would never do that to anybody.

dhconnelly2• 14 years ago

This seems a bit far fetched.

jnewland• 14 years ago

My wife is pretty convinced I brought this upon myself with my comment on that post. Lol, you never know.

11235813212• 14 years ago

I guess I didn't display my 37signals fanboy card prominently enough.

imajes• 14 years ago

extortionists.

There's a growing band of pirates (the web kind) who hold sites to ransom via their massive botnets, in exchange for payment -- 'protection money'.

pavel_lishin3• 14 years ago

But why Github? Wouldn't a less tech-savvy business be much more likely to cave in, and have a better cash flow? (e.g., porn and gambling sites?)

baddox• 14 years ago

Do any of these extortionists have a reputation for actually stopping the attack once they get paid? I certainly wouldn't trust some random group that's currently DDOSing me enough to pay them.

blibble• 14 years ago

or just bored kiddies, happens quite a lot

rplnt• 14 years ago

Besides reasons mentioned, it could benefit their competitors. Remember that sourceforge was under heavy ddos not even a year ago.

frisco• 14 years ago

I remember reading a comment a long time ago -- maybe here or Quora, I think -- talking about how people don't usually realize how much trouble organized crime gives small and midsize businesses. You'll start getting DDOSed and then an email appears a day later: pay us and we'll stop bringing down your site. I have no idea if extortion is what's happening here versus simple lulzseekers or a grudge someone holds against them, but I wouldn't be surprised.

jrockway• 14 years ago

"Organized crime" or just regular crime?

pavel_lishin3• 14 years ago

As I understand it, eastern european criminal organizations (I hesitate to use the term "mafia") control pretty big botnets, and use them to do exactly this.

Although github is sort of an odd target for them to be going after. Maybe their hired geeks are just testing their own software - github seems like the sort of place that would figure out how to defeat a DDoS attack, and by learning how, the black hats could improve their tools.

bdg• 14 years ago

In some countries that are absolutely poor, it is entirely common to have the local mob recruiter look for young talented kids that they can bring into their organization. They make them an offer when the kids are out begging for change or something like "we'll take care of your mom and make sure she doesn't break her legs if you come and get an education in our school, we might even pay you."

The youngster ends up in basically a run-down warehouse full of other kids each taught how to look for basic exploits like mysql injection or use root kits, and they sit around and do this all day.

This is why some countries are typically COEs for hacking, such as Russia with crimeware kits, and until recently Brazil for bank hacking.

This is also why you need coordinated efforts of special agents armed with assault rifles to clean out some spam-mills sometimes (they're taking them away from some mob).

What I've described is a bit outmoded, typically this behavior is turning into other things such as this: http://abh-news.com/cybercrime-china-hackers-training-camp-c... as well as the makers of exploit scripts are just getting better and recruiting these kids doesn't make sense any more... you can intelligently scan for millions of sites using phpmyadmin and automatically run your exploit with it.

As with any crime you find the trends all over the map: not all robberies are mob related, not all of them aren't, some of them are more elaborate than others, etc.

robbyt• 14 years ago

Github is really great, but I'm a cynic.

Indeed, why _would_ anyone want to DDoS github? How can we believe Github that the outages were due to a DDoS? They're smart people, aren't they using solid load balancesrs that can mitigate DDoS attacks? Why haven't they issued an actual statement describing the supposed attacks in better detail?

mike-cardwell• 14 years ago

That's not just cynical; that's down right paranoid.

dhconnelly2• 14 years ago

As another poster mentioned, DDOS attacks on successful companies are not uncommon. And there isn't an easy fix for this.

jtchang• 14 years ago

Load balancers generally don't offer as much mitigation against DDoS as you think. The load balancer will probably fall over if the backend servers don't do so first. The sheer volume of seemingly valid traffic is what kills you.

Which is why effective anti-DDoS means working with your upstream provider to figure out how to differentiate between real traffic and fake traffic. The best solution to a DDoS is the phone # of a tech that can implement firewall rules upstream and a good traffic analyzer to tell them exactly how to filter it.

swalberg• 14 years ago

Often you can't pick apart the good traffic from the bad.

Fortunately bots are usually pretty stupid. If you can outrun them on bandwidth, then change /victimpage.html to 302 to /victimpage-new.html. The web server or load balancer can send those redirects really fast and it doesn't take much bandwidth either. I have never seen a bot chase that redirect.

After a particularly nasty DDOS attack (where our upstream provider just shrugged their shoulders) I wrote an F5 iRule:

1. Check for the IAMNOTABOT cookie 2. If not there, redirect to /cookie-me?oldpage=the_page_you_were_trying_to_access 3. Set IAMNOTABOT=true cookie 4. Redirect to the old page

acdha• 14 years ago

Ask anyone who's been there and they'll tell you that load balancers only make DDoS attacks harder. Unless you have something like AWS AutoScaling, they'll just generate more traffic - and if you do, it would quickly become a financial DDoS. That's why sites like Yahoo.com have been taken out: you have finite server capacity but the attacker merely has to find another another exploitable system somewhere on the Internet.

The way you deal with a DDoS is hard: you can try to identify distinguishing characteristics of the attack so traffic can be blocked as far out as possible and you try to figure out who's behind the attack so you can get law-enforcement involved. Otherwise you're just playing whack-a-mole with compromised home computers all around the globe…

ceejayoz• 14 years ago

> Indeed, why _would_ anyone want to DDoS github?

"Some people just want to watch the world burn."

Organized crime extorting sites with DDoSes is certainly not unheard of, either.

mitchellhislop• 14 years ago

Responding to your last sentence, there was a comment on the blog post that they are being vague to not give anything away to the attackers. If they start describing the attack (even how they perceive it) and/or mitigation steps, they are giving targeting data to the people doing the attack.

brown9-22• 14 years ago

Arguably, if your load balancer setup is such that traffic can increase 10000x from the minute before and nothing falls over, you were probably incredibly over-provisioned in the first place.

Unless your regular traffic pattern has 10000x increases built into it, why would you spend all the extra expense to be able to handle that level of load?

AznHisoka2• 14 years ago

Couldn't they just keep adding IP's to some blacklist (not htaccess, but something more efficient), and the attackers will eventually run out of IPs to attack from? There's only a finite # of computers you can really have control of. So you end up blocking some innocents, but you take care of that after the attack is over.

rcthompson• 14 years ago

I thought that botnet operators had methods of rapidly changing bots' IPs so as to defeat IP blocks.

abraham• 14 years ago

If it is a smart DDoS the requests won't look any different than typical users visiting your site.

Tobu2• 14 years ago

Maybe a whitelist could work. My IP has visited in the past week / I have a legit session cookie, now gimme bandwidth.

brown9-22• 14 years ago

This only works if you decide you want to never accept a new customer again.

zxoq3• 14 years ago

The attacker is most likely using a program that spoofs the sender IP so every packet has a unique IP. It's not feasible to block that. You need a more intelligent router to block attacks of that kind (and lots of bandwidth).

IanDrake• 14 years ago

I wonder if something like http://www.cloudflare.com could help.

dbecker• 14 years ago

If I were a DDoS attacker, I wouldn't want to make enemies with the sort of people that rely on github.

Though it's easy to imagine that those guys think differently than I do.

ellie42• 14 years ago

I think if you build your app properly the only problem with DDoS is traffic. Github partnerships with Rackspace so their cloud traffic can't be expensive. Correct me if I'm wrong.

newobj• 14 years ago

I suspect you may be oversimplifying the solution or undersimplifying the nature of these kinds of attacks. What would constitute a 'properly' infallible defense to you?

tworats• 14 years ago

Are there any good guides on how to combat a DDOS? I'm curious how a team of good technical people without past experience in the area could react.

zxoq3• 14 years ago

Fighting a DDOS is very hard if you do not have the resources to purchase more bandwidth. The DDOS attacks I've dealt with have been almost exclusively UDP-floods in the 500-2000 mbit range. No amount of server-side configuration will fix that unfortunately. If the attack is some more creative form, but with less bandwidth (SYN flood etc.) a cisco router can help out a lot.

Your best bet is to move to a protected hosting provider (dragonara or ddoshostingsolutions are quite good for Europe / US). Failing that you can purchase a bunch of VPS servers, round-robin DNS to them and reverse proxy to your actual server (and keep that IP off the DNS system entirely). This way only the proxy servers will go down if you are attacked.

Edit: If they are not spoofing they sender address, you can achieve some success by mailing the hosting provider of the IP performing the attack. In 90% of cases the server has been hijacked and they will shut it down, this only works for US / European companies though since Chinese / Russian hosting providers never ever reply in my experience. For maximum success you can attach a tcpdump of the attack traffic.

newhouseb• 14 years ago

I was poking around a botnet today and saw that the particular one I was looking at was just DDOSing DNS (port 53). Could you just use an /etc/hosts file and turn off UDP entirely? My understanding is that part of the problem is that the reason 53 is so vulnerable is because the OS sits there waiting for any UDP replies from DNS on it.

alexchamberlain• 14 years ago

You can filter IPs on country given the correct software. During an attack, just prioritise the traffic from high use countries. It might work?

jvoorhis• 14 years ago

I've used Github frequently all week,personally and professionally, and if they temporarily lost a nine or three, I haven't noticed.

pron• 14 years ago

Very nice. Just one thing: I already know I'm awesome and I don't need my source revision control provider to boost my self confidence (I'm referring to the line "...you, our awesome GitHub users"). OTOH, it's clear that it's GitHub's employees, rather than its users that need constant reassurance of their coolness. GitHub is a nice company that provides a good, solid and necessary service - hosted SCM. Then why is it that their company blog focuses on their drinking habits?

Dear GitHub,

Although your product may sound boring to laymen, you provide a good service to a very important industry. Your "boring" day job reflects nothing on your personal and very exciting lives. Your customers, however, like you for what you provide them with, not for your companionship, and I'm sure your friends like you for the opposite reasons, as they very well should. In fact, everybody loves you for many reasons, and you are all very lovable. So please, keep your extra-curricular activities to your friends, and your work activities to your customers. You can call your friends "awesome" if that's your kind of thing, but for various reasons it is better to treat your customers with proper decorum. If you're unable to keep the two separate, you're in for some bitter disappointment later in life.

Love, everyone.

GitHub: About This Week's Availability

Related Stories