GitHub was down

https://status.github.com/messages?6-11-2013

by pegler • 13 years ago

39 26 13 years ago

status.github.com

veemjeem• 13 years ago

Crazy how many people are affected when github goes down for less than 2 minutes.

jborden13• 13 years ago

There goes our plans for a release tonight. This DDoS seems to be a fundamental flaw of popular SaaS services.

bennyg• 13 years ago

It's already back up (for me at least).

jamesbritt• 13 years ago

I happily created a new Github repo, imported some code, made some changes and pushed stuff, all OK. And just now when I want to post announcements of my new library, bam! Github is down.

Mostly I don't care if they are periodically unvailable, but the timing on this was amusing.

wluu• 13 years ago

Yeah, there was a brief major service outage according to the status page.

https://status.github.com/messages

Today

1:45 UTC Major service outage.

anthonyb• 13 years ago

2:01 UTC We're still experiencing a large DDoS attack. The site is experiencing major packet loss and is mostly unavailable. We're working to further mitigate the attack.

heyitsgarrett• 13 years ago

Genuine question – who DDoSes Github? From what I've seen, DDoS attacks often seem to be a response to a site or company entering the political sphere in a controversial way.

AYBABTME• 13 years ago

In previous threads about Github being DDoS'd, I asked the same question.

Apparently it's some scheme to ask ransom from companies/startups that have enough money to pay and/or not enough infrastructure to sustain the attack without giving up.

jackowayed• 13 years ago

Yep. There's a well-written article about an early DDoS extortion attempt where the victim fought instead of paying, developing some novel protection technology in the proces: http://www.csoonline.com/article/print/220336

xal• 13 years ago

Yea, all of us successful web companies face these all the time. We get them multiple times a day. Most of us end up giving up and pay for Arbor gear and ridiculously over provisioned pipes as well as good relations with our hosting facilities.

IceyEC• 13 years ago

Additionally, some group may be targeting a specific project on Github

not_chriscohoat2• 13 years ago

Unfortunately this occurred at the exact moment I was deploying some changes. Makes me think that being so dependent on GitHub for a production code repo is a little big dangerous.

Redundancy would certainly solve this issue, though.

apendleton3• 13 years ago

There's already built-in redundancy. Wherever you pushed to Github from has a complete copy of the current state of your repository. Github is effectively just a publicly accessible mirror, but you can always push to a different remote elsewhere, push directly to your production instance, or even host your repo in an s3 bucket, all while Github is down.

joeblau3• 13 years ago

I've been in some development environments where the GitHub master branch is what the build system pulls from in order to deploy to dev, staging and production. With that setup, it's more challenging since the build process probably involves more steps and potentially ssh keys that are only allow the build server to deploy.

Depending on how many repos you need, I would say set up an alternative remote on bitbucket.org.

not_chriscohoat2• 13 years ago

Right, I get that...I meant a separate remote when I said redundancy. GitHub has just been so reliable I haven't even needed other remotes for the past 8 or so months we've been using it like joeblau mentioned.

readme• 13 years ago

Guys, please don't forget the advantage of using DVCS to begin with. I like Github too, but if Github goes down it shouldn't put the brakes on your work. Push to multiple remotes. Solved.

dan_manges• 13 years ago

A lot of people rely on other features of GitHub for their workflow: issues, pull requests, comments.

apendleton3• 13 years ago

I've been telling people for a fair while that I wish Github issues were stored as part of the git repos themselves, not so much for availability reasons as that it'd be awesome if the state of bugs got branched and merged along with everything else, so that a bug could be fixed in a branch, but not yet in master, and would get fixed in master when the branch got merged. But I guess availability would be a nice additional perk.

joeblau3• 13 years ago

Does GitHub use a CDN? It seems like that might help them with the DDoSing.

tedivm• 13 years ago

CDNs are only useful for things that can actually be cached, and which have been accessed recently. Static content is easy, and I'm sure Github does it, but dynamic content doesn't benefit as much for a CDN.

As another note, anyone doing a DDoS is going to target parts of the system that are more intensive. They'd just skip the cached things and go after the ones that have to be regenerated.

joeblau3• 13 years ago

That makes sense, but can't a CDN also help identify the DDoS and block communications from certain servers. I remember seeing a captcha on Adria Richards website after she got those two devs kicked/fired. Maybe GitHub just needs better tools to identify and block the DDoSes quicker.

GitHub was down

Related Stories