Downtime last Saturday

https://github.com/blog/1364-downtime-last-saturday

by Pr0 • 14 years ago

199 104 14 years ago

github.com

gleb2• 14 years ago

It seems that every GutHub downtime I can recall was caused by automated failover.

rdl10• 14 years ago

I wonder how much downtime they've avoided through automated failover.

gleb2• 14 years ago

Yuh, I have the same question too :-). Hard to do cost/benefit analysis when you only see the costs.

gizmo686• 14 years ago

Every other issue automaticly fails over.

ghshephard• 14 years ago

This may sound selfish, but github does such a great job of writing up post mortems, that I almost look forward to their outages just because I know I'm going to learn a lot when they write their follow up.

tptacek• 14 years ago

Came here to say the same thing. This is Mark Imbriaco's wonder twin power. It really is something to generate goodwill from an outage writeup. Part of it is just unflinching transparency combined with nerdy details; you never feel like they're hiding anything, and you get to learn about all the operational doodads they're working with to run at this scale.

imbriaco4• 14 years ago

Thanks, I really appreciate that.

For me, the motivation for transparency came from too many frustrating instances of being kept in the dark after things had gone wrong. The worst thing both during and after an outage is poor communication, so I do my best to explain as much as I can what is going on during an incident and what's happened after one is resolved.

There's a very simple formula that I follow when writing a public port-mortem:

1. Apologize. You'd be surprised how many people don't do this , to their detriment. If you've harmed someone else because of downtime, the least you can do is apologize to them.

2. Demonstrate understanding of the events that took place.

3. Explain the remediation steps that you're going to take to help prevent further problems of the same type.

Just following those three very simple rules results in an incredibly effective public explanation.

thaumaturgy3• 14 years ago

GitHub and CloudFlare are my two favorites in this regard; if you like GitHub's post-mortems, you'll love CloudFlare's: http://blog.cloudflare.com/tag/postmortem

guptaneil• 14 years ago

Git itself is a distributed VCS, so the GitHub downtime shouldn't have been too devestating to most people. However, speaking as somebody who uses GitHub Pages a lot and had it go down at one of the worst possible times for me, I can say that this postmortem definitely quenched any possibility of me considering moving somewhere else, since I am confident my data/uptime is as safe with them as it could be with anybody.

Everybody suffers from downtime. It's how you handle it that matters.

apeace2• 14 years ago

It's about time we heard from them. I understand the timing of this was unfortunate (right before a holiday), but the trust alluded to in the conclusion of that post would be bolstered by faster post-mortems on major outages like this one.

nixgeek9• 14 years ago

Better to take a few days to gather evidence and really understand the problem, than rush out a post-mortem which is inaccurate or incorrect leaving the impression that downtime is not taken seriously and investigated thoroughly. IMO.

InclinedPlane2• 14 years ago

I disagree. Other than for curiosity needs what's the value in a faster post mortem? Especially considering that a faster post-mortem would likely be a less accurate and less complete post-mortem? What is the difference in actionability for anyone between an update in 2 days vs an update in 4 days? I see none.

apeace2• 14 years ago

To me, a post-mortem doesn't just satisfy curiosity, it eases fears that the problem will return and informs me of future plans which may help prevent the problem, or may bring it back. It helps me form my own plan, since I'm a user of Github.

I for one spent my break hawking my email, in case further Github outages caused any of my automated deploy scripts to fail. I realize it's my own responsibility to write scripts that handle failure scenarios, but the fact is my company pays Github to host our repositories. Downtime happens, I'm understanding of that. But when it does, I want to know what's going on as soon as that information's available--especially when I'm on holiday.

I don't think a blog post written after they resolved all the issues would have been less accurate. It just would have inconvenienced whoever wrote it on a holiday.

Not the biggest deal in the world--it would take me a lot more than this slip-up to switch from Github. But IMO a service provider should get at least some information out faster than this.

raverbashing• 14 years ago

And High Availability isn't. Again

It seems redundancy protocols end up grappling each other more often than not

Unfortunately, there is no easy answer, and I'm sure Github employs people with lots of experience.

This makes me wonder about after several people working on problems like this it's still a challenge

ChuckMcM• 14 years ago

You can't predict what you can't predict. This is what makes the Chaos Monkey experiment so interesting. And yes, HA is hard, and with many things it is hard with respect to latency.

The more latency you can tolerate, the easier HA becomes. At an extreme, if you can tolerate 1 minute latency than each request can come in and compute a most likely way to complete a request with the most authoritative set of actors. Few people though are willing to tolerate a commit taking 15 minutes, much less a couple of hours.

This was one of the most insightful things about NFS and the whole stateless design. By burdening the client with the state the server could be much simpler.

Once you get above a certain size the problems change becoming both easier and harder (easier in that you can disperse your data further, harder in that your confidence in agreement between copies takes longer to compute and thus increases latency). It would be interesting if Google shared their work on Spanner (they seemed to have tackled this problem at a large scale) and given Netflix's experience (Chaos Monkey's dad) it seems like Amazon still hasn't quite gotten the recipe right.

It is a deliciously thorny problem with subtle complexity and unexpected inter-dependencies.

onetwothreefour4• 14 years ago

Ahhh... good old Heartbeat.

We used to use Heartbeat in a similar setup back in 2001. It was the worst architectural decision we ever made, and after one too many a failure (where STONITH/split-brain/etc killed the wrong machine, or both machines) we threw it out.

TL;DR: This will happen again. Guaranteed.

cookiecaper• 14 years ago

What do you suggest as a replacement? Heartbeat is in use at many big players.

mitchty2• 14 years ago

I'll just point out that experience with heartbeat from 2001 is rather outdated.

That would be similar to comparing 2.4 kernels problems to the most recent 3.7 kernel. Likely not an overly useful anecdote.

For the record we use heartbeat at work with no issues such as this.

onetwothreefour4• 14 years ago

I don't know how good Heartbeat is these days. I'm just going based on our experience back then and GitHub's stated experience now.

We (in a new place) use our own fail over code now, but we're running databases instead of file servers -- and you usually don't want to STONITH a database.

For file servers we use Gluster, and it works great for us (I'm sure there are reasons GitHub are doing active-passive DRBD) and you don't need STONITH since it's shared-nothing.

justincormack• 14 years ago

I was brought up to only use stonith via serial or other non-switched network connection. Running over the same network is bound to cause problems. But its not a great solution anyway.

rdl10• 14 years ago

It's nice that modern servers often come with 4 NICs, and Gigabit is Auto-MDIX.

nixgeek9• 14 years ago

Agreed. I completely agree, but the world isn't as kind. Providers often have funny rules about how you can cable things up in their datacenters, and as noted in #4 of where GitHub goes from here that needs to be addressed.

onetwothreefour4• 14 years ago

We used to STONITH (with an APC network power switch) over serial. It didn't really help because serial ports have really bad quality control, and lose a lot of 'packets'. :)

We ended up with dedicated serial cards, and multiple network links just so STONITH worked properly. And even then, Heartbeat was buggy as hell back then, so we'd end up in a active-active situation way too many times and we'd end up with FS corruption.

We were running Reiser on DRBD (ahh, the good old days). Had to hack the kernel a fair bit to make it all work.

Anyway, in the end, we just abandoned the automated Heartbeat failover stuff, and just used it to alert us. We manually ran the failover scripts when a human determined that there was in fact a real failure.

ewokhead5• 14 years ago

Note to Github:

Freeze prod changes two weeks before and two weeks after all major holidays.

Your employees probably don't appreciate the hassle when all they are thinking about is "YEAH! DAYS OFF!"

Just my opinion and how I run my systems in the DC.

anu_gupta• 14 years ago

I guess one of the counter-arguments to this (very good) suggestion, is that holidays are probably the quietest time in terms of traffic and usage.

dlisboa• 14 years ago

Stopping coders from deploying stuff due to a risk of an unspecified "something" going bad makes for frustrated employees. By that time you might as well give them the days off as they'll have little to strive toward. You don't need to freeze all code deployments and other things that have little risk.

Also, they most likely scheduled this at this time due to the lower traffic, probably the lowest of the year for them. While half of Github was probably enjoying their families the other half was planning for this for a long time. Given the size of the operation I don't think anyone took it lightly.

ewokhead5• 14 years ago

Switch modifications should never stop work.

Prod code push != prod infrastructure changes. Which is what the article is talking about. Specifically the agg. switching layer.

My reply is not about code deployments. It is about managing network devices with high visibility and impact.

I still stand by my original comment with a critical detail added:

Freeze prod ~infrastructure~ changes two weeks prior and two weeks after major holidays.

Push code all you want.

The RFO that they provided addresses link aggregation changes which are a part of an infrastructure change.

nixgeek9• 14 years ago

Holidays are actually one of the best times to be making changes as traffic is significantly lower, and IMO, one should be aiming for an infrastructure where you can always ship changes without being afraid of the ramifications.

Architecturally that may mean many things - hitting "SHIP IT!" might push code into a staging environment for some final testing before delivering it onto a platter in production. Should you have multiple sites, it might involve rolling out the new stuff to just one of them until you see how it goes. Maybe you have feature flags and want to introduce a new change to all servers, but just 1% of the user population?

Fundamentally hitting "SHIP IT!" should be doing just that. Any constraints you put on how fast it gets to 100% of the user population are a risk control, and you need to optimize for a balance of developer happiness and system stability.

When you concede "We can't make changes because we're frozen" outside of a critical systems ('life critical') environment, you should quit your IT job and go become a fisherman or something.

rdl10• 14 years ago

I love doing major upgrades over Xmas/NY, Easter, and Labor Day. Lower traffic, and much better chances of fixing things if they go wrong.

I've always found that if you compensate people for it, it's not a problem. I personally would prefer 2-3 weeks extra vacation some other time in exchange for working through the holidays. It's way cheaper to travel in Jan/Feb, too.

It depends on your team, though -- if it's a bunch of people with kids who have school holidays at certain times, it might be more of an issue.

ewokhead5• 14 years ago

Shipping code changes is a different beast.

I am talking about the infrastructure side of things.

I have built large scale percentage-deployment, slice deployment (whatever you want to call them) scenarios like you speak of but modifying an AGG switch that provides connectivity to your entire prod space... Uh.. Go ahead and use your philosophy for managing large infrastructure and I will enjoy my days off thanks.

This change is not a SHIP IT! change. This is a switching infrastructure upgrade. This is not a push from your CI into your rolling rel. system that updates prod applications.

This is an underlying infrastructure change with high impact and high visibility with many stakeholders at risk.

Sorry for any confusion that my, very vague, post caused.

Maybe someday I will become a fisherman. But for now, I will keep these switches and servers up and running with 99.999% uptime. It is what I love to do!

Got any fishing tips?

drstewart2• 14 years ago

One month of no deploys seems pretty high for an organization that does upwards of a hundred deploys a day. 2 days seems far more reasonable for the organizational philosophy they're going for.

dsl4• 14 years ago

Code changes are not the same as infrastructure changes.

ewokhead5• 14 years ago

Yep, it is a long time which is why I mean to speak specifically on the infrastructure side of things. Modifying the agg layer providing connectivity for all prod systems during a holiday weekend would suck.

When I say prod, I mean prod infrastructure. Not code.

Sorry for the confusion.

RyJones• 14 years ago

This is (or was) the MSN way. No releases over winter holidays.

agwa• 14 years ago

It's not just the employees - if you provide a service that your customers depend on to provide their service, they're not going to appreciate the hassle on a holiday especially when it's not at all their fault. This isn't true so much for GitHub, but is for infrastructure providers like AWS.

taf2• 14 years ago

Tell that to rackspace not sure how many where effected on Christmas Day they decided to update a router - causing a 3-1/2 hr outage from 8:30 est to 11:00est aprox, took out part of my infrastructure - made for a nice Christmas morning surprise. Effected the ORD datacenter- only thing I could on RS status is https://status.rackspace.com/index/viewincidents?group=2&.... And of course the ticket in my account

BryantD• 14 years ago

A lot of people replying to this are talking about how holidays are their low period. I work in mobile; the Christmas holiday is the busiest time of the year for us because that's when everyone gets their new devices and downloads apps and has all day to play with them.

Github's a service provider and I would be surprised if none of their customers didn't have peak traffic over Christmas for this or a similar reason.

In any case, I agree with nixgeek; you shouldn't ever expect outages when upgrading infrastructure. On the other hand, you also have to measure the pain of fixing an outage, and that's likely to be higher during the holidays because of availability.

cbsmith2• 14 years ago

tl;dr: It's really hard to get high availability systems right, and we still run the entire service as a single colo.

I can totally understand this kind of thing going wrong, but particularly given the service they provide, why not have a second colo, with a relatively recent clone of the repo, that you can route people to? Heck, you can likely even do an automatic merge once the other repo is working again...

MichaelGG• 14 years ago

Relatively recent clone? Sounds like that would screw customers up pretty bad if they don't realise the problem.

If they went to a second site, having synchronous commit to both sites is how it should be done, no? The extra latency on infrequent git pushes is far less an inconvenience than the possibility of grabbing the wrong code.

cbsmith2• 14 years ago

> Sounds like that would screw customers up pretty bad if they don't realise the problem.

I think there'd be a variety of ways to have the system to fail until the customer made some kind of adjustment that indicated they grokked that there was a failure (like say... changing your upstream).

dos1• 14 years ago

Who's their network switch vendor? I'm not a networking expert, but boy - it sure seems like their switch vendor has screwed some things up royally. Or perhaps this is common with all complex network topologies regardless of hardware vendor?

EDIT: I would just like to say, along with others, I greatly enjoy their postmortems and I feel as though I learn something every time. Kudos to them for being forthright. I host my personal and professional projects with them and am supremely confident that my data is as safe with them as it is with anyone.

rdl10• 14 years ago

They are hosted at Rackspace, so I think the default is "Vendor C".

I'm curious if they'd even be able to support an Arista network. Although this would probably not be a problem with that kind of network.

sounds4• 14 years ago

They don't use Cisco for their Aggregation switches.

thaumaturgy3• 14 years ago

GitHub had a switch-related outage earlier in December, too: https://github.com/blog/1346-network-problems-last-friday

I've seen some really wonky behavior in Cisco equipment even in a really small ISP network, but I have no idea if this is normal or not. Maybe one of the very few people that's responsible for a network of GitHub's size could chime in; I'd be interested too.

nixgeek9• 14 years ago

Adapting a legacy network to deal with growing pains is hard. Anyone who says different is probably lying, or hasn't administered a network outside of 'bedroom scale' previously.

Whilst everyone prefers to work with a blank canvas that is a rare opportunity, and one which still has constraints like budget limitations or "We need this by Wednesday!".

All vendors have their foibles. No vendor is perfect. Datasheets are rarely 100% accurate, behaviour fluctuates based on what code train/version you're running on devices. Network vendors are like everyone else - humans - so just like everyone else they break things occasionally too.

One of the best ways to mitigate this as demonstrated by some of the largest players is to have N+1 physical sites and to be able to control traffic distribution to each of them, including taking one entirely offline without your customers noticing that it's happened. It's also extremely hard to do when you start looking at how CAP theorem applies to your given application, and not just the infrastructure changes required to network it all together, but the application changes required to not have it just explode on you.

sounds4• 14 years ago

This is likely the reason the large cloud companies all do testing that actively causes outages. I'm over-simplifying on purpose here: this is something that requires a lot of thought.

At first glance that seems foolish, but to quote you, "complex network topologies" are very prone to falling over badly. Since they all seem to be a one-off custom setup these days, how can you be sure it won't fall over?

Here are the testing approaches I know about:

1. Netflix Chaos Monkey: http://techblog.netflix.com/2012/07/chaos-monkey-released-in... - but that doesn't mean Netflix has it all together. They still have outages.

2. Google does it. They have a team that goes around unplugging network cables and monitoring how fast the engineers can find and fix the problem. I can't dig it up but it was only a few months ago - hey, Google, your search engine can't find an article about you. :)

tonfa• 14 years ago

http://www.wired.com/wiredenterprise/2012/10/ff-inside-googl... (search for DiRT).

madsushi• 14 years ago

That was my question too. My guess is that it's Arista, based on the MLAG, ISSU, and 'agent' references. There's a chance it could be Cisco or Juniper, but I don't think so. The author of their post-mortems does a very good job of removing any vendor-specific language that would give it away.

regularfry3• 14 years ago

It doesn't sound to me like the switch upgrade was the problem here. Yes, it was the trigger, but the massive failure was caused by the failover setup.

The learning seems to be that if you're going to have automated failover, it must go in one direction only. That means you're guaranteed never to get flaps or mexican standoffs, and determining which node was active is a (relatively) simple question of "did a failover happen?"

I'm sure there's a downside to this, but I can't see one that outweighs the gains from having much simpler failure modes.

lspruel• 14 years ago

As others have mentioned, it is indeed Arista: https://twitter.com/markimbriaco/status/276416483939213312

I think they should be more forward about naming and shaming, but I understand why they'd rather not. Personally I have horrible experience with Nortel/Avaya and would recommend anyone against using their equipment for core switching. The least worst, again in my experience, seem to be Cisco and Juniper.

mikec3k• 14 years ago

I always love to read post-mortems like this. It's fascinating to see how a simple event can trigger a massive failure & we can learn a lot from them.

chuhnk5• 14 years ago

GitHub as a sysadmin/system engineer I feel your pain, I understand it completely and know the horrors of failures leading into multi-hour recovery. That said, you need to do better. This is a heavily relied upon resource for the open source community. Perhaps you didn't estimate this sort of growth but now you are here I'm sorry but the weight does fall on your shoulders. I heavily commend you guys on the service you've provided thus far and what you've done to actually pull all the varying language/library communities together. I honestly want to see this scale and serve 5 9s year round. You need to take a good hard look at the architecture of the stack and find a way to get it multi-homed.

thaumaturgy3• 14 years ago

Please don't take this personally, but I'd like to propose a new rule of etiquette on HN: it's unacceptably lazy to say that some business "needs to do better" without explaining exactly what they should do better, or how.

GitHub is doing better by proactively upgrading their network. During the upgrade, they have run in to some technical difficulties. There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size? And far fewer who could have predicted this kind of trouble?

If we're going to shit on someone, we should at least make it nutrient-rich: include recommendations based on experience from dealing with problems of that nature and magnitude.

dsl4• 14 years ago

I understand they have a lot of nerd/hacker cred in and around the tech hotspots of the country, but GitHub is by no measure "big". The most recent number I could find puts them at 33 servers total. I've worked for non-tech companies that have that many as hot standbys.

One big recommendation I'd offer is to have a secondary network with as few moving pieces as possible (think a dumb unmanaged Netgear switch or two) to run things like heartbeats and DRBD over. That sort of thing should not be living on the same segment as your production traffic.

chuhnk5• 14 years ago

Actually you are right it is a bit lazy. Ok so the first question is, are they multi-homed (as in multiple datacenters)? If not, why not? Because that should be the first step to take. We could talk about all the ways in which you can build redundancy and automated recovery into a cluster of machines in a single datacenter but let me tell you the obvious point that the datacenter is a single point of failure. Rackspace are an extremely solid hosting provider, we used them in my old company for 5 years but even they can suffer fiber cuts, power cuts, UPS's dying, generators not working, someone performing network maintenance and crapping out the routing for the entire dc.

My first recommendation would be setting up a warm-standby or even a hot-standby if possible in another datacenter. It all comes down to state, meaning the data/repos. They are currently using drbd for block level replication of disks in their file servers, this can be done over wan using something like drbd proxy. Another option is to write something in house which replicates write requests to git repos hosted elsewhere. They wrote a collection of services to access git repos on the file servers via rpc, its entirely plausible to extend this. If they are using mysql anywhere thats a pretty simple task for master-slave replication or semi-sync replication in mysql 5.5. Redis also supports replication.

I'm sort of hinting at high availability with read/write but starting out with a read-only solution hosted in another datacenter on the order of 60-120 secs out of sync is not impossible. I think we would all benefit from that kind of continuity right?

Replicate your data to a secondary site, keep it in read-only mode, set low ttl's in dns. Take it a step further and start looking at globally distributed file systems if you are feeling adventurous.

Ricapar• 14 years ago

Sure, GitHub is "heavily relied upon resource for the open source community" - but what is really the impact of their downtime?

You have to wait a little longer to do your push/pull/merge/etc?

Give them a break.

chuhnk5• 14 years ago

Remember that individuals and organisations pay for private repo hosting. Aren't they entitled to an adequate SLA of say 4 9s a year?

gcr• 14 years ago

When seen in that light, lots of nonessential services seem equally unimportant.

What's really the impact of Facebook downtime? You have to wait a little longer to "poke" your crush's profile/look at their picture/etc? Give them a break.

kawsper• 14 years ago

Homebrew on OSX is unusable without Github.

Rubygems is kind of unusable without Github.

Several ppensource projects documentation is unavailable without Github.

robomartin• 14 years ago

http://www.youtube.com/watch?v=c8N72t7aScY

el_cuadrado2• 14 years ago

High Availability strikes again. No surprise there.

But I am mildly flabbergasted by the fact that GitHub uses STONITH. The technology is as safe as open-core nuclear reactor, and it works reliably only in very simple conditions.

mofraw• 14 years ago

STONITH is designed for critical failure so you don't end up with a split-brain situation, which is far worse than a dead node. STONITH is a good thing. The problem here is more than cluster wasn't configure to survive a catastrophic switching failure.

el_cuadrado2• 14 years ago

Yep, right - fencing solution that relies on the network to stop the service. How could that possibly fail?

ewokhead5• 14 years ago

I just realized that the Sys. Admin/Prod. Ops to Developer ratio here is crazy low. Everyone assumes I am talking about code changes when the article is about prod switching and network transit device changes.

MLAG or any LAG technology, LACP, bonds whatever should never impact the deployment of code. It should be invisible when it is working. Obviously it is very visible when it breaks though.

My heart goes out to the Github guys!

Sorry for the confusion everyone.

jetsnoc• 14 years ago

Wow, I'm very glad our company chose a routed design with an interior routing protocol (OSPF.) I've never been able to push the limits of a layer two network as far as GitHub. A routed network helps segment the network so when systems fail or a re-convergence mistakenly occurs only a few racks are having problems and not the entire system. It's also very helpful for us to push routes to our exterior routing protocol (BGP.)

I also find it interesting they don't use any additional out-of-band network for heartbeats/management especially as unstable as their layer two network has been. It sounds like file servers need a secondary stable heartbeat network even if it is only 10/100. No judgements being passed here it just seems like a lot of eggs in one basket. That being said, Thank you for this write-up and sharing so openly and honestly. Happy GitHub customer here!

EDIT: Yes, I know routed networks can have similar problems but they are designed for routing, pathing and redundancy with a lot less overhead on the broadcast domain.

imbriaco4• 14 years ago

You're right on all counts. We have a great many plans with regard to how we want our network to operate that are underway.

mprovost• 14 years ago

Routed networks tend to fail closed where switched networks tend to fail open. That usually seems to be the problem with layer two failures, they can easily spread traffic everywhere and it takes a while for things to calm down, in the meantime the network hardware is often overwhelmed. With layer three networks the failures tend to be that you lose the ability to route somewhere but in my experience it's often easier to recover from that situation.

akg_67• 14 years ago

I get the impression that issue wasn't network hardware but bad high availability design on fileserver side. Why do GitHub has failover network on same network hardware as primary network? But I am not surprised as I see this at lot of clients that they have failover network on separate VLAN on same network hardware. And, whenever they have network hardware issue, servers run into split brain problems.

The failover network should be totally different physically and logically from primary network. The heartbeat between file servers should be checked through both primary network and failover network. If a server can't be reached by its partner over primary network, it should be gracefully taken offline by partner through failover network.

ksec• 14 years ago

I have absolutely ZERO knowledge on Enterprise Networking. But it strikes me that something as dump as Router, Switches or Network Equipment are still so unstable.

A Hypothetical questions, why not something a 8 ARM 64 bit core Linux computer as switches? Making more logic resides in software instead?

regularfry3• 14 years ago

There is a move to put more of the switching and routing logic in software (google for openvswitch if you're interested) but part of the problem is that general purpose hardware doesn't stand a hope in hell of keeping up with interesting network data rates. You absolutely need to be doing a fair portion of the work in hardware, the CPUs just coordinate it.

nixgeek9• 14 years ago

It's probably worth reviewing this article on Intel FM6000 SDN to get a glimpse into the complexity involved.

http://www.intel.com/content/dam/www/public/us/en/documents/...

treskot• 14 years ago

I've noticed MC-LAG / MLAG failing quite often in my case. Any details over the MLAG fail? Are we doing it wrong? Alternatives?

Downtime last Saturday

Related Stories