Bitbucket: On our extended downtime with Amazon

http://blog.bitbucket.org/2009/10/04/on-our-extended-downtime-amazon-and-whats-coming/

by brodie • 17 years ago

86 36 17 years ago

blog.bitbucket.org

ErrantX• 17 years ago

Downtime like this sucks - I've been there beating my head against a brick wall trying to get someone else to admit it really is something they need to look at :( I'm impressed with the effort they put in in the end.

BitBucket has had quite a lot of down time this year - and I also had a few problems with payments a short time back.

but

It doesn't piss me off like it would with certain other providers.

The status update pages are amusingly maintained when they go down, updates are prompt and contacting support got a sensibly fast (it was a Sunday I emailed :D). At no point have I thought about moving on. So they do deserve a lot of kudos for that.

moe• 17 years ago

Good to hear you're back on track.

16h resolution time is actually not too bad when you look at your monthly bill - but the initial auto-denial (typical for large service providers of all kinds) is ofcourse annoying. Perhaps you get faster action next time, now that you're paying for the premium plan.

It is indeed strange, though, that it took them so long to discover what must have been a huge traffic spike on your instance. Their administration panel needs work if it doesn't point out such anomalies at a glance. Also your own monitoring needs work if you didn't notice the spike of inbound traffic.

Anyways, good luck for the future and if you decide to switch or extend I'd be curious to read about it.

durana2• 17 years ago

Yikes, always remember to check interface utilization when troubleshooting performance issues with anything network related!

tghw2• 17 years ago

http://news.ycombinator.com/item?id=859941

durana2• 17 years ago

Yeah, I saw that too. Too bad it seems like it was ~15 hours until Amazon checked an interface and saw the high utilization.

rbranson2• 17 years ago

This is surprising to me. It seems like Amazon should have dedicated ethernet (or at least a separate VLAN + higher QoS) for EBS.

tc4• 17 years ago

Wait! What about Network Neutrality? Using QoS would deprioritize other traffic. In particular it would unfairly advantage EBS over other cloud storage solutions you could be trying to access from EC2.

In all seriousness, I agree that Amazon should probably be using QoS here so outside traffic can't affect your EC2 to EBS link. I just want to point out that it is difficult to be simultaneously in favor of Amazon doing this while being against ISPs prioritizing voice traffic, which provides similar resilience for voice against outside influences [1].

[1] I'm not suggesting that the parent poster suffers this potential cognitive dissidence.

tsuraan4• 17 years ago

QOS and net neutrality are completely unrelated. What net neutrality is trying to prevent isn't prioritization by service, it's prioritization by server. Comcast wants to be able to hold Google's (or ycombinator's, or twitter's) traffic hostage, degrading their bandwidth unless they pay Comcast a fee. They'd also like to be able to ruin skype's voice services and netflix's video services so that their captive audiences can only get phone and video services through Comcast. That's what net neutrality is fighting, not perfectly reasonable QOS.

bcl2• 17 years ago

Why weren't they running any monitoring on their instances? A simple graph of incoming traffic should have revealed the massive influx of packets and helped track things down more quickly.

Just because a system is 'in the cloud' is no reason not to setup normal monitoring on the system.

I wrote http://www.brianlane.com/software/systemhealth/ to try and help with this situation, it is simple to install, doesn't require anything other then rrdtool and a web server to serve up the graphs.

rbranson2• 17 years ago

They may have been, but I imagine it's difficult to discern the difference between legitimate traffic and a DoS attack if you've got NAS volumes mounted over the same interface. You'd need to do a more thorough network analysis.

drusenko• 17 years ago

They claimed in many places that their "pipe" was "maxed out". If the attack was large enough to use up all of their available bandwidth, that should be easily noticeable in any basic bandwidth graphs.

tghw2• 17 years ago

Jesper addressed this in the IRC channel. Basically, the traffic was not actually reaching the box, it was just saturating the available bandwidth to the EBS store. Moreover, since the attack was against EBS and not EC2, I'm not sure that you could actually get any monitoring in place on the volume.

I'm kind of surprised your first assumption is that they have no monitoring in place. I like BitBucket, so I'm biased, but I can't really see anything about this instance that would make assume it was caused by some sort of incompetence on their part.

bcl2• 17 years ago

Sorry, I'm not familiar with how EBS connects to the EC2 instances so I assumed the incoming traffic was hitting their EC2 instance.

Are you are saying that EBS is accessible from outside and that the attackers were able to discover its IP and attack it directly? This doesn't seem like a very safe way to configure things.

I also wonder why Amazon didn't notice this with any of their monitoring of either traffic or system performance and latency. Assuming they have such monitoring in place of course.

anApple2• 17 years ago

After all, amazon was right to tell them that everything was in order. As it was.

He should fire his sysadmin for not checking the in/out network traffic...

spudlyo2• 17 years ago

Let's not be too hasty to play armchair sysadmin. Someone who claimed to be involved said the traffic never reached their servers.

http://news.ycombinator.com/item?id=859941

mingdingo• 17 years ago

Is that possible? How could they be the only ones affected then?

anApple2• 17 years ago

They said they moved their servers to 3 availabilty zones and still had the same problems. What's the probably of having the same attack in 3 zones on a subnet you are randomly assigned to?

Besides, firing would definitely be a little harsh. Everybody deserves a 2nd chance.

gojomo3• 17 years ago

I wonder what someone has against BitBucket... but I also understand that BitBucket is unlikely to share all of whatever they suspect.

In such cases, the target of a DDoS often only has hunches and pseudonymous threats from which to reason about attacker motivations. Airing such speculative info risks (1) unfairly implicating innocents who may have been framed; (2) encouraging the guilty with more attention for their grievances or destructive skills.

spudlyo2• 17 years ago

Going on nothing other than my overactive imagination, I speculate that somebody hosted exploit code on BitBucket that somebody else didn't want made public. There seems to be a growing trend of black hat security types who are very much against disclosure of exploits and holes.

pjhyett• 17 years ago

We (GitHub) were DDoS'd the other week when members of an open source project hosted on the site got upset at the other members and wanted to make the entire ecosystem around the project suffer. Not saying that's what happened to BitBucket, but it could be a similar reason.

Talking with Sourceforge, it sounds stuff like this happens fairly regularly. It's one of the disadvantages of running a site that has members capable of doing this stuff.

shughes• 17 years ago

The only thing that kind of bothered me was that Bitbucket focused too much on blaming Amazon. Here's a sports example. Let's say a team loses because the left tackle messes up and the quarterback gets sacked. After the game, during a press conference, does the quarterback say that they lost, but it's the left tackle's fault? No. Even if the quarterback believes he himself performed great, he'll still take part of the blame. That's because it's the professional thing to do, and people respect him more for it.

You could say that Amazon and Bitbucket aren't on the same team, so it's different. But Bitbucket is paying Amazon to perform better just like a football team pays a player to make the team perform better. In other words, based on the concept of a team recruiting players for better team performance, Bitbucket recruits Amazon for better company performance, so they're basically on the same team.

That said, Bitbucket should focus less on singling out Amazon, and focus more on trying to figure out what they could have done better. It's the professional thing to do, and I'll respect them more for it.

ubernostrum• 17 years ago

I certainly can understand why they did, though; for a very long time the only thing they knew was "Amazon's EBS seems to have failed". Once it was known that they were being DoS'd they quickly got that information out.

zaidf2• 17 years ago

DDOSes are a sucker the first time round.

We went through the symptoms these guys did for about a week(!) before figuring out and putting hardware DDOS protection in place with our dedicated provider.

The lesson was well worth it though. Next time we have these symptoms, we won't be reinstalling the OS, hardware etc.

mtw• 17 years ago

what did you get for hardware protection? Do you have your own data center?

zaidf2• 17 years ago

I believe we have Cisco Guard DDoS Mitigation Appliance. It was provided to our by our data center for additional $59/month. We certainly don't have our own data center.

Bitbucket: On our extended downtime with Amazon

Related Stories