Why did we take reddit down for 71 minutes?

http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-for-71.html

by icey • 17 years ago

100 44 17 years ago

blog.reddit.com

jbellis4• 17 years ago

"Memcachedb also has another feature that blocks all reads while it writes to the disk."

Seriously?

Wow.

davidw• 17 years ago

Sounds like it's time to look at Redis...

simonw• 17 years ago

Or Tokyo Cabinet / Tyrant, which is still a high performance key/value store but doesn't need to fit everything in RAM. Depends on how much they're storing.

jbellis4• 17 years ago

Well, no; if they're using memcachedb presumably their data doesn't fit in memory on a single machine.

Cassandra would be a better choice.

/cassandra committer, but it would :)

patio11• 17 years ago

Quite seriously.

I use memcachedb in my deployment of A/Bingo (it isn't limited to that), which avoids using a SQL database because I wanted it to be blazing fast for people who having significantly higher traffic sites than I do. The typical case performance on my site is fantastic. The worst case performance is an abomination -- when it flushes the site essentially pauses for all users for five seconds.

This is tolerable for me, but only just. The reason I use memcachedb as opposed to vanilla memcached is because I was worried about persistence in the event of a server failure, but given that I can count resets of my server over 3.5 years on a single hand, I might just decide "In the event of a server failure, I lose any A/B tests in progress and have to start over. Oh well!"

rubyrescue2• 17 years ago

sounds like a redefinition of 'in any way' to me...

So why am I singing the praises of Amazon and EC2? Mainly to dispel the opinion that the site getting slower since the move is in any way related to Amazon...(snipped)...Unfortunately, the single EBS volumes they were on could not handle these bursting writes.

aaronblohowiak• 17 years ago

This isn't an EC2 issue, this is a SAN issue. Wether it is EBS or an NFS drive, meh. This is an architecting issue and while it is a result of the underlying hardware, the underlying hardware is not the constraint.

rubyrescue2• 17 years ago

agreed it's largely an architecture issue, however poor EBS performance is contributing factor and he seems to go out of his way to say that it's not...

ShabbyDoo3• 17 years ago

I'm bothered by the need to run SW RAID on top of HW RAID. One would think that Amazon would sell faster EBS "disks" for a premium.

And slower disks for a discount? But, I guess that's what S3 is for.

psranga• 17 years ago

I think it works like this:

Even if Amazon uses 'k' hdds for one EBS "disk", since you're sharing the real hdds with other users, you don't get 'k' hdds' performance, you only get a fraction.

By RAIDing over 'n' EBS "disks", you are effectively compensating for the reduced performance due to sharing.

ShabbyDoo3• 17 years ago

I get what the stack looks like, but it seems really broken and likely quite inefficient. Thus far, Amazon has gone after greenfield applications which can be written within the constraints of their cloud platform. However, there are a ton of people hosting their own SQL database-based apps where a single DB is the bottleneck. Without significant refactoring, these apps can only scale vertically with the DB. So, while Amazon provides nice, big boxes to run SQLServer/MySQL/etc., disk performance is that of a desktop machine -- hardly a balanced system. How many more customers could they capture if they offered premium, high-performance storage options?

dgreensp• 17 years ago

Reddit is down super-often for me. This very minute, I can't log in or use the site logged in, it just serves me 503 errors.

jacquesm4• 17 years ago

One major reason not to hop on the cloud bandwagon just yet is issues like these. The more layers underneath that are not under your control the more layers you'll have to add to remedy that.

Systems with excessive complexity are hard to debug, especially when it comes to analyzing performance issues.

Given complete control of the hardware from the ground up it can already be quite hard to accurately pinpoint a bottle neck so you can solve it. Adding a lot of stuff between your code and the hardware is not going to make that any easier.

Typically a stack has 6 layers before you get to your application: drive, controller, driver, filesystem, database, app.

In a cloud environment anything under the filesystem layer is effectively out of your control and out of your ability to troubleshoot. The solution, to add another layer of complexity in order to combat the slowdown is really the opposite of what an ideal cloud environment would give you.

After all, the #1 selling point of the cloud is scalability and performance.

I think that it would be best if Amazon worked together with the OP to resolve the issue as a problem ticket rather than to try to solve it by adding a software raid.

Of course, that's just armchair reasoning, not being in the hot seat makes life easier.

scott_s• 17 years ago

After all, the #1 selling point of the cloud is scalability and performance.

I think you're missing the key component that it's scalability and performance that you don't have to manage yourself. And, yes, when you don't manage it yourself, it's going to be much more difficult to diagnose performance problems. But someone else managing the infrastructure also means that people who would be incapable of diagnosing the problem anyway (either from lack of expertise or lack of time) have access to the resources.

jacquesm4• 17 years ago

It definitely looks as though they ended up having to manage it themselves.

Basically this seems to put a fairly low upper limit to using 'the cloud' for something a little larger before you get back exactly the same kinds of issues that you were dealing with when using self-hosted hardware, only at a higher price point.

nudist• 17 years ago

If they had a dedicated server stack, they would probably have done the same thing as a hardware RAID anyway. They just replicated in software what they would have done in hardware.

rms3• 17 years ago

Reddit went on the cloud bandwagon not because they thought it was superior to managed servers, but because Conde Nast's IT department sucked. With the ongoing growth, they were having trouble procuring the additional servers as needed, and moving Amazon solved that problem.

pavs2• 17 years ago

Can you give us any reference to the "Condé Nast's IT department's fault" theory? From the best of my knowledge reddit infrastructure was always maintained by reddit stuff and Condé Nast had no influence on that.

The current reddit stuff did an "Ask me Anything" thread on reddit when the founders left and they said the reason for the move to AWS was purely because of price/scaling issues and some part of reddit was already using AWS even before Condé Nast bought them.

jedberg6• 17 years ago

That's not entirely true. Yes, it is true that getting servers was hard, and that was definitely a factor.

But the bigger factor for me was that I was tired of having to build, image and rack all those servers. I liked the flexibility of EC2, and also not having to waste resources ordering a full rack's worth of hardware every time.

Cost was also an issue. Datacenter space in SF is expensive, but it had to be in SF, because that is where I was. EC2 proved to be much cheaper than physical servers.

I also like the fact that I don't have to run to the datacenter anymore when there is an issue. I just file a ticket with Amazon.

jedberg6• 17 years ago

Actually, we did work with Amazon. The RAID was their suggestion.

shrike• 17 years ago

Does anyone know if reddit is doing anything special to create the RAID? Or is it just mdadm?

jedberg6• 17 years ago

Just mdadm.

mark_l_watson• 17 years ago

I really like 'behind the scenes' stories like this from big sites like Reddit, Facebook, Heroku etc. I work on a smaller scale of a few EC2 at a time, but I really enjoy the scaling info.

Why did we take reddit down for 71 minutes?

Related Stories