Jedberg on Reddit's most recent outage

http://www.reddit.com/r/announcements/comments/gva4t/on_reddits_outage/

by zacharycohn • 15 years ago

59 24 15 years ago

benologist• 15 years ago

Be interesting to see what hosting offers they get .... lot of benefits to being the hosting company that keeps reddit online, if they manage to.

cagenut• 15 years ago

Its actually very easy to beat the price/performance of ec2 using real hardware. The draw of ec2/aws is in the "everything is a monthly charge and an api call away" operational instant gratification. If you need to auto-scale by the hour there's few other places you can do it. If you can generally capacity-manage your setup a month or more into the future, then a 1-week latency on RunInstances isn't actually a problem.

alex1• 15 years ago

> All of our Cassandra nodes are now using only local disk, and we hope to have all of postgres on local disk soon.

Since local disk is temporary and can be lost any second, don't they still have to use EBS for persistence?

dialtone• 15 years ago

They probably would use replication to a different region/zone and then they implement their own snapshot/backup mechanism on those replica machines so that they can use them to re-create instances when they lose them. Also PostgreSQL ships logs that you can store in S3 and bring up an instance from another area starting from a known backup point.

EBS is a pretty bad product regarding reliability and consistency of performance unfortunately, it's better to design your system without it.

jjm4• 15 years ago

Re-building by reading logs out from S3 can be very slow. It should be seen as a last ditch effort. Not really a hot swap, or fail over solution (depending on data size obviously).

If you can't use any persistent storage, then these machines become pure processing nodes. Which in this case, I feel like it would be better to not design your system with EC2 at all. :-(

To me having EBS made EC2 a very powerful solution compared with their competitors. Not having durable and consistent EBS otherwise makes no point in differentiation and serves purely as non-functional fluff we end up paying extra for.

jjm4• 15 years ago

At any time an instance may become un-responsive and so will local storage, no?

I don't see how this helps?

joevandyk2• 15 years ago

postgresql replication to other instances?

i believe the point is that instance storage is more reliable than ebs.

jjm4• 15 years ago

Why would you want to take a chance on that? Every instance should be thought of as expendable. They can go down at anytime.

EBS is really just like any other NAS/iScsi vol and it wouldn't be such a problem if they did what they're supposed to. That is, be consistent in read, write, and durability.

cygwin98• 15 years ago

Actually, there was a heated discussion on such an issue a month ago at Reddit [1], when AWS started acting up back then. Someone was asking why cann't Reddit buy beefy servers and co-lo them. The arguments were that considering the traffic Reddit is having, it's very difficult to DIY technically if not possible. Looking back, it looks like co-lo is still the way to go, especially for big sites like Reddit. Anyway, if StackOverflow can make it, why cannot Reddit, both have comparable traffic?

[1] http://www.reddit.com/r/blog/comments/g66f0/why_reddit_was_d...

kbatten• 15 years ago

Whether you co-lo or run in the cloud it seems to me that you would still want the same types of redundancy/failover processes in place. The advantage of EC2 is you can easily set up servers in multiple places across the globe (or at least across the country if you just want to host in the US.)

jshen• 15 years ago

and there are many downsides to EC2, dependence on a third party to a greater degree, shitty disk IO, instances go down more frequently than real servers, inflexibility with hardware (no SSDs), etc, etc.

There are co-lo provides that have data centers in different places.

c2• 15 years ago

Does anyone else think that Reddit's usage of EBS might be the culprit?

Looking at their past outage response:

http://blog.reddit.com/2010/01/why-did-we-take-reddit-down-f...

Money quote: "In response, we started upgrading some of our databases to use a software RAID of EBS disks, which gives drastically increased performance (at a higher cost of course)."

RAIDing EBS disks seems like a really really BAD idea. There is a non-trivial failure rate of any single EBS disk, and if you RAID them together, your failure rate of the RAID will consequently increase. Am I understanding that correctly?

If they fix that, could that be a 'silver bullet' to fix these outages?

magicofpi• 15 years ago

They certainly think there's a problem with EBS: "Since that last failure, we have been doing everything we can to move ourselves off of the EBS product. We're about half way there. All of our Cassandra nodes are now using only local disk, and we hope to have all of postgres on local disk soon."

cookiecaper• 15 years ago

Reddit's software is not very impressive. They should really focus on consolidating to a more manageable setup. I'm aware that big sites often have complex environment to facilitate scaling, but frankly, reddit is a big monstrous mess. It'd be a lot simpler to move hosts if they would get some of those extraneous dependencies removed and put some simplicity and sanity into their data storage, instead of the half Cassandra/half PgSQL thing they do now.

I wrote a long post about it here: http://www.deserettechnology.com/journal/reddit-the-open-sou... .

Andys• 15 years ago

I've discovered through a bunch of similar lessons that buying really good hardware and hosting yourself sometimes requires the least amount of your time of any option. It provides a stable base on which to build additional redundancy such as shared network storage or database replication if you wish (and this is generally how expensive "enterprise" solutions work).

In practice and on cheap hardware, networked storage is flakey and has umpteen failure modes. Database replication is even worse. Both require babysitting by developers or sysadmins and hours to repair when it goes wrong. What is the point outsourcing hardware and scaling to EC2 if you end up with even more work to monitor and keep fixing the infrastructure you build on top?

iamjustlooking• 15 years ago

Jedberg last year stating that a startup buying physical machines today is foolish: http://www.reddit.com/r/IAmA/comments/a2zte/i_run_reddits_se...

tesseract• 15 years ago

Reddit is not a startup anymore.

jackowayed• 15 years ago

I wouldn't expect them to buy hardware, but I definitely could see them going to a dedicated hosting service like Softlayer (where you still just pay monthly and can usually bring up more capacity fairly fast, but you get whole machines instead of virtualized machines).

It seems like that would make a lot more sense for Reddit, since I/O is so slow and flakey on EC2 (from what I hear), and it's not like they're really taking full advantage of the elasticity that EC2 provides (by massively scaling up and down to fit major load variance)

Natsu• 15 years ago

I just love how someone sent them a bacon pizza during the outage: http://imgur.com/dunO2

turbojerry2• 15 years ago

Has anyone else noticed the Amazon status page says there are still problems today but on the status history show no problems yesterday only on the 21st?

http://status.aws.amazon.com/?a

turbojerry2• 15 years ago

They have corrected it now.

Jedberg on Reddit's most recent outage

Related Stories