All 37signals Apps Were Down

http://status.37signals.com/

by mef • 13 years ago

71 47 13 years ago

status.37signals.com

johnernaut• 13 years ago

Painful yet interesting. I'm looking forward to the post-mortem on this.

mikeevans• 13 years ago

Looks like a load balancer issue: http://status.37signals.com/status/campfire/2013/4/17

masonhensley2• 13 years ago

I'm not too familiar with load balancers. I'm slowly learning rails server stuff that's more complicated than heroku.

Why would all of their applications be sitting behind a single load balancer? Wouldn't you want to silo them so that you do not have a single point of failure for all of your products?

banachtarski2• 13 years ago

All roads lead to a single URL (aka IP address) so somewhere along the way, you're going to have a single point of failure that does a one to many multiplexing, logically speaking.

wlll• 13 years ago

I'm a sysadmin at 37signals (was working during this outage).

The load balancers are meant to be redundant, and they are meant fail over automatically when something goes wrong. We're talking to the vendor about why this failed.

facorreia• 13 years ago

They have a very impressive status dashboard. Clear, informative and dynamic. It refreshed itself while I was watching with progress updates.

thecodemonkey• 13 years ago

It's also responsive! Looks great on mobile.

mccolin• 13 years ago

Agreed! It's one of the best status pages I've seen for a service (or set of services) like theirs. It's almost as if it's the AWS status pages run through a prettification machine.

Scottopherson• 13 years ago

Heroku's page is also great: https://status.heroku.com/

oit• 13 years ago

One of the best status page!

jbail• 13 years ago

I'm wondering how the red and green indicators look to someone who is color blind. Yes, the page looks nice, but I think Apple's status page is better simply because the indicator icons are (I'm assuming) better for people with color blindness.

Anyone who's color blind want to weigh in?

http://www.apple.com/support/systemstatus/

pc86• 13 years ago

How common is color blindness? I'm not saying it shouldn't be a consideration, I've just heard it mentioned a lot and I'm curious if it's much more common than I had originally though (low single-digit percentages, higher among males)?

DigitalJack3• 13 years ago

I'm red-green colorblind (deficient would probably be a better term).

What I've heard fairly often is the percentage ranges from 8-12% of males.

josephruscio• 13 years ago

It's ~7-10% amongst males depending on where you source your number.

evilduck• 13 years ago

Total color blindness (i.e. no functional cones, inability to distinguish any colors) is pretty rare, but some form of color vision deficiency is pretty common.

http://en.wikipedia.org/wiki/Color_blindness#Epidemiology

rasur• 13 years ago

At my last company, the two other guys I worked with were both red/green colourblind. Luckily I'm not, so I could weigh in with opinions..

It seems quite common actually - I've met quite a few people that suffer it (not that this is anything other than anecdotal info, but...)

DigitalJack3• 13 years ago

It will vary some between colorblind individuals. I can see the difference, both at apple and 37 signals. I do appreciate when people use a texture/hashmarks or shapes to differentiate as well.

I struggle immensely with the bitcoinity charts though. red, green, look nearly indistinguishable on the depth chart to me. And when I can tell a difference, I couldn't tell you if it was red or green. Just that it's a different shade than another bar.

MartinCron• 13 years ago

The mnemonic I use for making color distinctions as clear as possible for as many people as possible is to just remember the orange/blue portal colors from Portal. Those work for most kinds of color-blindness.

nwh• 13 years ago

If you were red/green colorblind, of the most common type (Protanopia), it would look similar to this — http://i.imgur.com/QoGzeBf.png

Photoshop has a proof tool which includes the two most common form of colorblindless. It is incredible how many interfaces become totally useless when you turn it on.

dokuda• 13 years ago

If you don't have Photoshop, there's a free app that let's you simulate your screen with 3 different forms of colorblindness.

http://colororacle.org/

manojlds• 13 years ago

I thought your second link was an example of a broken interface. You might want to change the line a bit.

mountaineer• 13 years ago

Interesting that the "RSS" status link goes to rss.atom and is in fact an atom feed. RSS has more meaning I guess.

dude3• 13 years ago

I posted this code from here on Basecamp: http://stackoverflow.com/questions/6529728/html5-doctype-put...

Answer number one. It came in with the html with the stackoverflow profile pictures and all. I was like crap so I switched to non html formatting. I probably clicked a few times, it kept loading and loading. Basecamp went down right when that happened.

Could have been the problem...

It looks like it actually posts to basecamp servers to do the reformat. Why can't they just do this in javascript?

I am using Basecamp Classic with Firefox.

mhartl• 13 years ago

This is actually good PR for 37signals. We already knew that all apps go down from time to time. What we now know is that (a) 37signals has a great status page, (b) they are transparent about their uptime, and (c) that uptime is in the 99.98–98.99% range.

jebblue• 13 years ago

Google Apps was down this morning too, wonder if there are any similarities.

All 37signals Apps Were Down

Related Stories