Most of Slicehost is Down

http://status.slicehost.com/2009/11/3/dfw-interruption-2

by smanek • 17 years ago

42 18 17 years ago

status.slicehost.com

catch232• 17 years ago

Yeah the rackspace outage freaked us out a little. Seems like it would be nice if they could notify the customers before the customers discovered the downtime themselves. If Rackspace emailed their customers at the first sign of downtime, customers would probably be more patient.

idlewords• 17 years ago

Without knowing the details of the incident, it's hard to blame them. Personally I prefer the first responders to focus all their attention on diagnosing the problem. Once other people arrive on the scene (which in a hosting company I imagine is not many minutes after pagers start going off) they can take charge of notification and posting updates.

gommm• 17 years ago

Yes but in that case they took at least 40 minutes before notifying customers and posting on their status blog... I had to contact them on the live chat to be able to know what was happening...

And they could have a script that directly posts to the status page something like "A lot of our servers are down, we are working on it please be patient" when they detect some problem wouldn't take long...

catch232• 17 years ago

Usually the best way to do customer service is to first admit the problem, then solve it. I'm sure not everyone who works at Rackspace is technical, so someone could initiate the email notification 5 minutes into the downtime, maybe that could be done via nagios on an external system even. If the system sees that 50% of the servers are down, notify all customers, regardless if you were directly affected or not.

human_v2• 17 years ago

I work for a web host (not this one). From my experience, this is generally what is going on during downtime (note: times are BEST CASE scenario):

T0: Servers go down T5-15: Someone realizes something is broken. T15-45: Initial someone calls another someone to tell them it's broken because the night guy probably can't fix it all by himself. T60-70: All the engineers are woken up and know OF the problem. T90: Engineers figure out WHAT the problem is. T120: Engineers fix problems. T150: More problems arise. T180: All problems are solved, no more customers are mad.

Moral of the Story: As you can see, it already is a very involved process to get servers running again. Now, as a customer, you want an email to notify you that services are down. First, the admin needs to find out exactly which customers are affected and then write up a nice happy email to you letting you know we're working on it. Do you want a happy email or do you want your servers fixed? Your call. Downtime is chaos. Do not ask to be notified of chaos.

theli0nheart• 17 years ago

Honestly, I don't need well-drafted prose or a happy email to alert me of a server outage. I could care less. They should have a pre-drafted document that let's them fill in blanks based on the problem at hand. I was one of these affected customers and can say that not being notified of the servers being down cost me a few hours of uptime. I would say that customer communication during chaos is nearly as important as fixing the problem itself. Just my two cents.

shpxnvz• 17 years ago

Do you want a happy email or do you want your servers fixed? Your call.

It's not like the two are mutually exclusive. Are you really suggesting that the entire 180 minutes of effort by multiple engineers would have to be directed at writing a "happy email" rather than fixing the problem?

The real reason, I imagine, is that if they automatically sent out an email to everyone who could be affected then more people would know of the problem. It's probably smart to accept a handful of customers being unhappy about being in the dark in return for the outage being exposed to far fewer customers.

patio11• 17 years ago

If you're wondering "Thats funny, I'm on Slicehost and yet the Ride of the Valkyries is not playing on my cell phone, how odd": Both of their St. Louis data centers, where many of their older customers are, were unaffected.

buro9• 17 years ago

Interesting.

I'm at St Louis, but yet I have new slices too (couple of months old) and they appear unaffected as well.

Are slices allocated in the same data center as existing customer slices? If so, that's neat (and helpful as I'm routing traffic between them).

notauser• 17 years ago

You can ask for slices in different places if you have more than one.

The outage took out my primary and secondary app servers (resulting in an outage/apology e-mail to customers, ouch) and so now I'm thinking about accelerating my move to a more decentralized system.

lanaer• 17 years ago

By default, yes, all slices you get are kept at the same data center. That way, if you have traffic between slices, they're on the same LAN and they don't have to charge you for that bandwidth. As has been mentioned, you can request a slice at a different data center, but it does have to be requested.

chrischen• 17 years ago

Yay my site is back up. If only people actually go there so it matters...

DavidMcLaughlin• 17 years ago

The service I'm getting from Slicehost has really went to hell ever since Rackspace took over. I had something like a year solid of uptime with them when I was actively writing in my blog and getting traffic from the front pages of HN and proggit.

Now my server freezes up all the time despite little traffic.. I haven't written in my blog since May and yet I've had my server require a hard reboot three times in the last three months. I get so little traffic that I don't really care, but my mail forwarder for my personal domain is sitting on the same server and now I'm really having to consider moving my mail forwarding off to some other service.

rufo• 17 years ago

As a counterpoint, I've noticed no difference in support or stability since the Rackspace merger - if anything I'd say there's been less downtime for my slices since then.

(EDIT: My slices are located in STL2, in case that's relevant.)

cmelbye• 17 years ago

Rackspace's datacenter is DFW.

jyothi• 17 years ago

I think it is back up. at least my server on hived.in is up.

percept• 17 years ago

Did anybody get a notification from Slicehost about this?

buugs• 17 years ago

Kind of interesting with the lack of communication when you look at comments like this http://news.ycombinator.com/item?id=906820 (left parent for context)

Most of Slicehost is Down

Related Stories