Google App Engine Broken For 4 Hours And Counting

http://www.techcrunch.com/2009/07/02/google-app-engine-broken-for-4-hours-and-counting/

by peter123 • 17 years ago

61 46 17 years ago

tybris• 17 years ago

Cloud outages may not be frequent, but they sure are noticeable.

peter1232• 17 years ago

This is worse than the 8-hr outage of S3 sometime ago... most apps could still respond without S3 static assets. If your entire app is hosted on AppEngine, you're screwed for 4 hrs and counting...

slig• 17 years ago

> This is worse than the 8-hr outage of S3 sometime ago

I don't think so. Javascript files hosted os s3 would hang the page loading and without css/images the app would be useless too.

peter1232• 17 years ago

But you could quickly recover by hosting those JS files yourself and relinking them. If your app is coupled tightly with AppEngine APIs, then there is nowhere else you can host your app.

sschueller• 17 years ago

At least we got a detailed explanation from AWS as to what happened and what was put in place to prevent it from happening again.

ezmobius• 17 years ago

multi-tenant architectures are the geocities of cloud computing. This is the main problem with something like gae, if they have an internal problem it takes down their entire cloud and all the apps with it.

grandalf• 17 years ago

which means that a lot more people are mad and it gets fixed sooner. Have you ever had a problem with cable tv or phone service that just effects your home? It can take weeks. If it effects a whole city, it is fixed within hours.

andrewljohnson• 17 years ago

I concur, my blog on App Engine is down.

jpeterson• 17 years ago

What's the point of hosting in a cloud if there's still a single point of failure like this? I realize that it's currently free, but I thought one of the main advantages of moving to the cloud was redundancy and fault tolerance.

jacquesm• 17 years ago

this is a bit like flying vs driving. If you're in the drivers seat you have control - you hope - and your destiny is in your own hands, if you're in a plane it is someone else driving (unless you are an airline pilot). The accident rate is lower for planes per mile flown but if it goes wrong then it usually does so in ways that make the headlines. Still, more people die driving than flying.

When the 'cloud' goes down (or at least some part of it) then you'll notice this immediately because of the large number of sites going down all at once. But when you compare it with the accumulated downtime of all those users had they not been 'cloud users' but hosted on their own kit then it is very well possible that the balance is still in favour of hosting in the cloud.

wvenable• 17 years ago

I do my own hosting, but I agree with this completely. Downtime at any scale is inevitable, but with Google's massive infrastructure you're far better off.

asmithmd1• 17 years ago

Great analogy. I would rather have all the big Brains at Google working to solve the problem instead of my puny brain working trying to restart my server

enjo• 17 years ago

More specifically:

We have no real network administrators. Within our engineering team we collectively have the skill to be an effective at systems administration, but the hardware side is really a complete mystery.

Co-location mostly solves that, but the cloud takes it a step further. By running in a virtualized environment we can handle what we're good at it and let others build complex data centers to scale our traffic.

When your bootstrapping a start-up, that's just huge.

skorgu• 17 years ago

The cloud is just an enormous SPOF. I like having my own kit somewhere in the loop if only to throw a proper error message.

gizmo• 17 years ago

A lot of sites get 99.99% uptime easily. It's not difficult when you know what you're doing and have plenty redundancy. When you depend on any cloud stuff you can't do any kind of graceful degradation. Google goes down, you go down. Google goes bankrupt, you go bankrupt.

Considering how cheap dedicated servers are moving a service to the cloud makes little sense (exceptions notwithstanding).

ryanvm• 17 years ago

The hardware may be inexpensive, but "knowing what you're doing" and being able to pull 5 nines is definitely not cheap.

arockwell• 17 years ago

99.99% is only 52 minutes of downtime a year... I don't think any of S3, Rackspace, Goole AppEngine, or even www.amazon.com have uptime that good this year.

Getting that kind of uptime is much harder than it sounds and for a lot of websites not worth the extra cost. How much money would pay to go from 99.9% uptime (~9 hours /year) to 99.99% (52 minutes)?

nickb2• 17 years ago

Nice analogy but like all analogies, it's too simplistic and flawed. You left out one important and critical part: the hypothetical passenger in your example is tied to a specific plane/airline. If you don't like your pilot or plane type, you cannot move to a different airline or request a different plane or a different pilot since you're chained to the specific plane.

Due to Google App Engine's API lock-in, you're stuck with them as a provider... quite possibly forever due to heavy BigTable dependency.

Even though I'm a huge fan of cloud computing, I'd rather use a strategy that uses platforms/planes that are built from reusable parts and allow you to switch your plane/airline provider as you please. Don't like Delta? Just go to AA counter and you don't have to change your luggage, clothing etc.

Until there's a second, GAE-compatible, ISV provider that offers full compatibility with GAE, I'd avoid GAE like a plague.

drcode5• 17 years ago

I'm sorry sir, but if my pilot is stuck in a thunderstorm and I don't think he knows what he's doing, I can't "request a different plane."

jshen• 17 years ago

all understanding of the real world is simplistic and flawed. Even your analogy because there is no good cloud as service provided that offers anything as good as big table for distributed storage plus map reduce.

rjurney• 17 years ago

Actually, if this thing works then there is no longer an API lock-in with Google App Engine: http://code.google.com/p/appscale/

Assuming you can still dump your data out of Googlage?

vorador• 17 years ago

Actually, as django abstracts the GAE-api, there's still a way to escape.

rapind• 17 years ago

I think a better analogy for cloud computing is electricity. For mission critical applications investing the time and effort into a power / app backup is probably a good idea, but the onus is on us.

vaksel• 17 years ago

exactly, shit happens, if you host on your own, you are just as vulnerable to power outages etc. + at least when a cloud goes down, they have hundreds of pros trying to fix it

imajes2• 17 years ago

hope not. too many cooks?

bryanwoods• 17 years ago

I'm so tired of reading about web servers/services going down.

jrockway• 17 years ago

You are supposed to say something clever like, "Too bad TechCrunch isn't hosted on App Engine..."

defied• 17 years ago

At least their communication is good: http://groups.google.com/group/google-appengine-downtime-not...

jbox• 17 years ago

I disagree. The GAE status page was down for hours: http://code.google.com/status/appengine

What's the point of having a status page it's only up as long as your service? We shouldn't have to hunt around a Google Group for information about what's going on.

drcode5• 17 years ago

hmm... My appengine site is running just fine

Update: Actually, my app is in the "read only mode" they described... the moment I tried to update anything it went to hell :)

fauigerzigerk• 17 years ago

The bad thing is we can do nothing but wait. The good thing is we don't have to do anything but wait ;-)

drcode5• 17 years ago

Now fixed again, as per my own app and as per http://groups.google.com/group/google-appengine-downtime-not...

drcode5• 17 years ago

Not sure why I was downvoted: It just got fixed a few minutes ago, as stated on the link and as per my own app.

drcode5• 17 years ago

I guess anything I post in this thread is going to be downvoted :)

Google App Engine Broken For 4 Hours And Counting

Related Stories