Google App Engine: Datastore Performance Growing Pains

http://googleappengine.blogspot.com/2010/06/datastore-performance-growing-pains.html

by adora • 16 years ago

33 10 16 years ago

googleappengine.blogspot.com

If I'm reading the current status graphs correctly they're regularly seeing 1.5-2.5 second queries and 400ms direct fetches. That seems unbearably bad. Can anyone running mission critical apps on the platform comment?

adora2• 16 years ago

Yeah, it's really, really slow -- and the ETA is 2 weeks! Although I have some faith it'll be fixed sooner.

We're thinking of porting over to EC2. If anyone knows of any reliable tool that supports this for production, please feel free to shout. Otherwise, we'll probably be building something ourselves...

cmelbye• 16 years ago

There's always AppScale. It supports EC2 and I think it's been officially recognized by Google. http://code.google.com/p/appscale/

If you write your own, would you consider open sourcing it?

axiom• 16 years ago

Yeah we've been freaking out about this. We have our app used in the classroom, and have had several sessions disrupted because of this. Needless to say our customers were pissed.

We're in the process of doing an emergency move over to EC2 because of these issues. That being said, I must admit that this is a really decent response on Google's part. Unfortunately the problems go deeper than just these latest problems with datastore.

richardw2• 16 years ago

What are your more permanent worries?

aarontcheung• 16 years ago

Just wondering, are you porting to mySQL, Cassandra, or HBase? We're currently moving off app engine also, and would love to hear more about your final setup.

chasingsparks• 16 years ago

I just started writing a GAE open-source community tool for rare disease orgs two weeks ago. I thought it was slow, but wrote it off as a limitation inherent to the Datastore. I moved a lot of things to the Memcache level, mitigating (mostly) the slow queries. That being said, I am glad it was actually a real problem.

I'm also impressed that they are not charging anyone until the problem is solved.

richardw2• 16 years ago

GAE does have some performance surprises even during normal performance periods, so it's good that you're using Memcache etc now.

Couple other ideas:

Try to minimize read effort as much as possible, so e.g. pre-calculate as much as you can and rather optimize for performance than data cost. Data is cheap, so de-normalise where you need speed. Calc offline, so add a task to calculate something and return immediately to the user.

jasonkester• 16 years ago

I like the way they're dealing with this. Acknowledge the problem (and actually call it a "problem" instead of handwaving aroung it), and stop charging for the service until they fix it. This is a good pattern to follow if you run a service that ever runs into issues that affect your users.

On the performance issue itself, I actually used AppEngine for a project when it first came out and was astounded at how poorly it performed in general. Any page requiring more than a single "select where id={}" query would regularly throw CPU warnings at you.

I couldn't imagine building a site even as straightforward as HN on top of it. As soon as you had a page needing a dozen queries to construct, you'd run into the wall.

Google App Engine: Datastore Performance Growing Pains

Related Stories