Google App Engine Pricing Angers Developers, Kills PlusFeed

http://www.readwriteweb.com/hack/2011/09/google-app-engine-pricing-ange.php

by jzb • 15 years ago

223 143 15 years ago

readwriteweb.com

cd34• 15 years ago

1.5gb out with 880 frontend instance hours in 24 hours.

Someone needs to rethink their architecture.

trafficlight• 15 years ago

That works out to be about 36 instances each pushing 40 megabytes for the day. That's crazy.

marcc• 15 years ago

It's not fair to make assumptions like this. I have a Google App Engine app which doesn't push put a lot MBs each day, but does use a fair amount of CPU, and I'll pay for it. It's not a poorly designed service, it's a service which doesn't generate a lot of traffic (when measured in MB), but I'm getting value from it elsewhere. In this case, my app on GAE is an API which my front end (hosted elsewhere) is consuming.

tomjen3• 15 years ago

No he doesn't.

The problem is that google count each process of the python runtime as a separate instance and count them when they run only partial hours (amazon counts this way too, but you can run many, many python programs at the same time even on a micro instance).

rbanffy2• 15 years ago

I am currently playing with the idea of moving an application from all-GAE to part GAE, part EC2 with some spot instances running jobs when cheap enough. According to my sloppy math, this should reduce the load on the GAE side by at least 60%.

Anyway, this is all theoretical (in the worst sense of the word) - the app is not even public.

kaffeinecoma• 15 years ago

But is it really worth the headache of such a Frankenstein architecture?

I initially fell in love with the "no sysadmin" aspect of App Engine, and started building apps around it. Eventually I realized that (for me, anyway) the upside isn't really worth the trouble of having to contort my apps to work in Google's sandbox- can't run SQL, have to deal with datastore timeouts, CPU timeouts, etc. When you're done coding work-arounds for all of these things, are you really coming out ahead?

cellis• 15 years ago

This is so true. The first step to correcting a problem is admitting you have one, and everyone on App Engine clearly does (locked-in). What I'm doing about it? Moving to Tornado/MongoDB/EC2 as quickly as possible.

pbh2• 15 years ago

I went through this thought process. I really liked Python and auto-scaling, so GAE seemed really appealing. But the combination of lack of SQL and inflexibility of architecture led me to want to create the sort of combination architectures you describe with EC2 instances or whatever.

Ultimately, after a lengthy debate with a friend, I concluded that Heroku was a sufficiently better option that it was worth learning Ruby (which I've been writing more or less as an uglier Python anyway).

ww5204• 15 years ago

That is my current project's design. Web frontend stuff in GAE and AWS doing the backend processing. AWS is needed to run media servers which can't run in GAE. I was going with it despite the disadvantage of the split architecture because I liked GAE. Now with the new pricing model kicking in, the frontend cost has skyrocketed and it doesn't make sense to run frontend stuff in GAE.

Also the way Google handling this confirms my fear of proprietary lock-in, leaving a really bad taste in mouth. I meant I've spent lots of time to develop in their framework to work around their quirks and limitation. They have been advocating to developers to optimize for CPU time and that's what we did. Now they found out they don't make enough money with CPU optimization and change the whole billing scheme, screwing all the efforts done by developers. What would stop them to change again tomorrow if they find instance-hour is not making enough money for them?

gte910h• 15 years ago

I'm wondering if google is just trying to encourage an architecture which is less bad for their site.

I'm guessing a small minority of apps were doing things in a way that was eating up tons more resources than they were paying for. I bet for many apps, this could end up no worse or better.

catch236• 15 years ago

I don't think it's a small minority of apps. 100% of the people I know who are using GAE are seeing the minimum of a 4x price increase for their app (before the 50% discount). The price increase would actually prevent some small ISVs from reaching ramen profitability. There's a thread somewhere on the GAE google groups containing many angry users.

Smrchy2• 15 years ago

Even if most of my apps would increase by 10x they would still be cheaper than running a single AWS instance for them. Not even taking into account the easy of management and automatic redundancy that GAE offers.

buff-a14• 15 years ago

I'm seeing a 50x (5000%) $ increase and I wrote my app from scratch "by the GAE book". It almost seems as if those of us who put the effort in to do it the GAE way are worse off than those who just uploaded django.

buff-a14• 15 years ago

I wrote an app from scratch for GAE. Its cost is increasing 5000%. In fact the increase seems more than people who just slapped django on it.

wccrawford2• 15 years ago

I think we're missing a big part of this...

How many users were using it?

groks2• 15 years ago

More than 100,000: http://goo.gl/wiDj6

ddon• 15 years ago

We tried to use GAE several times on our heavy traffic production, and was just too slow. We contacted google several times, but never received any help from them. So, thanks to this, we don't use them and we are not affected but the price change.

edtechdev• 15 years ago

Yeah I still don't understand why the Khan Academy and other educational/open source developers are using Google App Engine when it suffers from vendor lock-in.

angryasian• 15 years ago

some of it yes, but if you write modular code then its a none issue. Doing Java development, GAE supports most Java EE and popular frameworks, so if done correctly porting would not be an issue. I can't say the same about Python stack.

Estragon• 15 years ago

Unless you're doing regular integration testing on an alternative platform, you're bound to inadvertently introduce some painful GAE-specific dependencies.

catch236• 15 years ago

It's still an issue -- you have all this data that you have to migrate off of app engine. Some of my friends who migrated on to GAE said it took them almost 2 full weeks to load data into GAE, and now I'm sure it would cost them at least a full month to load data off GAE. The code changes are probably the easy part of the migration.

kennystone• 15 years ago

His app generated no revenue, so any price other than trivial would have him jumping ship. Why does Google want apps like his?

robryan2• 15 years ago

The $2.63 a day comes out to $81 a month which for hosting is a non trivial amount. It does come down to whether Google can turn a profit at these levels for someone using this amount of resources but I don't think customers willing to pay $50-$100 a month are ones you want to push away.

wccrawford2• 15 years ago

$81/month for a fun project that gets you noticed and looks good on your resume is nothing to sneeze at. Blog about it, and suddenly you have a hidden revenue stream from it. Let people know its yours and suddenly you have some weight to throw around.

sasha-dv• 15 years ago

That's unreasonably expensive. For $68.46 he payed for a day you can have an "el-cheapo" dedicated box or a great VPS for a month. I still don't understand why people trade a bit of system administration and superb performance for a vendor lock and insane prices.

robryan2• 15 years ago

Yeah trading back a little bit of system admin you can get a pretty serious amount of server power on AWS for that amount of money.

jcheng2• 15 years ago

But it's 880 instance-hours per day. So assuming that "el-cheapo" dedicated box has similar performance characteristics to one GAE instance, then you'd need 37 of those boxes.

My question is why the heck 31.42 CPU hours turned into 880 frontend instance hours, and whether this is something a lot of other GAE apps are seeing as well.

jonknee4• 15 years ago

He has a lot of instances spun up waiting to do work. This is your choice, but if you want to be able to handle spikes without latency troubles, it's what you should do. None appear to be doing a lot of work in this case, so he should have definitely tried messing with the settings before closing shop.

ww5204• 15 years ago

Google's instance-hour is process-instance-hour. The current GAE Python can only serve one web request by one process. Multiple web requests coming in would spin up multiple processes. Besides the CPU, Google charges the app while the process is waiting for IO, like waiting for receiving/sending to the browser.

But measuring process-instance-hour is misleading. In a typical web box, spinning up an extra process takes very little resource since most process memory are shared with the parent process, and most web apps are IO bound, like waiting for network/file/DB, idling taking little CPU. In a server box, many processes can be crammed in (like the shared hosting box). But Google counts those duplicate processes as separate ones and charge the idle time these processes take. That's why you see the outrageous bill.

What's more insane is that Google charges the process-instance-hour like a machine-instance-hour. GAE charges $0.08/hr for process-instance-hour while AWS charges $0.02 to $0.08/hr for the whole machine.

ww5204• 15 years ago

And you basically trade sysadmin time with developer time in working around GAE's idiosyncrasy and proprietary API.

pbh2• 15 years ago

What's most interesting to me about this article is that the management of GAE seems to actually be getting worse over time.

GAE has always had two main disadvantages. First, there is vendor lock-in because you code specifically to the data store, worker API, and so on (though arguably there are alternative platforms that implement the GAE API). Second, you cannot run custom code (custom C in some virtual machine) or have a custom architecture (if, say, Redis might be useful to have around). These disadvantages probably aren't changing and are probably necessary for auto-scaling, security of Google's infrastructure, and so on.

However, there are lots of little things that GAE has been getting wrong for a while that are totally unnecessary. Lack of hosted SQL support. Lack of SSL for custom domains. Just little things that are probably annoying to implement and boring, but totally necessary for real websites or websites just gaining traction. (I know these are in varying stages of early support at the moment.)

But now, the GAE team almost seems to want to actively disappoint users. With hosted SQL being a request for years, Guido appears to have spent a bunch of time re-architecting the API for the datastore instead. With this pricing increase, they're pushing the many developers who came to their platform based on price (due to the very interesting scaling properties of the Google front-end) off the platform.

Overall, I'm very confused.

mcantelon• 15 years ago

Yeah, the hard limitation to request duration is pretty nasty too. It's a pretty limited platform overall, and never took aoff, so maybe they are trying to push people off it so they can eventually sunset it?

nroach• 15 years ago

Deliberately pushing people out? Probably not. They had a major decision point available when they chose to either exit beta and step onto a release product schedule or drop the project entirely. The fact that they chose to productize it bodes well for the immediate future. Adding SLA and SSL was also non-trivial from a product perspective.

But, that does mean that you have to figure out what your revenue-producing tenants are going to look like, just as you would do in physical real estate. Yes, it looks like the high-traffic commodity-product (think Halloween store) doesn't make a good tenant. But that doesn't mean that a jewelry store or office (low throughput, high value per-square foot) wouldn't be a good tenant.

skybrian• 15 years ago

SQL support is in progress: https://groups.google.com/forum/#!topic/google-appengine/l0D...

83457• 15 years ago

Can someone please explain frontend versus backend instance processing? Does backend here mean processing that takes place in data storage systems?

hung• 15 years ago

I believe frontend means requests handled through normal frontend hits from end users, while backend instances are for longer running background processes.

ChuckMcM3• 15 years ago

Heh. TL;DR version - Google starts recovering their costs, hits people in unexpected places.

So back when I worked there Google had no clue what it cost to run their infrastructure at a fine grain level. Sure they knew the aggregate cost, that was easy, but knowing on an application level didn't exist. This was a problem since as more and more things were using the machines, how did you "bill" a department for their machine usage? That really crystallized when the bottom fell out in 2008 and suddenly there was going to be no more new machines/data centers for a while and everyone had to 'make do.'

They mobilized an effort to figure this out, its not like it isn't knowable, and ever the data driven company the first signs of light were appearing just as I was leaving. It should not be a surprise but they discovered many things they did not previously believe was true, and I don't doubt it has driven a lot of change going forward. One of the more interesting outcomes was that projects/products were actually getting cancelled if they cost more to run than they could generate in revenue (I'm looking at you Goog-411)

So this knowledge is being applied to GAE, which is great, its also another way to back compute some of their operational efficiencies.

But that it costs money to run stuff? Well that isn't really news is it? That it costs that much? Well there is the whole if it doesn't make money it will get cancelled threat.

And the kicker is pricing out the scarce resource. It looks (and I've been gone over a year and half so I am speculating based on this move on their part) like their 'scarce' resource is web server front ends. (the labeled "Frontend instance") Traditionally they've been like most multi-tier web properties split between front end machines which host the web facing stuff, and back end machines that do the heaving lifting and storing. And by this change one can reason that residency on the 'front end' is more valuable than crunching in the 'back end.'

I'm guessing PlusFeed gets a lot of web traffic. So they spend a lot of time 'actively' on the front side, and from their numbers they do practically nothing on the back side. This fits well with the sudden massive price increase.

This gives you an insight into Google's business dynamics as well. Where page-views are the limiting resource, and computation is not. When you look at it that way, you can see that most of their 'revenue' has to be delivered through their front end services, and so consuming that resource reduces (potentially) their income. Hence the charge inconsistency.

Now contrast that to Billy-Bob's Web Farm (fictitious service) where every machine in their data center can be a web server, and front end serving is trivial, its all about the bandwidth. Their pricing would probably be more gigabytes transferred.

I would not be surprised at all if it is impractical to run such 'translation' services (basically all web traffic very little compute) on a hosted environment like Google's.

chubot2• 15 years ago

Your last point is very confusing. You're saying you think Google can't spin up more App Engine front ends if it wants? ("every machine in their data center can [not] be a web server") Or any other kind of front end?

phillmv• 15 years ago

I interpreted it more as saying that their architecture allows them to parallelize all sorts of 'back end computations' across their infrastructure, such that their scarcity is in internet-facing processing and not processing as a whole.

z92• 15 years ago

Probably he is saying that because their expertise and focus lies in backend, it's costlier for Google to increase frontend service and server compared with say, other web-hosting companies.

ChuckMcM3• 15 years ago

Google historically splits off their "front end" which is dealing with messages from the internet and their "back end" which is processing what those messages want to do. It was briefly described in the original search paper they presented.

Of necessity, most machines that Google runs have 'private' (as in not directly addressable) addresses. This is how it seems that everyone runs things when they get above a certain number of servers.

Thirdly because one can assume that 'attacks' (whether they depend on XSS, overflows, or what not) are coming from the Interwebz its prudent to present the smallest possible attack surface on a machine which is being fed internet stuff. As you can imagine, those security constraints make for uncomfortable restrictions on what you can do in the 'front end', so a machine that is 'front end' has a different set of security constraints than a machine that is 'back end' (they are still constraints there of course but having gone through the filter of the front end machine which has greatly attenuated the possible exploit vectors).

I believe it is this armored 'front end' resource which is the scarce resource, but again, I've been gone over a year and things change quickly inside the 'plex.

Locke16893• 15 years ago

I think I know what he's talking about. My LDAP is the same as the email in my profile. I'll wager a guess at what the real problem is.

buff-a14• 15 years ago

So they built a service called "App Engine" and it turns out that the "engines" that run my "app" are exactly the parts that Google doesn't have enough of, and so they are going to charge me uncompetitive amounts of money per day?

Lets suppose that google doesn't have enough machines. Do they:

  a) Massively increase the price to reduce customers?
  b) Massively increase the number of machines?

You're saying "a".

ChuckMcM3• 15 years ago

Actually no. I'm saying that they seem to have learned what their costs are and have priced their service based on those costs and what ever economic model they use. Given their pricing it suggested that "transactions" (which is to say a query from the web and its response) are either a good scalar for their cost to deliver, or are one of their scarcer resources.

It is entirely possible that they discover they can't make a business out of selling 'engines' like this, it wouldn't be the first time they decided they were leaving too much money on the table.

They chose "c" - scale the price so that AppEngine is revenue positive for the company using these constants for the various moving parts.

Clearly some customers will find that it no longer makes sense for them to use AppEngine. It doesn't say anything about whether the market reach will be sufficient to sustain that business.

iki23• 15 years ago

Yep, but one would expect Google being able to do better and get it work for lower cost at their scale and level of expertise, which is raising the doubt among developers.

Smrchy2• 15 years ago

I love using GAE and got 8 apps running currently. At first the new pricing model shocked me. But please take a 2nd look. Sure it's more expensive than the old model. But you can set the maximum number of idle instances in your Application Settings page. Just set it down so no more that X instances get spun up:

> The Idle Instances slider allows you to control the number of idle instances available to your application at any given time. Idle Instances are pre-loaded with your application code, so when a new Instance is needed, it can serve traffic immediately. You will not be charged for instances over the specified maximum. A smaller number of idle Instances means your application costs less to run, but may encounter more startup latency during load spikes.

There is another setting for latency:

> The Pending Latency slider controls how long requests spend in the pending queue before being served by an Instance. If the minimum pending latency is high App Engine will allow requests to wait rather than start new Instances to process them. This can reduce the number of instance hours your application uses, but can result in more user-visible latency.

So if you are fine with a little higher latency for your app then you can reduce your bill by a great deal. If you want all that GAE can offer with max. instances available and lowest latency you gotta pay - as you would when you run n instances at another cloud provider.

jcheng2• 15 years ago

Thanks for the info. Is there a setting for max number of simultaneous instances? If your app gets slashdotted are your potential costs completely unbounded?

(Just out of curiosity, I'm not a GAE customer)

crizCraig2• 15 years ago

There's a setting for the latency threshold causing a new instance to get spun up.

andymoe5• 15 years ago

Yes, there is.

angkec• 15 years ago

No there isn't. Just "max idle instances". So you'll get a huge bill if you get slashdotted.

mtogo2• 15 years ago

The problem is the price, though. Spinning up a couple EC2 micro instances to run your app is $0.02/hr per instance and you can run a few threads per instance. Appengine is $0.04/hr (double that soon), with one thread per instance.

wavephorm5• 15 years ago

Everything Google does, they do in a half-assed sort of way. And it's getting really annoying. Even their core business of search is showing signs of neglect. SEO experts have learned to game the system such that the quality of search results is pretty abysmal now.

GAE is a half baked AWS. Google+ is a half baked Facebook. Google Docs is a half baked MSOffice. They have no blood in these projects and don't really care whether they succeed or not, which coincidentaly means they probably won't.

Google is getting absolutely clobbered in every category other than search advertising dollars. Their products wreak of ambivilence and neglect, and I'm surprised anyone expected GAE to be a good platform.

jonknee4• 15 years ago

> Google is getting absolutely clobbered in every category other than search advertising dollars. Their products wreak of ambivilence and neglect, and I'm surprised anyone expected GAE to be a good platform.

Except for the largest business on the internet... Well search, video delivery, mobile phones, email, mapping, news, RSS, web analytics, and web browsers. Getting clobbered in almost everything else though.

wavephorm5• 15 years ago

They make no money on those products. They can't charge money for their products because they are half-assed knockoffs that nobody will pay anything for.

aaronf• 15 years ago

Our charges are going to be increasing from $0/day to $5-11/day. While bearable, it's a serious problem given so little notice, and disappointing since we invested in their infrastructure & optimized for their previous pricing plan. This is a serious hit for a bootstrapped startup getting off the ground. No doubt it will kill a lot of startups.

heliodor• 15 years ago

Can people list their choice for python hosting and the main benefits?

What are some good services layered on top of EC2 to provide more management and automation?

Does anyone have any experience with whether DotCloud is priced reasonably?

sirn• 15 years ago

I've been playing with ep.io for a while (nothing serious), it's a standard Python + PostgreSQL stack using few inis for configuration. Their pricing is a bit high on bandwidth though, so I'll probably wait for Heroku Cedar to officially support Python and use that instead.

igorgue2• 15 years ago

I hope PlusFeed's author read this:

Man that's some nasty code, horrible, don't open source that kind of "code"... it's bad, even for a 5 minute projects, it's really bad.

Open Source software is not a dumpster of your bad code.

thematt• 15 years ago

He wrote software that many people found useful and gave it away for free. That's awfully damn generous as far as I'm concerned. You see an improvement that can be made? Then make it. But stop complaining.

igorgue2• 15 years ago

I don't want to sound like a troll... but you're better off re-writing the whole thing... I mean look at it.

https://github.com/russellbeattie/plusfeed/blob/master/plusf...

Global variables galore, tabs no spaces, no documentation, compares to empty not just once but multiple times, pokemon style exception handling (you gotta catch them all!), 'is not None' lol, type checking instead of duck typing... Just to name a few.

14 pople forked it already, I hope those are Github bots.

andymoe5• 15 years ago

On the other hand to make a huge stink about a 400 line script that is poorly designed and takes advantage of none of the APIs provided by app engine to make things preferment like back end instances or task queues is a little disingenuous.

cHalgan2• 15 years ago

My understanding of all these GAE pricing story is that Google decided that GAE is not a strategic business and that business unit need to break even or they will be canceled. Does this make sense?

vessenes2• 15 years ago

Nope, Google doesn't work that way. No business units get P&L's, it's part of their overall financial strategy.

catch236• 15 years ago

If that were the case, wouldn't apps like google calendar get cancelled long time ago? I doubt the ads on google calendar are sufficient to sustain the number of users it has on a daily basis.

cHalgan2• 15 years ago

Maybe Google Apps (Calendar) are part of strategic vision (connected with Chrome OS)?

crizCraig2• 15 years ago

I've been developing for app engine since 2008 when it came out and absolutely love it. The price changes are a result of turning a successful and massively growing product into a profitable one a la search, youtube, etc... Google should be praised for this. The changes in price also accompany an SLA that guarantee developers will receive three years notice before a breaking API change or service shut down.

The SSL problem is a limitation in some browsers that causes the type of certificates that GAE needs to use a CNAME, not IP, based routing to display huge warnings.

andymoe5• 15 years ago

An alternate perspective: Google App Engine is still a fine platform even with the new price increases. (Which they told us were coming by the way)

And by the time the pricing takes effect the updated python runtime should bring costs down even more.

The instance costs are comperable to Heroku AND you get a high availability data store AND the ability to store really huge amounts of data in the blobstore AND a CDN for serving images from said blobstore. Not to mention background processes, task queue, XMPP, memcache, Multitenancy and multiple versions of apps so you can easily roll things back or test out updates painlessly.

Try and replicate that setup on Heroku or AWS for anywhere near the costs and time that you can get there with app engine.

While you're fighting with AWS and playing sysadmin or trying to think of ways to bring down the costs of Heroku's database services by using RDS instead or being nickel and dimed by add-on fees I'll be shipping code. Code that actually takes advantage of the platforms strengths.

georgemcbay2• 15 years ago

"An alternate perspective: Google App Engine is still a fine platform even with the new price increases."

This depends a lot on what you're doing. GAE is now pretty terrible if you're using it for high-bandwidth applications (say, a data proxy or similar). Especially when you compare it to other providers for whom bandwidth prices have been dropping (AWS, Linode, etc now offer incoming bandwidth totally free and the rates on outgoing are very low as well). If you're using it as a number-crunching backend with relatively small in/out datasets, the new prices aren't too bad at all.

andymoe5• 15 years ago

I agree that picking the right tool for the job is important but I just wanted to list the reasons I prefer GAE. There is quite a lot of outrage (raising prices does that) without really considering what the real costs of the alternative platforms are.

jamieb• 15 years ago

the new prices aren't too bad at all.

c1.xlarge on EC2 is $1.16/hour. How many of the $0.08 GAE units does it take to match one of those? How many of those GAE units can run optimized SSE4/AVX? (Answer: none of them). I've run 250 c1.xlarge units at a time. GAE wasn't even possible, let alone affordable.

socratic2• 15 years ago

I am really confused by your Heroku comparison.

The blobstore is basically S3, and the CDN for the blobstore is basically CloudFront. Both S3 and CF seem to me to have really simple APIs, plugins to Rails and other common frameworks, and are almost certainly battle tested by an order of magnitude more companies than the blobstore. What am I missing?

As far as I can tell, of your list, Heroku has all of the things you describe, with the possible exception of XMPP. Task queues (workers, SimpleWorker), many flavors of high availability data stores, memcache (or even better, Redis as a service), multitenancy, multiple versions of apps (git, releases).

Is there something bad about RDS that I don't know about? I had always assumed you just spun it up and could optionally replicate a MySQL instance a few times in different zones.

More broadly, it sounds like you had a really bad experience with Heroku. If so, what was it? I'm a bit curious because we are rather dependent on it.

andymoe5• 15 years ago

All I'm saying is with Heroku you have to clobber these features together and the real cost of that is higher than people may be admitting in these comparisons.

I've worked with both platforms and I'm not saying heroku is horrible. In fact it might be the easiest to get started with of all the PAAS providers but it's expensive (they all are) and does not provide the same level of features out of the box that GAE does. Saying that because I can eventually get all the features I need by clobbering together a bunch of random gems and bringig AWS into the mix is quite different than having a unified api out of the box.

The point I'm making is that there is an additional real cost this kind of integration work (just like there are real costs learning googles API and accepting a certain amount of lock in.) and I don't think these costs are being fairly represent in the current discussion.

davisml2• 15 years ago

It appears that the chart is being misinterpreted. The cost per hour is less under the new pricing and I believe the total hours is calculated for the month.

vessenes2• 15 years ago

Agreed. The number of instance hours is vastly different on the two charts; the price seems to be going from .10 / hr to .08 per hour (currently .04).

mtogo2• 15 years ago

That's the point. Appengine is changing their pricing from CPU hours to Instance hours, so an app that previously had 3-4 hours a day is now going to have 48-72.

buff-a14• 15 years ago

I wish that were true. Could there be some huge fuckup on the part of the kid programming the chart? I'm just going to keep clicking my heels and pretend that Google doesn't really want to take my monthly cost and charge me that per day.

nir• 15 years ago

I think Google just doesn't really get how unique GAE was. It was a fantastic platform for small apps, and inevitable some percentage of these would grow to big, paying apps.

Also (sorry for the armchair quarterbacking here, can't resist..) it was exactly what Google can do better than anyone - best server infrastructure + Guido Van Rossums - while stuff like Google+ is exactly what Google haven't a clue how to do.

switch• 15 years ago

This is amazing. You've hit the nail on the head. Google would be much more successful if it focused on being Google and didn't try to be Apple+Fecaebook+X companies all in one.

bane2• 15 years ago

In my mind, the problem is not so much the extreme pricing difference, but the change in what's being measured, so not only is there a price increase, but it's nearly impossible to figure out how to compare the old vs. new schemes vs. competitors.

The huge price jump is a serious problem, but the weird metrics switch makes it feel like such a bait and switch. I'm going to be highly surprised if this isn't challenged in court.

dasil003• 15 years ago

The strength of GAE is the magic scaling beans, but to take advantage of it you need to lock yourself in massively. It's probably not realistic to port an existing application that actually needs real scale to GAE given the complexity of most apps by the time they reach that point. Therefore, the key funnel for them is new apps with hopes of becoming truly massive. Fortunately for Google way more people dream of scaling than actually will, but they need to A) not scare poor startups away with the price today and B) not scare them that they're going to get bent over and raped on price changes later.

Personally I'll never touch GAE with a 10' pole simply because of the support issue and perpetual-beta-culture uncertainties.

maxent• 15 years ago

I think this comment from Wesley Chun on the GAE team is key:

https://groups.google.com/d/msg/google-appengine/obfGjbIkOTI...

"your app can be slashdotted or tweeted by demi moore -- http://adtmag.com/blogs/watersworks/2010/10/mobile-app-creat... -- or perhaps you may need to build/host something on the scale of both the royal wedding blog and event livestream with traffic numbers that are mindblowing -- http://googleappengine.blogspot.com/2011/05/royal-wedding-be... ... these are the reasons for using App Engine. it was not meant as free/cheap generic app-hosting but to provide a premium service that's difficult to get elsewhere in the market. if you're just after the former, there are plenty of options for you."

My take-away is that GAE is hard to justify unless your usage pattern is unpredictable and spike-y. I'm taking the long weekend to give dotcloud a serious test-drive.

systems• 15 years ago

DotCloud is not cheap if your application is database intensive.

georgemcbay2• 15 years ago

I still don't quite understand their market, and honestly I don't think they do either.

If you're expecting insane traffic to begin with, it makes more sense to create your own systems using a more standard stack and not tie yourself to GAE. If you're just some small to medium sized site that gets "slashdotted or tweeted by demi moore", I guess it is nice that your site will automatically scale to serve the millions of unexpected incoming users, but when you're filing for personal bankruptcy due to the unexpected and infinitely scaling GAE bill that comes along with that, how much consolation is the fact that your server stayed up during the rush going to be?

buff-a14• 15 years ago

Here's my complaint:

I, and many others, spent a lot of time figuring out how to write apps that do it the "app engine way":

  * Fast completes (30 second timeout)
  * Offloading to task queues when you cant
  * Channels
  * Blobstore two-phase upload urls
  * Mail eccentricities

We did so, because we believe Google when they told us If you write your apps in this really weird way then we will be able to give you scale and cost benefits that you wont be able to get elsewhere

We believed them, because it seemed reasonable. We laughed at those who complained that django would hit the 30-second limit: "Its not a general hosting! Figure out the App Engine way!" And we educated on how to do it right, and many were happy.

Well, it turns out that it is general purpose hosting, with all of the costs, and yet also with all of the (once rational, now bullshit) idiosyncrasies.

---

But that's not the biggest complaint. The biggest complaint is that when my friends and peers objected to App Engine, its strange requirements and its potential lock in, they were right and I am a fucking naive idiot. And I really don't like to be proven a naive idiot. I put my faith in Google's engineers and they have utterly destroyed my credibility. THIS more than anything is the cost to me.

greendestiny• 15 years ago

Yeah to me this decision is either an admission that the idea of hosted apps is a failure or that the people making the decisions at google don't understand app engine.

hga• 15 years ago

I feel your pain, but that said:

From what else I've read, it sounds like engineers who didn't also wear green eye-shades (or good enough ones, or who didn't possesses or use good enough crystal balls) set up this debacle. And it was people wearing green eye-shades (who we can sincerely hope are also engineers) who aligned it with reality. Causing way too many people way too much pain.

Object lesson: if you're going to sell a service for cash money to others, paying close attention to your costs from the very beginning is not optional.

buff-a14• 15 years ago

The problem is how does one get from 31 cpu-hours to 879 instance-hours.

You might be thinking that in the original measure they did something insane like measure only user time of a process, or only when its executing a request, not booting or whatever (or fuck, I don't know because honestly there is no reasonable explanation). That is to say, that the 31 cpu-hours is a misread, and if the fellow in the article ran his code on EC2, he really would need 879 EC2 instances that day.

But this is not my experience. An extreme example: my app that served 14 pages was rated as taking 0.02 cpuhours, or 720 cpu seconds. This is entirely reasonable, if not excessive (because looking at the app it only took about 200 seconds including warmups). Under the new system, it is claimed that these 14 pages will require 2.8 instance hours.

  0.02 => 2.8
  31 => 879

So when the author of the article is told his app is going to take 879 instances hours per day, there is something seriously fucked up and wrong. It doesn't mean that the guy is running a realtime raytracing server. It means that GAE is horribly, amazingly, inefficient.

The app in the article serves 1.5gb/day and takes 879 instance-hours. What server would you need to do that on EC2: 1mb/s? The hourly cost on GAE is $1.46. Can I do that on a $0.085 EC2 instance? Yeah, I think so.

EDIT: My figures were wrong as I was comparing a $16 (wrong) figure to a $0.8 EC2 figure. The actual figure is $1.46, not $16. So I looked at the bandwidth/cpu numbers to see if a $0.8 EC2 instance is what is required, and I don't believe that it is. I think a $0.085 instance would be enough. YMMV.

ww5204• 15 years ago

Amen bro. The biggest frustration was we followed Google's preaching and spent lots of time to fit our apps to their restricted model, then have them turned around to destroy it. I don't have faith to do another round of optimization to fit their new restricted model. What is stopping them to change the ground rule again. I rather spend time to develop to a generic model that I have more control.

wisty3• 15 years ago

I'd recommend GAE to people who are prototyping - it's easy to do simple stuff in.

But mostly, GAE doesn't make sense for larger apps. You can't buy your way out of trouble, by putting your db on a dedicated server with fast drives and tonnes of RAM. You can't really use relational data without performance and reliability issues.

It's not just about the "app engine way". It's not like learning C or Haskell, and having to find a new way to write the code. You fundamentally cannot do big ad-hoc database operations.

And consider this - it was July last year that they introduced the MapperAPI. Before then, I don't think you could do Map-Reduce without manually re-implementing it yourself (on top of the cantankerous Appengine Datastore). Just think about that for a minute - how were you meant to do stuff the Appengine way without map-reduce?

Anyway, I don't think your credibility was "utterly destroyed". It was really hard to know whether or not the learning curve was worth climbing until you had tried. You just had to judge the book by its cover, and the "Google" brand is pretty compelling to an engineer. It's not the first time someone has been fooled into buying something because the provider has a good reputation.

wmf• 15 years ago

I would argue that all the forced scaling in App Engine makes prototyping harder. You can't use SQL. You can't reuse whatever open-source components you find on GitHub. You can't just let your app run slow and optimize later.

buff-a14• 15 years ago

I have several ideas that would scale well on GAE and neither want not need a relational db. I'm not making do without SQL, I'm actively not using it and its very successful. Even when I move to EC2, I still wont be using SQL. In fact I have only one idea that needs any kind of relational data and that is so relational that SQL is a bad fit too. EDIT: Actually, sorry, all my data is relational. Its just that I put the (small) effort in to figuring out how to make it work without joins. In the last 15 years, I've not done a single project that could not have been done with a NoSQL database.

In fact if you look at the recent comments of certain GAE engineers, they seem to believe that GAE is precisely for scaling, and that's why it now costs so much: its only for the big boys.

The problem is that I can never become one of the "big boys" on their system, because pretty much as soon as I get any traction, I have to move to EC2 or heroku or go broke. Their new found belief in the scalability of their system is just arrogance. Anyone can claim to handle lots of traffic when you require that your customers run 20 times as many frontends as they should reasonably need.

Good luck to them in the Royal Wedding market.

EDIT: on the nature of "relational"

andypants• 15 years ago

Actually, I think it's still not possible to perform map-reduce. The mapper api only is just for map, no reduce.

ja27• 15 years ago

I put my faith in Google's engineers and they have utterly destroyed my credibility.

Come to a Microsoft event some time and let a Silverlight developer buy you a sympathy beer.

angusgr2• 15 years ago

Can anyone tell me how the new and old prices compare to running a similar app on AWS, Heroku or other competitors?

ie was GAE ridiculously cheap before compared to other options, and now comparable? Or was it somewhat cheaper than competitors before, and now somewhat more expensive?

I realise it's never that simple, but nearly everyone's complaints seem to be (understandably) given in relative terms of before vs. after. I'd be interested to know how it stacks up before vs. after vs. if-we'd-taken-another-route.

buff-a14• 15 years ago

It was cheap by comparison. Now it appears to be orders of magnitude more expensive. I base this on the fact that their billing estimator is claiming that 1 cpu-hour on the old system equates to 100 instances-hours on the new system. Its not the $0.04/hour cost that worries me - that is a competitive number - its that their instances only appear to be able to do the work of a 80286.

angusgr2• 15 years ago

Thanks. :)

herf• 15 years ago

It sounds to me like the scarce resource was found to be RAM, not CPU. If an "instance" uses a big piece of RAM to serve a request, and doesn't give it up while waiting for a backend, then scalability is RAM-bound, not CPU-bound, and they should probably charge that way.

nicksdjohnson3• 15 years ago

Hi folks,

I'm on the App Engine team, and I just wanted to clarify one thing: The main difference between CPU hours and Instance hours is that CPU hours are charged based on CPU usage, while instance hours are based on wallclock time. The high ratio between the two you can see with PlusFeed is because it's spending a lot of time to serve each request, most of which is spent doing nothing - likely because it's doing outgoing HTTP requests.

Previously, we had no way to account for apps like this, that take a lot of wallclock time but very little CPU time, and as a result we couldn't scale them well. Under the new model, the charges reflect the real cost here - memory pressure. Every second an instance sits around waiting is a second that the memory occupied by that instance can't be used to serve other requests.

As others have pointed out, we're in the process of launching Python 2.7 support - it's currently in Trusted Tester phase - which will support multiple concurrent requests, and services like PlusFeed are likely to be able to take great advantage of that, reducing their instance hours by a large factor. Likewise, doing asynchronous URLFetches (where that's practical) can cut a huge amount off instance time.

buff-a14• 15 years ago

First of all, thank you for chiming in, here, on HN. Your presence might also be welcomed on the google forum, since I've read most of the posts there and no one has managed to answer this question.

If memory pressure is the issue, how are the trusted testers finding their memory pressure when they have a whole number of in-flight requests? If the PlusFeed fellow got 2.7 working, we'd expect to see 100/3.5%= 28 in-flight requests. Do you have data on how big the base memory vs per-thread memory requirements of these apps are? Python isn't famous for freeing up memory.

Which is to say, do you have any solid numbers that tell us that when we switch to 2.7, you wont have exactly the same memory pressure and either have to up the instance-hour cost, or start charging for ram, or just limit in-flight request to 3 or 4 so that our costs are only 5 times as much instead of 20?

Bottom line: what we all woke up to is that fact that as of right now:

* you set the price of an instance, and

* you get to decide how many instances I'm going to pay you for

Some of us are thinking that while that was a great idea when engineers were in charge, its not such a great idea now the bean-counters have taken over.

nicksdjohnson3• 15 years ago

That's a good question. I can't point to published figures, since the 2.7 runtime is still fairly new, but I can say that based on both my personal experience and based on fairly basic reasoning, the per-thread memory overhead is definitely a lot less than what's required by the whole instance. The entire of the Python standard library, along with your framework and other libraries, are shared overhead between all the threads.

The issue with charging by CPU hour was that you could occupy memory-seconds as much as you wanted without charge; that's no longer the case - by charging for instances, we're implicitly charging for the memory they use.

As far as determining how many instances you run - you can do this to a large degree, both by setting budget limits, and by setting scheduler parameters.

mark_l_watson• 15 years ago

A question for anyone on the AppEngine team that may also be useful for other developers:

I have an app that I want to deploy and control my costs. I would like to pay for 1 FE instance to always be running and limit temporary FE instances to a maximum of 1 (free?) instance when needed. I expect my app to have a relatively small number of users, but they will be active.

I would like to prevent the scheduler from ever spawning more than these 2 FE instances. Occasional unavailability when the site is busy is OK. Except for bandwidth and storage, I would like to know roughly what my costs will be.

Just to be clear, how do I set this up?

allad• 15 years ago

I also have an app on GAE. http://www.tubesmix.com Right now there's no charge as there's only a couple hundred users and the resources used are low. But this change worries me a lot. Even more so since all my backend code is tied to GAE infrastructure. This is very disappointing coming from Google. Since when did they become so nickel-and-dimey?!

Google App Engine Pricing Angers Developers, Kills PlusFeed

Related Stories