Amazon Aurora PostgreSQL Serverless

https://aws.amazon.com/blogs/aws/amazon-aurora-postgresql-serverless-now-generally-available/

by sahin • 7 years ago

123 30 7 years ago

aws.amazon.com

atonse• 7 years ago

Can anyone comment on how this compares to CitusDB? Probably is a lot more expensive?

How about things like support for PostGIS, etc? Is that moot since Aurora is mostly at the storage layer?

rbranson• 7 years ago

Citus is a completely different animal. This is just vanilla PostgreSQL with a different storage backend. No PostGIS either because just like RDS, no custom native extensions.

davnicwil2• 7 years ago

You can install PostGIS on a regular RDS Postgres instance - it's true that you can't install arbitrary extensions but there is a list of allowed ones, and PostGIS is one of them.

olefoo• 7 years ago

Citus is a fork of Postgresql and you would need to run it on it's own infrastructure.

AWS Aurora Postgresql regular flavor does support some of the popular extensions but that may not be the case with this flavor since it has some memory, image space and execution time limits that the other managed Aurora solution would not.

That said, being able to keep a pool of warm database clients at the ready for unpredictable write loads looks like it would be a winner and AWS has reliably improved to add requested features.

daurnimator• 7 years ago

Citus isn't a fork any longer, it's just an extension.

tedd4u• 7 years ago

Citus AFAIK doesn’t have scale to zero. Aurora serverless does.

Citus gives you horizontal write scalability, Aurora Serverless doesn’t. (There is a preview of multi-master Aurora for MySQL but I think that’s more of an HA active/active feature and not sharding — which Citus does)

Citus is available hosted in Azure (Hyperscale), hosted on AWS, and run-it-yourself (a couple flavors). Obviously Aurora serverless is only in AWS.

Microsoft acquired Citus last January.

malisper• 7 years ago

I can speak a bit to this. I previously lead the database team at Heap where we had one of the largest deployments of Citus with ~1PB of data.

Citus and Aurora Serverless are pretty different. The only thing in common is that they are both related to scaling Postgres. Citus makes it easy to horizontally scale Postgres. It let's you take a cluster of Postgres instances and make them appear as a single Postgres instance. This let's you scale Postgres to petabyte scale datasets (and beyond) because under the hood you are storing the data across multiple servers.

On the other hand, AIUI, the benefit of Aurora Serverless is the auto-scaling. It will automatically add or remove compute on an as-needed basis. This means you no longer have to worry about provisioning a Postgres instance of the right size. You can also save money depending on your usage pattern. Even though it will scale Postgres, you are still limited to a maximum DB size of 32TB. Aurora Serverless is probably a good fit if you are under this size, but once you pass 32TB, you will need something like Citus to scale further.

As for extensions, you are limited to the extensions Amazon provides. As you point out, Aurora is mostly at the storage layer. For that reason I think they could add most extensions if they wanted to. Each extension brings on some risk to them so they audit every extension before approving them. I've heard issues with extensions are one of the biggest sources of support requests.

mbesto• 7 years ago

Curios - what's the use case for a Serverless DB?

davnicwil2• 7 years ago

It says it's ideal for unpredictable load or infrequent demand. My reading is that billing is done per minute only while the database is being read / written to (active), but what I'm not sure about is if the storage is completely free whilst it's inactive. If that's the case it would seem like a great solution for BI and analytics type applications where you want to collect data over time (and are able to batch writes) and produce reports relatively infrequently.

013a• 7 years ago

The storage is definitely not free, even while inactive.

danenania• 7 years ago

A few I can think of:

- Your workloads are both bursty and mission critical. Returning errors while you scale up manually isn’t acceptable.

- You’re deploying self-hosted software. A serverless db makes it easier for your customer and your support team—capacity planning and scaling are much easier.

- You have a small team and want to minimize how much time you spend on ops.

northstar702• 7 years ago

Is there any scenario where you wouldn't want to use a serverless db? Makes me wonder why any db shouldn't be serverless in this data and age for the benefits that you mention.

icheishvili2• 7 years ago

Our experience with pg aurora was awful. I cannot recommend it to anyone. Our primary would run out of memory and fail over every few hours and all amazon support had to say is that it’s a known issue.

We ended up adding even more functionality to our audit trigger package (based on a SecondQuadrant fork, https://github.com/icheishvili/audit-trigger) to allow us to get off of aurora. AWS does a good job at locking you in.

nwah1• 7 years ago

Other downsides are that it is more expensive than Aurora MySQL and there's no automated upgrade tools.

js2• 7 years ago

Here's another recent report of a bad experience with a reply from the Principal Product Manager for Aurora PostgreSQL:

https://www.reddit.com/r/aws/comments/bv70k8/aurora_postgres...

jitl• 7 years ago

Came here to echo this. I’ve heard the same story from a few different companies that tried to use Postgres Aurora as their primary database: insurmountable production issues, limited response from Amazon, eventual migration away.

qatanah• 7 years ago

Very interesting audit trigger. Would like to try it w/ partition table instead of doing separate tables.

dstaley• 7 years ago

The biggest roadblock to me using Aurora Serverless is cold start times. Last time I checked it was over 20 seconds.

incadenza• 7 years ago

I experienced this with Lambdas behind a VPC, and yeah it's completely crazy. This is true with the DB offerings as well?

I don't see what the use case is then. Intermittent background jobs or something?

Untit1ed• 7 years ago

In my experience 20s is good sadly - quite often the cold start is longer than the max timeout for a lambda function, so my app always shows an error when woken up.

strin• 7 years ago

Aurora Database + auto-scaling #instances = Serverless??

cowsandmilk• 7 years ago

No, that was Aurora Autoscaling[0]. Is there an actual difference in implementation? Not sure, but there are many limitations in serverless not present with aurora auto scaling[1].

[0] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

[1] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide...

truth_seeker• 7 years ago

I wish it allowed to install community extensions which are not part of core software.

sudhirj• 7 years ago

At the risk of sounding like an AWS apologist, this is really great if you have a use case for it. Don’t use it if you don’t.

We’ve been waiting to use this for our dev and test environments - the devs and testers work irregular schedules, sometimes only a few hours a day and obviously 5 days a week. We currently have manual and semi automatic start and stop switches for those environments, can get rid of all that.

Some of our systems are low volume high value - they kick into high gear once or twice a week (not necessarily on a schedule, whenever customers feel like placing and order) at which point hundreds of thousands of queries need to be run. We currently run large or extra large DB servers 24x7 at 0.01% overall utilisation because of this.

Some of the services are used only during daytime at a single timezone, but once in a while there are a few night owls.

It’s perfect for use cases like this. Sure, if you compare it cost wise to a similar 24x7 instance it’ll be more expensive. Sure, it takes lots of seconds at for a cold boot. Both these things are not the point. The tool is what it is, and it’s clear about its shortcomings in the same way that a Gatling gun is clear that it has a non negotiable spin up time. That doesn’t mean it’s not useful, it needs to be matched to the work it can do effectively.

don_neufeld• 7 years ago

Add me to the chorus of those with severe Aurora Postgres regrets. We are in the process of shutting down a very large installation of it.

I strongly discourage use of this product. It has serious problems that serverless does not fix.

gmueckl• 7 years ago

Could you name some of these problems?

sidcool• 7 years ago

I have a basic question here. While RDS manages everything that we typically need, how is it different from the Serverless counterpart in the form of Aurora Postgres here?

iamed2• 7 years ago

In response to some of the complaints about Aurora PostgreSQL in this thread:

Aurora PostgreSQL Serverless should be considered a different product with different performance characteristics. We've been experimenting with Aurora PostgreSQL and have encountered the same memory issues mentioned here and elsewhere. It's impossible to autoscale read replicas based on memory usage, and there's no swap for temporary spikes. We haven't yet seen those problems with Aurora PostgreSQL Serverless. There is swap, which handles spikes, and whatever automatic scaling Amazon has implemented adequately deals with increased memory usage (if you're okay with a few terminated connections). I can't comment on the index-related issues, but I do suggest considering this product separately from Aurora PostgreSQL.

icheishvili2• 7 years ago

The index related issues themselves should be a show stopper. I couldn't believe it until I ran into other people saying the same thing, but having secondary indices on your tables slows writes down by something like an order of a magnitude.

I've yet to see an OLTP system that doesn't use secondary indices on tables. And if your workload is OLAP, there are far better relational tools.

Amazon Aurora PostgreSQL Serverless

Related Stories