Notes on the Amazon Aurora Paper

https://blog.the-pans.com/amazon-aurora/

by uvdn7 • 7 years ago

139 42 7 years ago

I don’t understand how aurora achieves the speed it does with a log based approach. Can someone please clarify?

dmlittle3• 7 years ago

Last time we benchmarked Aurora (~2 years ago) the write speed of Aurora is pretty slow compared to RDS (Postgres RDS was able to achieve 3x write throughput)

danielecook2• 7 years ago

I never did a thorough benchmark but I was working with a poorly indexed DB (actually...no indexes) that had millions of records and despite the lack of indices the database still queried quickly.

misframer• 7 years ago

What was the concurrency like in those benchmarks?

PetahNZ• 7 years ago

Anecdotally after migrating from RDS MySQL to serverless Aurora there was a noticeable slowdown of our dashboard and reporting tools. Our typical workloads (ecommerce transactions) are slightly slower on average, but the peaks seem down.

wgjordan2• 7 years ago

I don’t understand how aurora achieves the speed it does with a log based approach. Can someone please clarify?

Aurora splits out 'database' nodes (the server instances you provision and pay for) from 'storage' nodes (a 'multi-tenant scale-out storage service' that automatically performs massively-parallel disk I/O in the background). Instead of MySQL writing various data to tablespaces, redo log, double-write buffer, and binary log, Aurora sends only the redo-log over the network to the storage service (in parallel to 6 nodes/3 AZs for durability).

No need for extra tablespace, double-write buffer, binary-log writes, or extra storage-layer mirroring, since durability is guaranteed as soon as a quorum of storage nodes receives the redo-log. The reduced write amplification results in 7.7x fewer network IOs per transaction at the 'database' layer for Aurora (vs standard MySQL running on EBS networked storage, in the benchmark described in the paper), and 46x fewer disk IOs at the 'storage' layer [1].

[1] https://www.allthingsdistributed.com/files/p1041-verbitski.p...

gigatexal2• 7 years ago

That’s an impressive savings in io

wgjordan2• 7 years ago

But here in the paper, it compares Aurora to a MySQL synchronous mirroring setup, which is really unfair, IMO. Why the active primary has to replicate everything over the wire to the active standby? Doesn't MySQL's replication support Statement-Based Replication?

The comparison in the paper is against the synchronous-mirroring pattern underlying RDS Multi-AZ [1], which is the existing fully-managed, high-availability MySQL solution Amazon offers, so the direct comparison is appropriate for existing RDS Multi-AZ users considering a migration to Aurora.

[1] https://aws.amazon.com/blogs/database/amazon-rds-under-the-h...

aptxkid• 7 years ago

Thanks! That makes much more sense!

ddorian433• 7 years ago

Everything is faster when you move your data to ebs \s.

Though google has done this for all databases (at least their internal ones) (even bigtable). And some companies say they can do it for nvme & ram.

You just need real-good networking and some type of asic and removing kernel from the path.

ddorian433• 7 years ago

For something open-source like this, look at tikv(transactional key-value (rust)) + tidb(query layer (go)).

matharmin• 7 years ago

Separating the storage layer from the database and query layer seems to be the future of databases - there's no reason for every database to re-implement the storage and replication layer just to provide a different query language or data model. This is also what makes FoundationDB so attractive for me. Are there any other large projects doing this?

Artemis2• 7 years ago

CockroachDB is made up of several layers to implement SQL and transactions on top of a distributed key-value store: https://www.cockroachlabs.com/docs/stable/architecture/overv...

TiDB has a similar architecture: https://www.pingcap.com/docs/architecture/

gigatexal2• 7 years ago

The TiDB approach is a lot more like Aurora though as you have much finer grained controls over how many storage and query nodes you have.

qaq• 7 years ago

Depends on the use-case you can look at performance of say ClickHouse vs alternatives which separate storage layer. The performance difference is fairly significant.

jandrewrogers2• 7 years ago

People often greatly underestimate the performance loss incurred by separating the storage layer from the execution layer. The difference is an integer factor. In a high-performance database design, the execution scheduling behavior must be tightly and intricately coupled to the I/O operation scheduling behavior or you lose much low-hanging throughput optimization.

That said, this can be solved by changing the level of abstraction. If you tightly coupled the execution and storage engine as a single module, which is not intrinsically dependent on the type of data model, it would allow high-performance multi-use within reasonable constraints. Albeit with a very different API than a storage engine. But that is not a thing that exists in open source AFAIK.

The loss from separating storage and execution may not matter for databases where operational efficiency is not paramount (e.g. because the scale is too small for it to matter) but many database engine designers are not going to take that use case limiting performance hit because the CapEx/OpEx implications are large.

ralusek2• 7 years ago

I think the point is that now that we're in the era of distributed/cloud/partitioned databases, the query logic cannot necessarily be colocated with the storage, so might as well get the advantages of decoupling.

The performance gains at the distributed level seem to rely on either parallelized compute (a-la MapReduce) or over-indexing (a-la ElasticSearch/GIN) rather than the sort of efficiency gains that were made at the level of optimizing between cpu/cache/memory/disk. I think modern problems regarding distributed databases are concerned with the time complexity not exploding as the size of the data does.

evanweaver• 7 years ago

If you tightly coupled the execution and storage engine as a single module, which is not intrinsically dependent on the type of data model, it would allow high-performance multi-use within reasonable constraints. Albeit with a very different API than a storage engine.

This is what we are doing with FaunaDB; Fauna's own native semantics are essentially a unified intermediate representation of a variety of high level query languages, forthcoming.

LoSboccacc• 7 years ago

CosmoDb has a similar approach but still a long way to go

senand• 7 years ago

Datomic does the same. Storage, query and writing are completely decoupled.

See more https://docs.datomic.com/on-prem/storage.html#storage-servic...

BrentOzar• 7 years ago

Microsoft’s new Azure SQL DB Hyperscale does something very similar:

https://www.brentozar.com/archive/2019/01/how-azure-sql-db-h...

I sketched out a lot of architecture diagrams in that, showing how it differs from traditional RDBMS’s, and I tried to keep the post approachable for database folks from different platforms.

mirlyn86• 7 years ago

It's too bad it doesn't perform very well. Was looking forward to an Aurora-like SQL Server.

I hope they fix it

Arqu• 7 years ago

In my eyes, Google pioneered the commercial application of this approach. The advent of BigTable and it's underlying colossus storage engine they literally pushed the advent of HDFS and all the BigData tooling from a decade ago.

necubi2• 7 years ago

Indeed, but it's not a new idea. This is Bigtable's architecture after all, which is now 15 (?) years old. There are a bunch of advantages: DB developers can focus on the DB, not the storage layer (which is in many ways much harder to get right), you can scale storage and processing independently, you can easily support different tiers of storage (SSD/HDD).

But by far the biggest advantage is operational. Your DB is now stateless! Nodes can die and be replaced in seconds, because they don't store any data; this makes designs like Bigtable's much more practical. You can quickly scale up the processing to deal with spikes without spending hours re-replicating. Operating the storage layer is still inherently challenging, but just has to be figured out once for all your DBs and other processing systems.

While performance can take a hit, this can be mitigated by local caching and good internal networking. It isn't able to meet the lowest levels of latency (sub ms), but in practice it works great for web applications.

ddorian433• 7 years ago

Bigtable was always colocated on the data server. If it failed, you had to have nodes with bigtable having the same ranges of data.

Though later it went stateless.

eecc• 7 years ago

Funny how the CQRS circle closes: it is an architectural design that overhauls some concepts of database design (the redo log, views) into the application layer (for some, to the extreme of reducing the database to a simple log.) And now this same separation is appearing in a DBMS design :)

tirumaraiselvan• 7 years ago

As per my understanding, CQRS fits here if you consider the database log as the "event source". But wouldn't it be terrible performance wise to create the "view" from the event source rather than using the storage layer.

ricardobeat• 7 years ago

I think there’s a parallel with how apps keep a trailing materialied view of the data instead of re-creating from the log, unless necessary.

ralusek2• 7 years ago

Given this architecture, do we know why Aurora caps at 64TB max size?

kakwa_• 7 years ago

I'm wondering how aurora would behave with queries mixing read and write.

For example, with something like "INSERT INTO my_table(field1, field2, field3, ...) SELECT f1, f2, f3... from root_table WHERE ...", I'm wondering how it will spread the load across nodes.

benmmurphy• 7 years ago

One of the neat things about aurora is there is a very small slave lag between the primary and read replicas. If you only have a single thread doing replication then you can easily have a situation where a multi-cpu master can get ahead of a slave only using a single thread. Lazily materializing the pages gives aurora 'parallel replication' for free. I think Mysql supports parallel replication now but I think it is limited if transactions conflict [https://mariadb.com/kb/en/library/parallel-replication/].

Another thing I noticed with Aurora is the incremental cost of storage for Aurora is extremely cheap compared to the cost of storage in EBS or the cost of storage if you used instance storage in EC2. The Aurora storage is 0.1/GB but this is replicated 6 times which is more than EBS and EBS costs the same 0.1/GB. This also might be why no-one is going to build a similar system to Aurora. It will be hard to sell something like Aurora to cloud users because the storage costs are going to more than what EC2 charges.

anentropic2• 7 years ago

dumb question which maybe has a well-known answer: has anyone done the same thing but for postgres instead of mysql?

a-robinson• 7 years ago

Amazon - https://aws.amazon.com/rds/aurora/details/postgresql-details...

anentropic2• 7 years ago

aha cheers!

LOL

Notes on the Amazon Aurora Paper

Related Stories