Upgrading Uber's MySQL Fleet

https://www.uber.com/en-JO/blog/upgrading-ubers-mysql-fleet/

by benocodes • 2 years ago

236 189 2 years ago

tiffanyh2• 2 years ago

Why upgrade to v8.0 (old LTS) and not v8.4 (current LTS)?

Especially given that end-of-support is only 18-months from now (April 2026) … when end-of-support of v5.7 is what drive them to upgrade in the first place.

https://en.m.wikipedia.org/wiki/MySQL

gostsamo• 2 years ago

Several compelling factors drove our decision to transition from MySQL v5.7 to v8.0:

Edit: for the downvoters, the parent comment was initially a question.

johannes12343213• 2 years ago

Since direct upgrade to 8.4 isn't supported. They got to go to 8.0 first.

Also: 8.0 is old and most issues have been found. 8.4 probably has more unknowns.

tallanvor• 2 years ago

According to the article they started the project in 2023. Given that 8.4 was released in April 2024, that wasn't even an option when they started.

hu34• 2 years ago

The upgrade initiative started somewhere in 2023 according to the article.

MySQL 8.4 was released in April 30, 2024.

Their criteria for a "battle tested" MySQL version is probably much more rigorous than the average CRUD shop.

paulryanrogers6• 2 years ago

Considering several versions of 8.0 had a crashing bug if you renamed a table, waiting is probably the right choice.

pizza2342• 2 years ago

I suppose they opted for a conservative upgrade policy, as v8.4 probably includes all the functional additions/changes of the previous v8.1+ versions, and moving to it would have been a very big step.

MySQL is very unstable software - hopefully this will be past - and it's very reasonable to go for the smallest upgrade steps possible.

hu34• 2 years ago

MySQL is very unstable software

I've worked on 20+ projects using MySQL in consulting career. Not once stability was a concern. Banking clients would even routinely shut down radom MySQL nodes in production to ensure things continued running smoothly.

As I'm sure users like Uber and Youtube would agree. And these too: https://mysql.com/customers

Unless you know something we don't and we're just lucky.

EVa5I7bHFq9mnYK• 2 years ago

With all the migration code already written and experience gained, I imagine upgrading 8->8.4 would take 1/10 of effort of 5.7->8.0.

PedroBatista• 2 years ago

MySQL 8 and beyond has been riddled with bugs and performance regressions. It was a huge rewrite from 5.7.

8 has nice features and I think they evaluated it as stable enough to upgrade their whole fleet to it. I'm pretty sure from 8 to 8.4 the upgrades will be much simpler.

cbartholomew• 2 years ago

They started in 2023. v8.0 was the current LTS when they started.

whalesalad• 2 years ago

So satisfying to do a huge upgrade like this and then see the actual proof in the pudding with all the reduced latencies and query times.

hu34• 2 years ago

Yeah some numbers caught my attention like ~94% reduction in overall database lock time.

And to think they never have to worry about VACUUM. Ahh the peace.

anonzzzies5• 2 years ago

Yeah, until vacuum is gone, i'm not touching postgres. So many bad experiences with our use cases over the decades. I guess most people don't have our uses, but i'm thinking Uber does.

InsideOutSanta3• 2 years ago

As somebody who has always used MySQL, but always been told that I should be using Postgres, I'd love to understand what the issues with VACUUM are, and what I should be aware of when potentially switching databases?

brightball• 2 years ago

There are always tradeoffs.

tomnipotent3• 2 years ago

MySQL indexes can contain references to rows in the undo log and has a periodic VACUUM-like process to remove those references, though no where near as impactful.

edf13• 2 years ago

3 million queries/second across 16k nodes seems pretty heavy on redundancy?

withinboredom• 2 years ago

That's 200 qps per node, assuming perfect load balancing.

sgarland9• 2 years ago

I was going to say, that's absolutely nothing. They state 2.1K clusters and 16K nodes; if you divide those, assuming even distribution, you get 7.6 instances/cluster. Round down because they probably rounded up for the article, so 1 primary and 6 replicas per cluster. That's still only ~1400 QPS / cluster, which isn't much at all.

I'd be interested to hear if my assumptions were wrong, or if their schema and/or queries make this more intense than it seems.

pgwhalen• 2 years ago

assuming even distribution

I don't work for Uber, but this is almost certainly the assumption that is wrong. I doubt there is just a single workload duplicated 2.1K times. Additionally, different regions likely have different load.

620gelato2• 2 years ago

2100 clusters, 16k nodes, and data is replicated across every node "within a cluster" with nodes placed in different data centers/regions.

That doesn't sound unreasonable, on average. But I suspect the distribution is likely pretty uneven.

candiddevmike• 2 years ago

Does Uber still use Docstore? I'd imagine having built an effectively custom DB on top of MySQL made this upgrade somewhat inconsequential for most apps.

geitir• 2 years ago

Yes

gregoriol3• 2 years ago

Wait until they find out they have to upgrade to 8.4 now

gregoriol3• 2 years ago

And also all the passwords away from mysql_native_password

sgarland9• 2 years ago

They've got until 9.0 for that, it just gives deprecation warnings in 8.4.

johannes12343213• 2 years ago

Which one should do anyways. mysql_native_password is considered broken for a out ten years. (Broken for people who can access the hashed form of the password on the server)

rafram• 2 years ago

Did they have ChatGPT (re)write this? The writing style is very easy to identify, and it’s grating.

OsrsNeedsf2P• 2 years ago

The writing style is very easy to identify,

Really? At n=1 the rate seems to be 0

m4r1k• 2 years ago

Uber's collaboration with Percona is pretty neat. The fact that they've scaled their operations without relying on Oracle's support is a testament to the expertise and vision of their SRE and SWE teams. Respect!

tiffanyh2• 2 years ago

Aren't they using Persona in lieu of Oracle.

So it's kind of the same difference, no?

paulryanrogers6• 2 years ago

Word on the street is Oracle contracts are expensive and hard to cancel, like a deal with the devil. Not sure if their MySQL support is any different than Oracle DB itself

paradite• 2 years ago

I can tell from a mile away that this is written by ChatGPT / Claude, at least partially.

"This distinction played a crucial role in our upgrade planning and execution strategy."

"Navigating Challenges in the MySQL Upgrade Journey"

"Finally, minimizing manual intervention during the upgrade process was crucial."

notinmykernel• 2 years ago

Agree. Repetition (e.g., crucial) in ChatGPT is an issue.

brunocvcunha• 2 years ago

I can tell just by the frequency of the word “delve”

traceroute662• 2 years ago

I can tell from a mile away that this is written by ChatGPT / Claude, at least partially.

Whilst it may smell of ChatGPT/Claude, I think the answer is actually simpler.

Look at the authors of the blog, search LinkedIn. They are all based in India, mostly Bangalore.

It is therefore more likely to be Indian English.

To be absolutely clear, for absolute avoidance of doubt:

This is NOT intended a racist comment. Indians clearly speak English fluently. But the style and flow of English is different. Just like it is for US English, Australian English or any other English. I am not remotely saying one English is better than another !

If, like me, you have spent many hours on the phone to Bangalore call-centres, you will recognise many of the stylistic patterns present in the blog text.

brk• 2 years ago

I agree (I've posted a similar comment in the past and collected a handful of downvotes). Much like ChatGPT, you tend to see a slight over use of more formal and obscure words and a tone that tends to feel like the topic being discussed is being given just a touch too much focus or dedication relative to the grand scheme of things. It is hard to fully describe, more of a "you know it when you see it".

calmoo2• 2 years ago

There's nothing that sticks out to me as obviously Indian English in this blog post. It's almost certainly entirely run through an LLM though.

620gelato2• 2 years ago

(Speaking as an Indian engineer)

Hate to generalize, but this has less to do with "Indian style" but rather adding a lot of fluff to make a problem appear more complex than it is, OR maybe someone set a template that you must write such and such sections, despite there not being relevant content. [ Half the sections from this article could be cut without losing anything ]

In this case, the _former_ really shouldn't have been the case. I for one would love to read a whole lot more about rollback planning, traffic shifting, which query patterns saw most improvements, hardware cost optimizations, if any, etc.

excitive• 2 years ago

Can you elaborate on the last part? What are some stylistic patterns that are different when something is written by a US author v/s Indian?

kaeruct• 2 years ago

ChatGPT says "While it's plausible that a human might write this content, the consistent tone, structure, and emphasis on fluency suggest it was either fully or partially generated by an LLM."

gurchik• 2 years ago

How would ChatGPT know?

godshatter• 2 years ago

That sounds like regular old English to me. I could see myself saying all those things without thinking it's pushing any boundaries whatsoever. I'm starting to fear that LLMs are going to dumb down our language in the same way that people feared that calculators would remove our ability to calculate mentally.

aster0id• 2 years ago

Because the authors are likely non native English speakers. I'm one myself and it is hard to write for a primarily native English speaking audience without linguistic artifacts that give you away, or worse, are ridiculed for.

rand_r• 2 years ago

I know what you mean, and you’re probably right, but there’s a deeper problem, which is the overuse of adjectives and overall wordiness. It’s quite jarring because it reads like someone trying to impress rather than get an important message across.

Frankly, ChatGPT could have written this better with a simple “improve the style of this text” directive.

Example from the start:

MySQL v8.0 offered a compelling proposition with its promise of substantial performance enhancements.

That could have just been “MySQL v8.0 promised substantial performance improvements.”

mannyv3• 2 years ago

Once ChatGPT puts in "we did the needful" we're all doomed.

greenchair• 2 years ago

Dear sir, we are having a P1 incident, Prashant please revert.

John23832• 2 years ago

Anyone else get a "Not Acceptable" response?

internetter• 2 years ago

I did but it worked on a private tab

menaerus• 2 years ago

Lately there's been a shitload of sponsored $$$ and anti-MySQL articles so it's kinda entertaining that their authors are being slapped in their face by Uber, completely unintended.

greenie_beans6• 2 years ago

yall should prioritize your focus so you can do better at vetting drivers who don't almost kill me

lenerdenator3• 2 years ago

Their focus is prioritized according to what returns maximum value to their shareholders.

greenie_beans6• 2 years ago

beep boop i'm a capitalist robot

pretty sure safe travels is critical to maximum value to their shareholders (aka stfu or tell me how this blog post has anything to do with maximize shareholder value https://www.uber.com/en-JO/blog/upgrading-ubers-mysql-fleet/... ... shareholder value is a dumb ass thing to prioritize over human life)

photochemsyn• 2 years ago

It's undeniable that the worst drivers on the road are those working for ride-hailing services like Uber. It's a big point in Waymo's favor that their automated vehicles behave predictably - Uber drivers are typically Crazy Ivan types doing random u-turns, staring at their electronic devices while driving, blocking pedestrian walkways and bike lanes, etc.

JamesSwift• 2 years ago

I'm not sure how effective the database engineers are going to be at solving this, but I guess we can ask them to try...

greenie_beans6• 2 years ago

thanks for your help

jauntywundrkind• 2 years ago

Having spent a couple months doing a corporate mandated password rotation on our services - a number of which weren't really designed for password rotation - happy to see the dual password thing mentioned.

Being able to load in a new password while the current one is active is where it's at! Trying to coordinate a big bang where everyone flips over at the same time is misery, and I spent a bunch of time updating services to not have to do that! Great enhancement.

I wonder what other datastores have dual (or more) password capabilities?

johannes12343213• 2 years ago

I can't answer with an overview on who got such a feature, but "every" system got a different way of doing that: rotating usernames as well. Create a new user with new password.

This isn't 100% equal as ownership (thus permissions with DEFINER) in stored procedures etc. needs some thought, but bad access using outdated username is simpler to trace (as username can be logged etc. contrary to passwords; while MySQL allows for tracing using performance_schema logging incl. user defined connection attributes which may ease finding the "bad" application)

donatj2• 2 years ago

Interestingly we just went through basically the same upgrade just a couple days ago for similar reasons. We run Amazon Aurora MySQL and Amazon is finally forcing us to upgrade to 8.0.

We ended up spinning up a secondary fleet and bin log replicating from our 5.7 master to the to-be 8.0 master until everything made the switch over.

I was frankly surprised it worked, but it did. It went really smoothly.

takeda• 2 years ago

AFAIK the 8.0 release is one where Oracle breaks compatibility. So anyone considering MariaDB needs to switch before going to 8.0, otherwise switching will be much more painful.

paulryanrogers6• 2 years ago

IIRC it wasn't a big break from 5.7 unless you used some odd grouping or relied on its old+lossy defaults.

Though the single threaded performance fell off a cliff, tanking my CI performance.

Maria DB evolved very differently. I'm not sure how they stack up performance wise.

sandGorgon• 2 years ago

so how does an architecture like "2100 clusters" work. so the write apis will go to a database that contains their data ?

how is this done - like a user would have history, payments, etc. are all of them colocated in one cluster ? (which means the sharding is based on userid) ?

is there then a database router service that routes the db query to the correct database ?

bob1029• 2 years ago

I imagine it works just like any multi-tenant SaaS product wherein you have a database per customer (region/city) with a unified web portal. The primary difference being that this is B2C and the ratio of customers per database is much greater than 1.

ericbarrett• 2 years ago

A query for a given item goes to a router*, as you said, that directs it to a given shard which holds the data. I don't know Uber's schema, but usually the data is "denormalized" and you are not doing too many JOINs etc. Probably a caching layer in front as well.

If you think this sounds more like a job for a K/V store than a relational database, well, you'd be right; this is why e.g. Facebook moved to MyRocks. But MySQL/InnoDB does a decent job and gives you features like write guarantees, transactions, and solid replication, with low write latency and no RAFT or similar nondeterministic/geographically limited protocols.

* You can also structure your data so that the shard is encoded in the lookup key so the "routing" is handled locally. Depends on your setup

denysonique• 2 years ago

Why didn't they move to MariaDB instead? A faster than MySQL 8 drop-in replacement.

evanelias3• 2 years ago

While it is indeed often faster, it isn't drop-in. MySQL and MariaDB have diverged over the years, and each has some interesting features that the other lacks.

I wrote a summary of the DDL / table design differences between MySQL and MariaDB, and that topic alone is fairly long: https://www.skeema.io/blog/2023/05/10/mysql-vs-mariadb-schem...

Another area with major differences is replication, especially when moving beyond basic async topologies.

aorth• 2 years ago

Wow, I hadn't realized that MySQL and MariaDB diverged so much! In the last year I've started seeing some prominent applications like Apache Superset and Apache AirFlow claiming they don't support—or even test on—MariaDB at all.

dweekly• 2 years ago

Am I the only one who saw "delve" at the top of the article and immediately thought "ah, an AI generated piece"? Well, that and the over-structured components of the analysis with nearly uniform word count per point and high-complexity but low signal-to-noise vocabulary using phraseology not common to the domain being discussed. (The article doesn't scan as written by an SRE/DBA.)

devbas• 2 years ago

The introduction seems to have AI sprinkled all over it: ..we embarked on a significant journey, ..in this monumental upgrade.

xyst• 2 years ago

I wonder if an upgrade like this would be less painful if the db layer was containerized?

The migration process they described would be less painful with k8s. Especially with 2100+ nodes/VMs

remon6• 2 years ago

Their entire setup seems somewhat suspect. I can't think of any technical justification for needing 21k instances for their type of business.

zemo• 2 years ago

upgrade clients and testing the application logic, changes to the queries themselves as written, the process of detecting the regression and getting MySQL patched by percona, changes to default collation ... all of these things have nothing to do with whether the instances are in containers and whether the containers are managed by k8s or not.

shakiXBT• 2 years ago

running databases (or any stateful application, really) on k8s is a mess, especially at that scale

meesles• 2 years ago

A pipe dream. Having recently interacted with a modern k8s operator for Postgres, it lacked support for many features that had been around for a long time. I'd be surprised if MySQL's operators are that much better. Also consider the data layer, which is going to need to be solved regardless. Of course at Uber's scale they could write their own, I guess.

At that point, if you're reaching in and scripting your pods to do what you want, you lose a lot of the benefits of convention and reusability that k8s promotes.

jcgl• 2 years ago

it lacked support for many features that had been around for a long time

Care to elaborate at all? Were they more like missing edge cases or absent core functionality? Not to imply that missing edge cases aren’t important when it comes to DB ops.

__turbobrew__• 2 years ago

I can tell you that k8s starts to have issues once you get over 10k nodes in a single cluster. There has been some work in 1.31 to improve scalability but I would say past 5k nodes things no longer “just work”: https://kubernetes.io/blog/2024/08/15/consistent-read-from-c...

The current bottleneck appears to be etcd, boltdb is just a crappy data store. I would really like to try replacing boltdb with something like sqlite or rocksdb as the data persistence layer in etcd but that is non-trivial.

You also start seeing issues where certain k8s operators do not scale either, for example cilium cannot scale past 5k nodes currently. There are fundamental design issues where the cilium daemonset memory usage scales with the number of pods/endpoints in the cluster. In large clusters the cilium daemonset can be using multiple gigabytes of ram on every node in your cluster. https://docs.cilium.io/en/stable/operations/performance/scal...

Anyways, the TL;DR is that at this scale (16k nodes) it is hard to run k8s.

remon6• 2 years ago

Impressive numbers at a glance but that boils down to ~140qps which is between one and two orders of magnitude below what you'd expect a normal MySQL node typically would serve. Obviously average execution time is mostly a function of the complexity of the query but based on Uber's business I can't really see what sort of non-normative queries they'd run at volume (e.g. for their customer facing apps). Uber's infra runs on Amazon AWS afaik and even taking some level of volume discount into account they're burning many millions of USD on some combination of overcapacity or suboptimal querying/caching strategies.

Jgrubb2• 2 years ago

See, the problem is that the people who care about cost performance and the people who care about UX performance are rarely the same people, and often neither side is empowered with the data or experience they need to bridge the gap.

bushbaba• 2 years ago

Hardware is cheap relative to salaries. It might take 1 engineer 1 quarter to optimize. Compare that to a few thousand per server.

nunez• 2 years ago

Didn't realize their entire MySQL data layer runs in AWS. Given that they went with basically a blue-green update strategy, this was, essentially a "witness our cloud spend" kind of post.

pocket_cheese• 2 years ago

They're not. Almost all of their infra was on prem when I worked there 3 years ago.

aseipp• 2 years ago

Dividing the fleet QPS by the number of nodes is completely meaningless because it assumes that queries are distributed evenly across every part of the system and that every part of the system is uniform (e.g. it is unclear what the read/write patterns are, proportion of these nodes are read replicas or hot standbys, if their sizing and configuration are the same). That isn't realistic at all. I would guess it is extremely likely that hot subsets of these clusters, depending on the use case, see anywhere from 1 to 4 orders of magnitude higher QPS than your guess, probably on a near constant basis.

Don't get me wrong, a lot of people have talked about Uber doing overengineering in weird ways, maybe they're even completely right. But being like "Well, obviously x/y = z, and z is rather small, therefore it's not impressive, isn't this obvious?" is the computer programming equivalent of the "econ 101 student says supply and demand explain everything" phenomenon. It's not an accurate characterization of the system at all and falls prey to the very thing you're alluding to ("this is obvious.")

0cf8612b2e1e• 2 years ago

Simple enough just to think about localities and time of day. New York during Tuesday rush hour could be more load than all of North Dakota sees in a month. Even busy cities probably drop down to nothing on a weekday at 3am.

Twirrim• 2 years ago

They're not on AWS. They use on-prem and are migrating to Google and Oracle clouds.

https://www.forbes.com/sites/danielnewman/2023/02/21/uber-go...

remon6• 2 years ago

It's sort of funny how can you immediately tell it's LLM sanitized/rewritten.

aprilthird2021• 2 years ago

Let's delve into why you think that

fs0c13ty00• 2 years ago

It's simple. Human writing is short and to the point (either because they're lazy or want to save the reader's time), yet still manages to capture your attention. AI writing tends to be too elaborate and lacks a sense of "self".

I feel like this article challenges my patience and attention too much, there is really no need to focus on the pros of upgrading here. We reader just want to know how they managed to upgrade at that large scale, challenges they faced and how the solved them. Not to mention any sane tech writers that value their time wouldn't write this much.

Starlevel004• 2 years ago

every section is just a list in disguise, and gpts LOVE listts

blackenedgem• 2 years ago

I'm enjoying the replys to this not getting that it's a joke

msoad• 2 years ago

Yeah, I kinda stopped reading when I felt this. Not sure why? The substance is still interesting and worth learning from but knowing LLM wrote it made me feel icky a little bit

greenavocado• 2 years ago

Scroll to the bottom to see a list of those who claimed to have authored it

jdbdndj• 2 years ago

It reads like any of those tech blogs, using big words where not strictly necessary but also not wrong

Don't know about your LLM feeling

exe34• 2 years ago

I always thought 90% of what management wrote/said could be replaced by a RNN, and nowadays LLMs do even better!

est31• 2 years ago

It contains the word "delve", a word that got way more popular in use since the introduction of LLMs.

Also this paragraph sounds a lot like it has been written by LLMs, it's over-expressive:

    We systematically advanced through each tier, commencing from tier 5 and descending to tier 0. At every tier, we organized the clusters into manageable batches, ensuring a systematic and controlled transition process. Before embarking on each stage of the version upgrade, we actively involved the on-call teams responsible for each cluster, fostering collaboration and ensuring comprehensive oversight.

The paragraph uses "commencing from" together with "descending to". People would probably write something like "starting with". It shows how the LLM has no spatial understanding: tier 0 is not below or above tier 5, especially as the text has not introduced any such spatial ordering previously. And it gets worse: there is no prior mention of the word "tier" in the blog post. The earlier text speaks of stages, and lists 5 steps (without giving them any name, but the standard term is more like "step" instead of "tier").

There is more signs like "embark", or that specific use of "fostering collaboration" which goes beyond corporate-speak, it also sounds a lot like what an LLM would say. Apparently "safeguard" is also a word LLMs write very often.

remon6• 2 years ago

Nah this isn't a big word salad issue. The content is fine. It's just clearly a text written by humans and then rewritten by an LLM, potentially due to the original author(s) not being native speakers. If you feel it's natural English that's fine too ;)

maeil• 2 years ago

This [1] is a good piece on it. Here's [2] anorher good one.

We don't just carry out a MySQL upgrade, oh no. We embark on a significant journey. We don't have reasons, but compelling factors. And then, we use compelling again soon after when describing how "MySQL v8.0 offered a compelling proposition with its promise of substantial performance enhancements", just as any human meatbag would.

[1] https://www.latimes.com/socal/daily-pilot/opinion/story/2024...

[2] https://english.elpais.com/science-tech/2024-04-25/excessive...

l5870uoo9y• 2 years ago

AI has a preference for dividing everything into sections, especially "Introduction" and "Conclusion" sections.

1f60c• 2 years ago

I got that feeling as well. In addition, I suspect it was originally written for an internal audience and adapted for the 'blog because the references to SLOs and SLAs don't really make sense in the context of external Uber customers.

cheema33• 2 years ago

it's LLM sanitized/rewritten

LLM is the new spellchecker. Soon we'll we will wonder why some people don't use it to sanity check blog posts or any other writing.

And let's be honest, some writings would greatly benefit from a sanity check.

bronzekaiser• 2 years ago

Scroll to the bottom and look at the authors Its immediately obvious

karthikmurkonda• 2 years ago

I don't get it. Why is it so?

lawrjone• 2 years ago

Yeah I found this really off putting: it’s not possible for you to have several goals that are all ‘paramount’, and the word ‘seamless’ adds nothing in every place it appears!

I wish it didn’t turn me off the content as much as it does but it’s very jarring.

anitil2• 2 years ago

It was hard to read in places because of that, I need to work out a reverse prompt to make it clearer

jeffbee3• 2 years ago

File under "things you will never need to do if you use cloud services".

martinsnow• 2 years ago

Nah random outages because the RDS instance you were on decided to faceplant itself, or the weird memory to bandwidth scaling AWS has chosen will make you pull your hair out on a high traffic day.

It's just different problems.

jeffbee3• 2 years ago

The company in the article is doing < 200qps per node. Unless they are returning a feature-length video file from every query, they are nowhere near any hardware resource limits.

paxys• 2 years ago

At Uber's scale they are a cloud service.

mannyv3• 2 years ago

That's not true. The RDS 5.7 instances are EOL so you have to upgrade them at some point.

At least in RDS, that will be a one-way upgrade ie: no rollback will be possible. That said, you can upgrade one instance at a time in your cluster for a no-downtime rollout.

jeffbee3• 2 years ago

Hosted MySQL is not what I meant. That just means you're paying more to have all the same problems. The kind of cloud service I am alluding to is cloud spanner, cloud bigtable, dynamodb.

vivzkestrel• 2 years ago

Anyone has any ideas why Uber doesn't use PostgreSQL?

hu34• 2 years ago

https://eng.uber.com/postgres-to-mysql-migration/

cyberax• 2 years ago

They switched from PG to MySQL because they need to update highly concurrent tables, and PG created tons of bloat as a result. MySQL uses locking instead of optimistic concurrency.

drastic_fred• 2 years ago

Wow! Less than 200qps per node, quite a redundancy. They also mentioned its read heavy. With aws dynamo db it would be at least 10 times cheaper for their given workload. Although, i do not know how much data they store and serve or how they are additionally (analytics?, etc) using this fleet for.

anitil2• 2 years ago

I wonder why they did a large version jump in one shot (v5.7->v8) and didn't do incremental upgrades (v5.7 -> 6.x etc)?

I wonder because the promotion of the secondary v8 node to primary is a breaking change in this path, whereas in an incremental upgrade it might not have been. But I also understand at this sort of scale things might be as easy as that.

Upgrading Uber's MySQL Fleet

Related Stories