Migrating Uber's ledger data from DynamoDB to LedgerStore

https://www.uber.com/blog/migrating-from-dynamodb-to-ledgerstore/

by gronky_ • 2 years ago

328 335 2 years ago

uber.com

drexlspivey2• 2 years ago

Uber migrated all its payment transaction data from DynamoDB and blob storage into a new long-term solution

No way they have 1 trillion transactions right?

rco87862• 2 years ago

1T "records". Any given transaction can have N records. I'm assuming this includes Uber Eats as well.

drexlspivey2• 2 years ago

Still, they have 10B rides in 2023 including Eats, say 75-100B since inception. What would be a record such that each transaction needs 10-15 on average?

bjornsing21• 2 years ago

The blog post says billions of transactions and trillions of indexes (or rather index entries I presume), if I remember correctly.

sha_r_roh• 2 years ago

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

bjornsing21• 2 years ago

Payments would not likely be a long-term bet as well

How so? It’s a pretty ubiquitous problem…

kondro2• 2 years ago

The estimate sounds suspiciously similar to just the data storage component of DynamoDB. 1.7PB of data and indexes is about $5.1m/year in DynamoDB storage at list.

sakjur• 2 years ago

Supporting that, Uber’s blog post linked from the article mentions cost savings as a benefit from going from three systems to one, and doesn’t really mention any dollar figure afaict.

https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...

bachmeier2• 2 years ago

I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance

I'm guessing they know a lot about their costs, and you know very little. There's little value in insulting the team members like this.

whamlastxmas• 2 years ago

It’s not insulting to speculate in a conversational way around the errors we very very commonly see

szundi3• 2 years ago

That was not a nice reply for a non-insult. Do you have anything to add maybe?

inoop5• 2 years ago

I'm guessing they know a lot about their costs, and you know very little.

I'm curious what makes you believe the OP doesn't know about cost? They might be director-level at a large tech company with 20+ years experience for all you know...

There's little value in insulting the team members like this.

I'd argue it's not insulting to question a claim (i.e. 'we saved $6MM') that is offered with little explanation.

smokel2• 2 years ago

At one end of the spectrum, some people here claim to write this kind of software over a weekend. Some others claim they require a salary of $600,000, and still need nine additional colleagues to pull something like this off.

There is a lot of room in between, where cost estimates are more realistic.

szundi3• 2 years ago

This answer pretty much sums a lot of my experience. Of course when the guy somehow pulls this off in 2 weeks it is seen as an easy side project with proof that it is, haha

renegade-otter• 2 years ago

Plenty of things can be prototyped over a weekend, but many will require months and even years to get production-ready, feature-complete, and useful, especially at scale.

cdchn4• 2 years ago

Developing and maintaining a totally bespoke DB system with that kind of volume even for $5m/yr, spitball you could get yourself 25 top-notch engineers without AI PhDs and have another mil left over for metal. Sounds plenty feasible to have a nice tailored suit for a core part of your business.

inoop5• 2 years ago

you could get yourself 25 top-notch engineers without AI PhD

Not in the US though. According to levels.fyi, an SDE2 makes ~275k/year at Uber. Hire 25 of those and you're already at $6.875MM. In reality you're going to have a mix of SDE1, SDE2, SDE3, and staff so total salaries will be higher.

Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number.

And that's just the cost of labor, you haven't spun up a single machine and or sent a single byte across a wire.

aeyes• 2 years ago

It doesn't sound like they needed to implement a new DB system for this.

This is using existing features of Docstore which is Uber's own DynamoDB (sharded MySQL) which they seem to be using for almost everything.

davedx• 2 years ago

Is accounting really a core part of Uber's business? They're a transportation company not a bank. I kind of question the premise really

mlrtime• 2 years ago

You're assuming that the team only works on this product. It is possible they are owners of a lot more than just 1 db.

inoop5• 2 years ago

I'd be curious as well to see a more complete cost-benefit analysis, and I'd be especially interested in labor cost.

We don't know how much time and head count Uber committed to this project, but I would be impressed if they were able to pull this off with fewer than 6-8 people. We can use that to get a very rough lower-bound cost estimate.

For example, AWS internally uses a rule of thumb where each developer should generate about $1MM ARR (annual recurring revenue). So, if you have 20 head count, your service should bring in about $20MM annually. If Uber pulled this off with a team of ~6 engineers, by AWS logic, they should about break even.

Another rule of thumb I sometimes see applied is 2x developer salary. So for example, let's assume a 7-person team of 2xSDE1, 3xSDE2, 1xSDE3, and 1xSTAFF, then according to levels.fyi that would be a total annual salary of $2.3MM. Double that, and you get $4.6MM/year to justify that team annual cost footprint, which is still less than $6MM.

Of course, this is assuming a small increase in headcount to operate this new, custom data store, and does not factor in a potentially significant development and migration cost.

So unless my math is completely off, it sounds to me like the cost of development, migration, and ownership is not that far off from the cost of the status quo (i.e. DynamoDb).

Rastonbury2• 2 years ago

Not an engineer, but something like this takes 6-8 people working on only this for a full year?

shrubble4• 2 years ago

If the savings are 6 million per year, then in later years it should pay off since the development is a one time cost.

qaq2• 2 years ago

If you read the article the system was a layer on top of DynamoDB they updated it to use internal product Docstore which required adding a feature to Docstore. So it's not as involved as people make it out to be. Also records are immutable which makes a lot of things way easier.

lfmunoz4• 2 years ago

Off the self software doesn't make sense for a company that is planning on lasting a long time. These solutions are all designed for multiple use cases. That means that there is complexity and inefficiencies that are not required for your particular problem. If you were to just focus on your problem wouldn't you just end up at an ASIC as the most optimal solution? Reason most software doesn't is 1) people like to re-invent the wheel 2) As you go start going lower level the less qualified people you can find.

bjornsing21• 2 years ago

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears.

1. https://www.haystackdb.dev/

sanswork• 2 years ago

You need to be doing enterprise sales not marketing. There is a lot of advice here and in general on that but you definitely need to be making calls with that type of business.

padjo• 2 years ago

Yep nobody with the problem you’re offering to solve is going to solve it by googling and picking some random company they’ve never heard of with no track record.

PaywallBuster• 2 years ago

Maybe find more articles like the above

try connect to the respective people at said teams via LinkedIn and ask feedback

bjornsing21• 2 years ago

Perhaps. But I have a feeling it’s too late once they’ve started building something in-house. Any ideas on how I could find the ones that will publish an article like this one year from now? That’s the ones I’m after, I think.

smokel2• 2 years ago

With a disclaimer that I have no formal nor practical background in marketing, here are some ideas:

1. It is a bit unclear to me when I would use Haystack. The main advantage seems to be cost cutting. It would be nice to see some realized examples of this.

2. When competing for price, you may look like the cheap, and thereby untrusted alternative. There is a risky business paradox here, for which I am sure a fellow HN poster will supply the name: you charge less, therefore you make less, and you will not be able to sustain the service, making me not want to spend money.

3. Have you tried looking for companies that may actually need this solution? Have you tried contacting them directly?

bjornsing21• 2 years ago

1. Good point, thanks.

2. True. One reason I haven’t priced it ridiculously cheap is to avoid this judgement, and fate. With this pricing I won’t necessarily have a smaller profit margin than competitors. The cost advantage comes from a smarter architecture. Any ideas on how I can communicate that would be greatly appreciated.

3. I used to work for one that needed it. I’ve also interviewed at one that had the same problem. A bit hesitant to reach out to potential customers though before I have a solid product I can deliver. But perhaps I shouldn’t be?

superasn2• 2 years ago

One observation I have regarding your homepage is that the message isn't very clear. The headline doesn't mention any benefits I get from using your software.

I think you should invest some time into improving your landing page and maybe you may see some traction. A good resource for this which I've bookmarked is here(1). Hope that helps.

(1) https://www.indiehackers.com/post/my-step-by-step-guide-to-l...

bjornsing21• 2 years ago

Thanks. So you think it would work better if it would just say “save money”, rather than jump straight into the “what”?

To me, when I read the below, that just screams “save money”. But maybe I should do that conversation for the reader so to speak?

From benefits box: “Sometimes you need to index a huge amount of data, to accelerate just a few search queries. But building indexes and keeping them in hot storage can be expensive. HaystackDB builds only the indexes needed for sub-second query latency, across billions of keys, while keeping all your data in low-cost object storage like S3.”

csomar2• 2 years ago

Looking at pricing, it's crazy expensive (and that comparing to AWS, which is crazy expensive). How do you justify that?

bjornsing21• 2 years ago

The idea is that it should be about a tenth of the cost compared to S3 or DynamoDB. Is that not how you read the pricing? Or do you just think that’s still too expensive?

EDIT: Or maybe it’s because reads are expensive? That’s a consequence of the write optimization. The idea is that potential customers will be doing 90%+ writes.

emerongi• 2 years ago

Besides all the other good feedback here, I will offer my extremely petty reason why I wouldn't spend much time evaluating this product. In the FAQ, under the "Are transactions fully ACID?" heading, there is a typo: "simultaineously". It gives me the impression that not enough care has gone into an important part of this product. I know it's not a fair jugdgement, but first impressions matter.

bjornsing21• 2 years ago

That’s easy to fix, thanks.

jgalt212• 2 years ago

Some B2B and B2B2C products just don't work with walk-in leads. You need someone to create and chase down a set of leads.

bjornsing21• 2 years ago

Good point. Any ideas how I could experiment with this “on the cheap”? Any tools I could use to identify and contact leads?

macspoofing4• 2 years ago

The market is saturated with storage products, so it's tough to differentiate yourself. Your site does not help by the way. You're also not selling an end-user product to the public, rather you're selling a technical and infrastructure solution to very technical people - that's a different type of sale. To get those people interested, you must put together technical whitepapers/blog posts/webinars/youtube videos/etc. to explain your approach.

shrubble4• 2 years ago

Find just 1 customer.

Sevii• 2 years ago

Most places I've worked we couldn't even consider using a product that wasn't supported by a major cloud provider like this. What problem does your product solve that customers absolutely 100% need it?

Structurally, you are a small entity trying to compete on cost with hyper scalar cloud providers and open source software. Most ISVs like you charge a ton of money for big problems very few enterprise customers have.

I think you need to find a specific use case where your product is a clear winner. Like 'HaystackDB is the best option for healthcare exchanges to use when receiving claims'.

bjornsing21• 2 years ago

This is a good summary of why I’m hesitant to put more work into it.

The counter argument I guess is that developing your own data store in-house should be even more of a no-no, and companies do that. (One example is obviously Uber, but my previous employer is another example.)

Do you think the option to self-host the product would help tip the scale?

What problem does your product solve that customers absolutely 100% need it?

To be blunt there is no such problem: you can always throw more money e.g. at DynamoDB. But if you have a very write intensive workload (such as the use-case described in the OP), then you can save 90% of that money.

bastawhiz2• 2 years ago

$20 per million reads

Quite frankly, this is not gonna work. I manage a system with a very write-heavy workload (lots of small writes) and even though our writes far outpace our reads, this pricing makes your system about ten times more expensive than an RDS cluster.

There's no data about performance. There's no information on how or whether data is persisted to durable storage before a write is acknowledged. There's not even any information on how big keys or values can be. There's no public information on support.

When choosing a system like yours, my priorities are:

1. Data safety

2. Performance

3. Cost

... In that order. You've done nothing to educate me on 1 and 2 and your pricing isn't better than what you're seeking to displace.

When your product is a tool for developers, show up with hard facts about your product. Zero people (as you've seen) are even remotely interested in building a product on top of a system without knowing whether the system will hold up to their use case. And other than a very anemic FAQ section, you have no documentation at all, whatsoever.

bjornsing21• 2 years ago

All valid points. I guess I’m hesitant to put time into documentation and similar, if I can’t somehow find a steady stream of sales prospects.

even though our writes far outpace our reads, this pricing makes your system about ten times more expensive than an RDS cluster

That indeed sounds off… Are you sure you’re comparing the total cost to that of an RDS cluster? I am aware that reads will be more expensive (due to the write optimization), but I was hoping most customers would make it back on cheap writes. Also the storage itself ($0.23 per GB-month) should be much cheaper than RDS.

victor106• 2 years ago

I would seriously consider an open source business model with an appropriate licensing model. I see lot of companies are open to experimenting with open source db's.

bjornsing21• 2 years ago

Good point. I’m thinking about releasing an open source (or source available) “frontend” for it, and just charge for the “cold storage backend”. How would you feel about that?

oldprogrammer2• 2 years ago

The homepage could benefit from more tangible examples, because right now I can't discern where it fits into my current stack. For most companies, it would be replacing something in a specific context.

Like a side-by-side example. Doing "work" on BigTable (show code examples) versus doing the same "work" on Haystack. Then show the specific metrics on how Haystack is cheaper/faster/better.

cess113• 2 years ago

To consider your product an alternative I'd like to see benchmarks that seem trustworthy, something like a Jepsen analysis or case studies at existing customers, and be able to test it within the EU, i.e. not on US:ian services.

Seems you're in the vicinity of Lund, should be a 'science park' or similar close to the uni where you can find companies that have problems you could solve. Talk to 'incubators', 'accelerators' and the like there.

rguillebert• 2 years ago

So they saved $0.000006 per record, it's really about the little things...

theanirudh• 2 years ago

I wonder if they considered https://tigerbeetle.com

geodel3• 2 years ago

Would be interesting. Considering TigerBeetle is written in Zig. And Uber is probably only rare big company which has support contract with Zig foundation.

sa46• 2 years ago

One interesting caveat is that Uber only uses the Zig toolchain, not the Zig language.

https://www.uber.com/en-US/blog/bootstrapping-ubers-infrastr...

rmccue• 2 years ago

Original story looks to be https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...

deadbabe2• 2 years ago

So did the engineers who proposed this get some kind of bonus considering how much money they saved the company?

Galanwe• 2 years ago

Employees are constantly saving cost or adding value, that's what they are paid for.

zinglersen4• 2 years ago

If a project fails, do you pay for the loss since you want a share of the profits as well?

deadbabe2• 2 years ago

Someone probably gets fired so I guess someone does pay the ultimate price.

chasd00• 2 years ago

Getting to say you led the effort that saved $6M and resulted in some blog posts is probably the reward. At my firm, associating your name to dollars is the fastest way up the corporate ladder.

HeatrayEnjoyer2• 2 years ago

Exactly. Work should be owned by the workers.

drpotato• 2 years ago

The original[1][2] articles are a better read IMO. The link is just a summary of the two with added spelling and grammatical errors that materially impact the meaning.

1. https://www.uber.com/blog/how-ledgerstore-supports-trillions...

2. https://www.uber.com/blog/migrating-from-dynamodb-to-ledgers...

intunderflow• 2 years ago

Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?

gronky_• 2 years ago

Yes, that’s definitely the main reason. It’s called “burying the lede”.

Saving $6M is key information that makes this story interesting. It’s buried all the way at the bottom of the first blog and is completely missing from the second blog which focuses specifically on the migration

masklinn• 2 years ago

I don't get why the originals don't get upvoted

Because they were never submitted? I looked for the first one, it doesn't seem to be on HN.

brushfoot• 2 years ago

Personally, yes, the rehash's title is stronger. It tells a story whose ending piques your curiosity to read more.

"Uber Migrates" (beginning: company that I'm interested in does something) "1T records" (middle: that's a lot of records; I wonder what happened) "from DynamoDB to LedgerStore" (hmm, how do they compare?) "to Save $6M Annually" (end: that's a good chunk of change for me, but was it worth it to Uber? Why did it save that amount? Let me read more)

It's a simple and engaging "there and back again" story that paves the way for a sequel.

Versus:

"How LedgerStore Supports Trillions of Indexes at Uber" (ah, okay, a technology supports trillions of indexes. Moving on to the next article in my feed)

"Migrating a Trillion Entries of Uber’s Ledger Data from DynamoDB to LedgerStore" (ah, a big migration. I'm not sure who did it or whether anything interesting came of it, or even whether it happened or is just theoretical because of the gerund, and moving one trillion of something is cool but not something I probably need to read about right now, so let's move on)

YMMV. Some probably prefer the more abstract/less narrative titles, but the first one is more of an attention grabber for me.

beanjuiceII2• 2 years ago

i mean it could use a few "blazing fast" sprinkled about

IanCal5• 2 years ago

Other than the comments about titles, the entire blogpost doesn't show for me with ublock. So I'll open it, see a picture of some birds, scroll around for a bit then give up.

ckluis• 2 years ago

Just put all your articles into a customGPT with the examples from the rehashes for each one and then ask the GPT to rewrite your title to the a “rehash” like title for the new posts ;)

dang2• 2 years ago

Ok, we've changed to the second link from https://www.infoq.com/news/2024/05/uber-dynamodb-ledgerstore....

Submitters: "Please submit the original source. If a post reports on something found on another site, submit the latter." - https://news.ycombinator.com/newsguidelines.html

_xnmw2• 2 years ago

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite.

Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

kondro2• 2 years ago

Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB.

I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.

bayindirh2• 2 years ago

If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.

Neil44• 2 years ago

At the moment they're just paying someone else to buy $5000 SSD's and run a database on them at many X markup.

ndriscoll2• 2 years ago

There are 61.44 TB NVMe drives (best price I've seen right now is ~6200. They were ~4800 earlier this year). You can have a 1U server with 32 E1.L slots so you should be able to fit ~1.9PB raw storage into 1U for a little over $200k. Don't know how business financing works, but at 8% interest with a 5 year amortization, that's a bit over $4k/month.

choppaface2• 2 years ago

StorageReview plays with 2PB flash machines all the time https://www.youtube.com/watch?v=UQMKtlIjeuk

1PB in a rack with spinning rust + flash buffer has been easy for years now.

BlackLotus89• 2 years ago

No it won't. sqlite "only" works with up to 281TB [0] [1]

[0] https://www.sqlite.org/releaselog/3_33_0.html

[1] https://www.sqlite.org/limits.html (#12)

sgt3• 2 years ago

You can split up into 10 SQLite DB's on this individual server.

tinyspacewizard• 2 years ago

Also a bit scary to have a system without a scaling mechnism built-in in the path of customer traffic. At some point you may be racing to upgrade it.

sgt3• 2 years ago

How would you replicate that SQLite DB onto other hosts to achieve redundancy?

thangngoc894• 2 years ago

One could use Litestream [1]

[1] https://litestream.io

riku_iki2• 2 years ago

it will take forever to create that index. Link describes 10B rows dataset.

sanderjd2• 2 years ago

Sometimes things just aren't nails, even when you have a really good hammer.

khaki54• 2 years ago

The value proposition of commercial loud isn't cost savings unless you manage to quantify all of the ancillary and extrinsic factors such as security risk, HVAC, datacenter personnel, and hardware lifecycle. Any well capitalized and organized company could build their own cloud much more cheaply, but really a significant portion of the calculation is outsourcing the risk components.

_xnmw2• 2 years ago

The problem with outsourcing the risk components is that you don't know for sure whether they are properly taken care of. Major cloud providers have been caught "oopsing" your data, and bam it is gone. Furthermore, they have no incentive to be more efficient about it, they could easily be using 10x the amount of resources necessary, and you wouldn't even have a clue, you're just paying for evermore expensive crap that becomes less reliable over time.

ownagefool• 2 years ago

Any well capitalized and organized company could build their own cloud much more cheaply

Lots of orgs fail to turn money into talent and then talent into products.

It just takes one bad hire at senior level and suddenly your cloud is a vmware install where all machines are boot off network disk, and contention makes the entire thing fall over.

pclmulqdq• 2 years ago

You wouldn't want to do 1T records on one server even if you could. At that scale, you would prefer to be somewhat distributed for availability and scalability. Also, SQLite has issues at large scale.

A reasonable number for one server is about 32-128 TB, and 1.7 petabytes with some redundancy fits nicely in ~30 servers with a decent distributed database.

klysm• 2 years ago

Sure but then you get a whole new set of costs and folks you have to hire to maintain that hardware.

siva7• 2 years ago

Maybe it could and now you got 99 new Problems. That's why more experienced decision makers won't allow this to happen.

Closi4• 2 years ago

1.7 petabytes on Sqlite?

Sqlite's own advice:

If your data will grow to a size that you are uncomfortable or unable to fit into a single disk file, then you should select a solution other than SQLite. SQLite supports databases up to 281 terabytes in size, assuming you can find a disk drive and filesystem that will support 281-terabyte files.
Even so, when the size of the content looks like it might creep into the terabyte range, it would be good to consider a centralized client/server database [over SQLite].

cheeze• 2 years ago

This is the worry IMO. It's fine to dump it on a server with SQLite, but once you start hitting scaling limits, you're in for a potentially rough migration.

callalex• 2 years ago

One of the main reasons you put up with the annoyances of tuple-based storage like DynamoDB is because you want extremely high availability that simply cannot be provided by one computer in one physical location.

benterix• 2 years ago

I read the article so I roughly know what LedgerStore is - but I have no idea where it is hosted.

tiew9Vii• 2 years ago

From one of the original sources linked in this thread

LSG promised shorter indexing lag (i.e., time between when a record is written and its secondary index is created). Additionally, it would give us faster network latency because it was running on-premises within Uber’s data centers.

https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...

xiwenc• 2 years ago

Is this another outlier when you reach certain scale, it’s more beneficial to roll your own? Pretty amazing what Uber has to deal with.

Also it’s not very clear from the original articles, what is the new total “cost of ownership” of this new refactored service. Like now they need to manage their own databases and the storage backing them. Or did i miss it?

crabbone2• 2 years ago

I worked for a company which used Redis at the prototyping phase, but then wrote own database to improve performance and resilience. The company wasn't selling an end-user facing product, the product was a distributed filesystem.

My take on this is that most companies don't have the expertise to build systems like databases, and even if the costs would otherwise suggest such a development as desirable would be simply afraid of doing it.

ForHackernews• 2 years ago

Does no one ever delete data? It's hard to believe there's much business value in keeping every individual payment record dating back to 2017.

sanderjd2• 2 years ago

I'm not sure if they have regulatory obligations to keep them, or what, but it still seems like you could back them up to cold storage after a reasonable period of time.

robertlagrant• 2 years ago

It might just be an internal policy to cover all the crazy combinations of regs the world over. They might just say 10/20/100 years is their policy, now figure out how to store it.

moooo99• 2 years ago

Payment information is often subject to pretty strict regulatory requirements, including archival durations. Having to keep all the original information for 10 years is not entirely uncommon.

crabbone2• 2 years ago

In systems that deal with money, money-related data is virtually never deleted. The reason is the fear that deletion can be exploited somehow in the future, rather than the old data being actionable.

For example, if a customer registers with the name of a deleted customer, which will resurface some "unfinished" transactions or rules associated with the older version of the "same" customer that haven't been properly deleted but appeared to be deleted for a while.

Also, in general, deletion is very difficult because money doesn't just disappear. You'd need some sort of compaction (think: Git squash) rather than deletion to be able to balance the system's books... but then you'd be filling the system with fake transactions...

From my experience from working with these kinds of systems, the typical solution is to label entities with active/inactive labels to substitute deletion. But entities never go away.

kobalsky• 2 years ago

I agree with you, but there is a plus for deleting old data.

If you are not required to keep the information for more than X years, and you still keep it, then you have to provide it when it's requested.

If you didn't keep it, then it can't be used against you.

If you delete it after it was requested, then you are in trouble.

nolongerthere2• 2 years ago

At an individual level I appreciate when an app or service I use maintains all records from the start of our relationship, I’ve infrequently found myself going back and looking for something, and it’s always a breath of fresh air to see that nothing was deleted.

dang2• 2 years ago

Sorry for the offtopicness, but please see https://news.ycombinator.com/item?id=40418627 regarding a flamewar that happened over a week ago. It's important that this not happen again.

washywashy• 2 years ago

I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some critical fix.

gtirloni3• 2 years ago

If anything, it reduces Uber's exposure to AWS' proprietary technology. I don't know how to measure how much that's worth but they probably do.

vertis3• 2 years ago

Companies this size almost certainly have different terms of use. I worked for a smaller, but still ASX200 company that had a custom contract, and assigned staff that would drop by 2-3 days a month. Of specific note was that if AWS wanted to stop doing business with us they had to give at least X notice (from memory that was 12 months for us).

For our risk profile this was more than enough time to migrate off any AWS' proprietary technology.

That makes it worth less to avoid exposure.

scarface_743• 2 years ago

This usually comes from people who have never done a mass migration at scale.

You’re always dependent on your infrastructure. Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate.

No, just use Terraform and Kubernetes is not the answer.

The typical enterprise is dependent on depending on the source between 80 - 120 SaaS products - ie outside vendors.

scop• 2 years ago

That was where my mind went to when I saw the headline. Granted that while I’m not on the Finance side of things and am in fact a developer, “six million” didn’t seem like much at all considering engineer salaries. It’s certainly an achievement, but at what short and long term salaried cost?

Rastonbury2• 2 years ago

$6m across 2.5 years is like $15.5m, how many engineers man-months to breakeven, I'm pretty sure it was worth the work.

rapht• 2 years ago

A primer on valuation: in many financial contexts, $1 of operating savings may be worth much more than $1 of investment.

That is because an investment is a one-off, so it's actually worth $1, but the savings are recurring, so they are worth the same number of years that a company's profits are valued. Depending on sector and investors' beliefs in the future of companies, this factor is typically in the 5-20x range. That means that $1 of savings is well worth at least $5 of investments.

Factor in anything you want!

vertis3• 2 years ago

A better strategy as a company this size would be to write the PRD for moving and then call AWS and negotiate.

rmbyrro• 2 years ago

I think it's likely that they tried this. But DynamoDB is expensive to consume probably because it's expensive to run and maintain. If you develop for a particular use case, a lot of optimizations can reduce these costs. For a large enough business, the fixed costs of in-house are easily amortized. It'd be hard for AWS to compete.

dboreham3• 2 years ago

$6M/y is something like 20 heads (depending on where they are, could be more). So probably it's a win. Hard to see that this could take more than about 5. Add cost of hay and water of course.

tinyhouse2• 2 years ago

It's less that 20 heads. The gross spend for each engineer is probably closer to $0.5 million annually. You can layoff 5% without any impact on the company and save so much more. A company like Uber ($130B market cap) isn't going to bother with building something internally to save $6M/year. The only reason to do it is improve efficiency that actually improves the user experience, which then we're talking about a big deal. Sometimes those things happen only because engineers don't have anything else to do and someone needs a promotion...

hackernewds• 2 years ago

is it? if you consider the value 20 engineers could drive instead in that time

if you assume they wouldn't have had anything else meaningful to work on during that time to save money, then you have a different problem in the company. $6M seems like the value 1 engineer can drive in a company at the scale of Uber

tinyhouse2• 2 years ago

Spot on. $6M/annually is not much of a saving for a company like Uber ($130B market cap). It only makes sense if it's also more efficient and actually improves the app.

tayo42• 2 years ago

Yeah in one quarter from a quick google it looks like Uber profits $1.6 Billion. Basically why I never thought about cost savings projects after I learned to put things into perspective.

People really struggle with large numbers in business I've noticed.

smrtinsert• 2 years ago

A quick look at their careers shows they hiring eng seemingly anywhere but the US so maybe they've already saved the dollars there.

dangus2• 2 years ago

Because they often are a sunk cost.

E.g., you can’t lay off an entire SRE team and have nobody on the on-call rotation. If some of their project work is cost control that is basically free cost savings.

jrockway• 2 years ago

At Google we had a conversion chart between things like memory / cpu and SWE hours, so if you had an idea that would save 8TB of RAM and take 4 hours to do it, you could decide if it would be worth your time or not.

debarshri• 2 years ago

There was an era around 2015, when all the cool tech companies like netflix, spotify, soundcloud, uber and others were building alot of infrastructure and database tools. Nowadays, engineers often talk in AWS/Cloud terminologies.

It is breathe of fresh air to see that orgs are still building tools like that.

augunrik• 2 years ago

Is there some information on why they need to store this much data for immediate retrieval? And why is it so much?

Antony90807• 2 years ago

Wow crazy amount of work went into this. Well done

influx2• 2 years ago

I would gladly pay 6 million/year to not be on call, and have to worry about things like bios and ssd firmware ever again.

geodel3• 2 years ago

Thats a great situation to be in when one can spend 6 million even when there was some chance to save.

I tried same for ready to eat meal everyday to save me from potential kitchen disasters but sadly numbers didn't work out.

influx2• 2 years ago

You're not saving money, controlling your own destiny sure. That's worth something, maybe even more than 6 million, but I was a SRE at Uber who had to be oncall for systems like this, believe it or not, people like me aren't free either :)

tedd4u• 2 years ago

My "favorite" non-cloud issue was dying/dead RAID card batteries in DB hosts (to preserve unflushed cache in RAM on the card in case of power failure).

ec109685• 2 years ago

They didn’t say they are running this on prem.

tuananh• 2 years ago

Additionally, it would give us faster network latency because it was running on-premises within Uber’s data centers.

overstay8930• 2 years ago

Yup, every time we bring up migrations of this sort it pretty much stops when we start talking about maintenance and on-call schedules.

Turns out it costs more to DIY when people start quitting because they have to work during a holiday.

boringg• 2 years ago

How much did the migration effort cost?

geodel3• 2 years ago

More power to them. At this point even technically decent teams/companies have given up on developing large, complex systems in favor of SaaS. After carefully evaluating our strategic course of action answer always is AWS.

Its only team who propose alternative they have to justify rigorously how come they differ in conclusion.

dboreham3• 2 years ago

My Amazon stock thanks you.

tonyhart7• 2 years ago

"At this point even technically decent teams/companies have given up on developing large, complex systems in favor of SaaS"

Yeah until those bill come, They would consider alternative

cess113• 2 years ago

Not if you're in the EU, due to, among other things, Schrems II.

graemep• 2 years ago

AFAIK Schrems II prevents transfers of data to the US.

AWS has datacentres around the world, including multiple locations in the EU.

ramesh31• 2 years ago

Another victim of the "Great Normalization", i.e. that entire generation of garbage tech debt generated during the 2010s that was built on NoSQL stores that never should have been, is now coming due. You could probably make an entire consulting business out of migrating these things to MySQL.

cgh• 2 years ago

This is exactly the comment I came here to make. The NoSql technical debt accretes like dead leaves blocking a sewer drain. Eventually someone has to wade in and normalize…the sewer grate, I guess. Okay, it’s not a perfect analogy.

riku_iki2• 2 years ago

. You could probably make an entire consulting business out of migrating these things to MySQL.

I think it will be very complex task to run MySQL for 1PB 1T transactions..

collyw• 2 years ago

This seemed so obvious at the time, you were throwing away so many useful features for the promise of "web scale". I argued with many developers at the time, but they insisted that NoSQL (Mongo) needed to be used.

jcims• 2 years ago

Every time I've ever used DynamoDB it cost way more than I would have ever expected.

alexey-salmin2• 2 years ago

Yes

ledgerdev• 2 years ago

Say you wanted to build an app on a database like LedgerStore but at much smaller scale, what are the best open source options out there right now?

superzamp• 2 years ago

We have a pretty minimal setup at formancehq/ledger[1] that uses pg as a storage backend, and comes with a programmability layer to model complex transactions (e.g. sourcing $100 from three accounts in cascade).

[1] https://github.com/formancehq/ledger

qwertyuiop_• 2 years ago

Assuming there are a minimum of two teams a total of 20 maintaining this in-house software, I gave 250k as cost per engineer (salary plus health and other benefit costs to the company). Thats $5 million right there. I am estimating lowest range. Thats why Amazon calls these efforts undifferentiated heavy lifting. is there a slight premium to pay than rolling your own and maintaining yes. Its worth all the trouble and security and management overhead into rolling your own.

xyst• 2 years ago

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time.

Usually to aim for a significant promotion.

“Designed and built homegrown system to save $Xm! Give me promo, bro?”

Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

cj3• 2 years ago

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1]

If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings.

It very well might be the case that "We spent $25m of engineering resources to save $6m annually".

[0] https://www.levels.fyi/companies/uber/salaries/software-engi...

[1] https://www.levels.fyi/companies/uber/salaries/software-engi...

booi• 2 years ago

That’s what I was thinking and fully loaded cost at least 35% more than their salary.

Imagine trading 5 headcount full-time to manage the 1T+ fully custom database on an ongoing basis when they could have just used DynamoDB and have been done with it.

Or better, having to engineer a new feature that already existed in DynamoDB and just losing money at that point.

dangus2• 2 years ago

Of course now we are assuming that the existing solution didn’t also require engineering salaries to maintain.

leoqa2• 2 years ago

Bitter story time:

I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings).

My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the manager reach out to apologize.

shawabawa33• 2 years ago

we spent $25m of engineering resources to save $6m annually

This is a huge ROI. Borrowing $25m costs about $1.25m/yr so you're winning even with no upfront costs

choppaface2• 2 years ago

Yes but since revenue is growing at over 70% due to squeezing out the drivers, there's more money to spend on fighting Amazon over the DynamoDB contract https://www.forbes.com/sites/lensherman/2023/01/16/ubers-new...

amanda99• 2 years ago

Right, but that's $25m in R&D investment. Much better than $6m in cost of good/services delivered! Former is great innovation and will be ignored by investors because it's just a fixed cost on the way to becoming profitable. The latter is going to appear in the marginal cost of services calculation.

chii• 2 years ago

$25m spend to save $6m annually is a return on investment of 4 (and a bit) years to break even, then it's positive afterwards. So as long as the project runs for more than 4 years, they'd be ahead.

The only problem is that the database code might need maintenance, specs change or situations change, and the cost is _not_ 25m, but way more per year than anticipated. So much so that after a few years, a new batch of engineers start eyeing writing a new database system to purportedly save money...

password4321• 2 years ago

https://news.ycombinator.com/item?id=38300425#38322311

Uber is famous for NIH syndrome

amluto2• 2 years ago

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics.

If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly expensive. 1.7PB of Parquet files in on-prem or collocated object storage, even replicated a zillion times, is quite cheap. And quite a few companies and open-source projects are currently competing aggressively to provide awesome tools for querying that data.

The hot data would fit on basically anything — the choice should be about robustness and barely even consider cost per TiB. Datomic got written up recently and seems credible for this type of application. FoundationDB is bulletproof. Postgres could probably handle it without breaking a sweat, although active/active replication isn’t free. Heck, writes straight to a warehouse with a cache in front to help with reads seems credible — Uber rides rarely go for longer than a couple hours, and back-of-the-envelope math suggests that the total data rate is maybe 50GB/hour. An entire day of data for an entire country would fit on a single very ordinary commodity server, and the live data for the entire world would fit on one mildly beefy server. The indexes involved sound straightforward.

crmd• 2 years ago

The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes.

Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to deal with in order to query on transactions.

smileysteve• 2 years ago

This. DynamoDB will export to parquet (or csv); s3 select becomes possible for data. More importantly, Athena can query it all

marcyb5st• 2 years ago

In some countries (e.g. Italy, France) you are required by law to keep records of transactions and other data for 10y. But I agree with the data tiering part. Anything older than X months should go to storage (parquet files in some cloud storage).

MarkMarine2• 2 years ago

I heard from some X-Uber people that you could call Uber a database company as much as you could call it a transportation company. Something like 80+ databases invented there in one form or another.

Promotion-driven development. I suppose better than blog post driven development, but marginally so.

vineyardmike2• 2 years ago

I suppose better than blog post driven development, but marginally so.

I find DB development quite interesting, and I find Uber's core product quite not interesting (from an engineering perspective).

So as an outsider without any financial stake in the company, please keep writing about databases!

surfingdino2• 2 years ago

I heard from some X-Uber people that you could call Uber a database company

Someone once called Airbus planes "Sun servers with wings". All businesses today have an IT department and those build around the idea of real-time scheduling done by computers are compute/storage businesses first. That's why Uber was able to launch Uber Eats or e-bikes. I am not surprised they have a bunch of busy devs building cool stuff. Not all of it is ego-driven. I liked their logger for Go last time I used it.

danpalmer• 2 years ago

I'm not sure if this is true of modern Google. Certainly there are a ton of custom things around, but it also seems like in the last 5 years there has been a huge push towards not re-inventing more databases/etc, and normalising everything onto a few core technologies. A good example is Spanner, many products have been moved off other home-grown systems onto Spanner, basically to reduce complexity and cost. Sure Spanner is a Google technology, but at least it's one, instead of having all sorts of different esoteric systems, and Spanner is just one example, this is happening in a number of different domains.

foota• 2 years ago

The article states that they already had an in house solution for cold data, so one of the benefits they claim is simplifying by moving to one system for both hot and cold data.

citizenpaul2• 2 years ago

I think there is some reckoning of cloud service providers coming(assuming logical actors...). I was doing some contract work for a small place that had a GCP Bigtable that was costing $11k+ per month for some reports that were based on data from a 375MB !!! mysql db into big-table for the reports to run.

They hired some out of school data scientist to do reports and they were doing crazy ineffective things with the tiny dataset. Wanted me to fix it for pennies tomorrow and I declined.

remus• 2 years ago

Not that I disagree with your overall point, but I don't think this

I was doing some contract work for a small place that had a GCP Bigtable that was costing $11k+ per month for some reports that were based on data from a 375MB !!! mysql db into big-table for the reports to run.

Is a good example. It's just a badly architected system, and you'd have exactly the same problem if you were running the same thing on a massively over provisioned on premise db.

citizenpaul2• 2 years ago

I think its a perfect example that cloud providers are profiting on a unsustainable scheme. They rely on their services papering over (literally with money) lack of skill/knowlege of their customers. This tactic either kills the customer or creates someone that is desperate to no longer be your customer.

PeterZaitsev2• 2 years ago

I think this is fantastic illustration of how expensive proprietary cloud based data stores can be... and what it is feasible to migrate from them to something else.

benced• 2 years ago

$6M... isn't that much?

SkyMarshal• 2 years ago

It seems LedgerStore is not open source [1], and finding any info on it requires following a trail of backlinked Uber blog posts. Here's one with the most info on LedgerStore that I can find, from 2021:

https://www.uber.com/en-US/blog/dynamodb-to-docstore-migrati...

[1] https://github.com/uber

PeterZaitsev2• 2 years ago

Yeah. This looks like some internal solution. In general Uber seems to be high on "not invented here" scale - they like to conclude no existing Open Source solutions are good enough for them and they need to build their own... this is different from Facebook approach for example which chose to made MySQL better by adding MyRocks/RocksDB to it and keep them Open Source.

vertis3• 2 years ago

It's a weird world where Facebook/Meta is becoming a small bastion of hope. Llama 2/3 being an example of bucking the trend of going closed source for LLM models.

Granted it's not quite in the same calibre as OpenAI/Claude, and the real test is when it is and they still release it.

rubiquity• 2 years ago

Don't worry it's 99% likely LedgerStore is built on top of MySQL.

alexey-salmin2• 2 years ago

I don't know about economics of this particular project but damn dynamodb is expensive. At some point I was thinking that everyone else was just using it wrong, doing scans and queries instead of point-wise lookups into pre-computed tables.

It turns out however that even when you use it as a distributed hashtable you still pay a huge premium.

kikimora2• 2 years ago

Why? 120 usd per 100 WCU per year, 30 usd per 100 RCU, does not sound expensive. 1 RCU reads up to 4Kb, to read 100 MBs you would need 100 000 RCU which would cost 30 000 usd/year, or 2500/year. Unless my math is off I don’t think anything come close in terms of price.

pojzon• 2 years ago

You look at stuff like that and think about

„how much talent is wasted on pointless things that help noone in the world while getting paid heaps for nothing”

We could accomplish everything if ppl stopped wasting time on pointless tasks.

otterley• 2 years ago

Does anyone know whether Uber considered Amazon QLDB for the implementation? Seems like it might have been a good fit, at first blush.

yazaddaruvala2• 2 years ago

Reading the article it’s clear pretty quickly that Uber was using DynamoDB poorly.

It seems they need strong consistency for certain CUJs and then a lot of data warehousing for historical transactions.

It’s strange to me that they didn’t first convert their 2 table DynamoDB architecture into DynamoDB and Redshift architecture or similar. This is a pretty common pattern.

PartiallyTyped• 2 years ago

I don't understand why they needed 2 weeks of immutable transactions in Dynamo. Could anyone give any hints?

cced• 2 years ago

Can you post some references to this pattern?

yazaddaruvala2• 2 years ago

Mostly just look into “archiving into data warehouse” type patterns.

You can also look into “realtime data warehousing”.

Meanwhile, it’s so common, AWS built it into their product.

https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-dy...

awinter-py• 2 years ago

they do 2 billion rides per quarter

what are the 'trillions' here?

^ also that translates to ~1000 transactions per second with some assumptions; have never understood why they care so much about infra scaling

1000 tps is like 1 box

Migrating Uber's ledger data from DynamoDB to LedgerStore

Related Stories