Large-Scale Transactional Data Lake at Uber Using Apache Hudi

https://eng.uber.com/apache-hudi-graduation/

66 31 6 years ago

eng.uber.com

planck01• 6 years ago

Is Hadoop still the way to go for big data solutions? My feeling based on my social bubble is that less and less solutions are built on it. The recent stackoverflow survey seems to indicate it is not very popular anymore.

monkeyfacebag• 6 years ago

I think you may be conflating Hadoop the mapreduce framework with Hadoop the ecosystem, which includes hdfs, Hive, Spark and others. To the best of my knowledge, former is waning in popularity (supplanted by tools like Spark), but the latter remains in wide use.

throwaway_pdp094• 6 years ago

Straight up, what's people's views on all this big data stuff? I see it in so many job ads and I really can't believe it's necessary. Sure, at the far end of one side of the bell curve are companies like uber but otherwise, are people using it to process a few terabytes that could be done better on one multicore server? How many companies have enough data to justify it? Personal opinions welcome.

teraku3• 6 years ago

Spark being an example of a tool that (also) exists inside the Hadoop ecosystem and is still gaining acceptance due to popularity, but at the same time losing traction as better option are arising.

fongitosous• 6 years ago

is spark that widely used? arent people moving away from it for deep learning frameworks like tf?

StreamBright4• 6 years ago

Not at all. There are way better solutions out there.

nknealk• 6 years ago

Is anyone non-Uber using Hudi in production? Can someone comment on what developing against it is like compared to other warehousing technologies?

cmollis• 6 years ago

We're about to evaluate it since it's incubating now in AWS' EMR. I think if you have existing spark workloads that require update or delete of existing data stored in parquet, then it might be a good choice since it fills in a major gap there. The other choice is Delta Lake which provides similar capability.

xyzzy_plugh• 6 years ago

I believe Netflix's Iceberg (now also Apache) aims to solve the same problems.

hashhar2• 6 years ago

Yes, we are using it in production (somewhat) - for a a few tables.

Our workflow with it is:

1. Stream new data into a partitioned S3 bucket available as a table in metastore.

2. Use Hudi to create a copy-on-write compacted version of the table (think SELECT * FROM (SELECT *, row_number() over (partition by uuid order by ts desc) as rnum) WHERE rnum=1).

Now people who are interested on the snapshot of the table rather than the event log can query the Hudi table instead of the raw event log tables.

This has improved perf for the common use-cases while allowing us to use a common pipeline. Earlier we would have to dump such data into an RDS (Aurora Postgres) with UPSERT but that meant table bloat was an issue and cost was high depending on the date-range we wanted to store compacted.

loic-sharma• 6 years ago

The image in the 'The road ahead' section seems to be using Azure's icons. For example:

Azure Event Hubs icon - https://images.app.goo.gl/NLu8jSKWYPwMyTtD9

Azure Marketplace icon - https://images.app.goo.gl/X9vZeWWAR2TDsawe8

Azure Service Health icon - https://images.app.goo.gl/fJcUZqwUWbzK7tC99

They probably should change that...

Large-Scale Transactional Data Lake at Uber Using Apache Hudi

Related Stories