What’s new with our in-house load balancing service

https://dropbox.tech/infrastructure/robinhood-in-house-load-balancing-service

by ABS • 2 years ago

49 47 2 years ago

dropbox.tech

leetrout• 2 years ago

I am surprised to see zookeeper being used with etcd.

Why use both?

Why even consider ZK in 2024? Is it just popular in java shops?

Maybe I am misunderstanding?

bittermandel• 2 years ago

Since it's written ZooKeeper/etcd-based, I assume they use ZK and are going over to Etcd.

pgwhalen2• 2 years ago

Why even consider ZK in 2024?

I'm no expert on these consensus-stores, but I guess my question would be "why not?" ZK is proven and works.

zaphirplane• 2 years ago

Great question

aftbit2• 2 years ago

Dropbox has been using Zookeeper since the beginning. I assume they are just familiar with running it and understand its guarantees better.

nepthar• 2 years ago

We used a deterministic aperture load balancing to solve this at twitter - https://blog.x.com/engineering/en_us/topics/infrastructure/2....

Cool to see some other approaches, although I believe they've complicated theirs a bit more than necessary.

renewiltord2• 2 years ago

Very cool. Low-state approaches are always more interesting.

jauntywundrkind• 2 years ago

Ah nice, a "Power of 2 (random) choices" strategy. Article credits "The power of two choices in randomized load balancing" (1996) by Michael Mitzenmacher. PDF link: https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis....

I heard about someone using PO2C for cache eviction recently. Maybe Oxide talking about their storage work? Seems obviously smart. No need to keep LRU or carefully track & decide, just use statistics to compare two random elements and drop the least used one.

renewiltord2• 2 years ago

I see: load-aware routing. Why skip the weight update phase on low weight reports? Isn’t that precisely the period when those nodes are likely overloaded?

dang• 2 years ago

[stub for offtopicness]

terramauthe• 2 years ago

[flagged]

yawnxyz• 2 years ago

"load balancing" is a cold way to describe a layoff...

guessmyname2• 2 years ago

He does → https://www.linkedin.com/in/yi-shu-tai-13160154/

According to his LinkedIn profile: 7 years 6 months into the job.

Although, it looks like he transfered to a new team a year ago to do ML Ops.

satvikpendem• 2 years ago

[flagged]

stavros• 2 years ago

Probably because it takes from the rich and gives to the poor.

kergonath• 2 years ago

You cannot avoid all names that were or will be ever used by anyone. Collisions will happen. And Robin Hood is not particularly obscure…

guessmyname2• 2 years ago

Round Robin [1] is a popular algorithm for load balancing.

That’s the only reason they named the software “Robinhood”.

[1] https://en.wikipedia.org/wiki/Round-robin_scheduling

almost• 2 years ago

Weird that a gambling app would take thename of a popular character from legend!

Or maybe not, try actually naming something sometime and you'll see. It's not easy to come up with something that fits all the requirements AND has never been used before.

And it's not like anyone is ever going to get confused and accidentally try and use Dropbox's internal load balancer to buy options on a stock or something. Or try and make a gambling app the star of their next hit movie for that matter.

whalesalad• 2 years ago

because it is an internal tool? why does that cause an issue?

xyzzy47473• 2 years ago

[flagged]

bayindirh3• 2 years ago

There's a tool called Robinhood which started its life around 2004 [0], In turn, Robinhood Markets are founded in 2013.

Imagine knowing that Robinhood exists as one of the most popular (or the only) policy engines for large (EB sized) filesystems for but still naming your new company Robinhood.

I think Robinhood Markets should C&D immediately and find themselves a new name.

[0] https://github.com/cea-hpc/robinhood

bityard• 2 years ago

Were Facebook, Google, and Reddit also the names of centuries-old fictional storybook characters?

pgwhalen2• 2 years ago

The Robinhood being discussed in this post is not a product, it's internal technology. The name clashing doesn't really matter.

say_it_as_it_is• 2 years ago

[flagged]

qeternity• 2 years ago

Totally. They should just continue to carry a workforce they don't need, risking the viability of the business and the other 80% of the jobs.

ryandvm• 2 years ago

[flagged]

aftbit2• 2 years ago

Bees gather pollen. In the fall, as temperatures get lower, they still want to keep gathering pollen, but sometimes they run out of warmth and energy to fly all the way back to the nest. Thus you will find bees sleeping on flowers through the night. When morning comes and the sun warms them, they'll pick up and fly off to the nest to drop off the last night's load of pollen.

What’s new with our in-house load balancing service

Related Stories