Google Doorman: Global Distributed Client Side Rate Limiting

https://github.com/youtube/doorman

by ryancox • 10 years ago

73 25 10 years ago

github.com

dedalus• 10 years ago

It should have been called "Velvet Rope" :-)

cm3• 10 years ago

Care to you explain the joke for the culturally unaware?

CaveTech• 10 years ago

When waiting to get into a club or party, you wait behind such a rope until the bouncer allows you to enter (or doesn't).

mixonic• 10 years ago

Hopefully there is a Googler reading who can answer this: I'm curious why the README.md says

Note: This is not an official Google product.

However the copyright is Google, and you must sign their CLA. That seems pretty official to me? Or is there some other implication to "official" beyond ownership?

dekhn• 10 years ago

It works like this. Google has an "open source office". When you open source code at Google, it's either a product, or it isn't. Even if it's not a product (which has its own extra set of launch procedures), it's possible to open source it. It's still a Google-owned thing, but not a "product". Google ownership does not necessarily mean it's a product, but it still requires the copyright stamp and the CLA.

If it's easier, just drop the word "official". It's not a Google product- IE, a saleable or free service that google provides to customers- but rather just a thing Google put out in the world, subject to the rules that Google applies.

yegle• 10 years ago

This is a project whose author is a Googler thus Google has the copyright and all the CLA stuff. However Google is not directly involved in the project, in the sense of having employees solely work on the project and have an OKR on this project, or roadmap whatsoever.

Think about this: someone work for Google wrote a software and Google is kind enough to open source it.

Disclaimer: I work for Google but my interpretation could be wrong.

brunoqc• 10 years ago

Does Google also own code that you write in your spare time at home?

wereHamster2• 10 years ago

That is because Google claims copyright on everything you create while you work there, even if you create it in your garage in your free time. You baked some cookies for your daughters birthday party? Sorry, can't distribute them for free, consumers of those must sign a CLA (Consumer License Agreement) and Google retains copyright on your recipe.

kuschku3• 10 years ago

Question: Does this also apply to Google employees at Google Hamburg, and, if yes, how far? Because, as far as I know, German law directly prohibits such contracts.

schwa5712• 10 years ago

This is false. Source: I am a Googler who has released open source code, written in my own time, using my own hardware/software.

Here's how it works (in California, anyway):

Google reserves the right to review open source projects to determine whether there is overlap with a current or potential Google business. This is done by submitting a form that describes one's project and then waiting a while for a decision. If the decision is "yes", then you are free to work on your project with your own time/resources, and publish it under your own copyright.

Google being Google, almost anything could be construed as overlapping with Google business interests. However, my anecdata suggests that Google is quite liberal in this regard: I had no problems myself, and know a few others who also had no problems.

In this case (i.e. this Hacker News story), I'm guessing that we're talking about something different. I suspect that the project in question was written using Google time and resources, and it is IMO appropriate for Google to own the copyright.

wslh• 10 years ago

Offtopic but related: I want to like GRPC but I think Google is not working hard to make it quickly usable. Just try to perform a pip install in Visual Studio and obviously... it will not work. This is not the first time it happens, try to compile V8 (NodeJS is easy) or Chrome in Windows and it will be difficult even following the step by step instructions.

secure• 10 years ago

I have the opposite experience: The Go version of gRPC works like a charm, following the official instructions.

harryf• 10 years ago

So do I understand this right: Doorman could be used to rate-limit access to a website by integrating it with the sites proxy server, the result being individual clients would only get a certain number of requests per minute based on available server side resources?

tonfa• 10 years ago

I think it's more meant for internal RPC within a cluster, to e.g. avoid overloading a service.

stingraycharles• 10 years ago

This is the correct answer, since this exact problem is a major issue when dealing with distributed system failovers. Here are a few examples of AWS service disruptions due to exactly this problem, where they were unable to recover the system because nodes kept failing over under load:

https://aws.amazon.com/message/5467D2/ https://aws.amazon.com/message/2329B7/ http://aws.amazon.com/message/65648/

An example quote:

"When this network connectivity issue occurred, a large number of EBS nodes in a single EBS cluster lost connection to their replicas. When the incorrect traffic shift was rolled back and network connectivity was restored, these nodes rapidly began searching the EBS cluster for available server space where they could re-mirror data. Once again, in a normally functioning cluster, this occurs in milliseconds. In this case, because the issue affected such a large number of volumes concurrently, the free capacity of the EBS cluster was quickly exhausted, leaving many of the nodes “stuck” in a loop, continuously searching the cluster for free space. This quickly led to a “re-mirroring storm,” where a large number of volumes were effectively “stuck” while the nodes searched the cluster for the storage space it needed for its new replica. At this point, about 13% of the volumes in the affected Availability Zone were in this “stuck” state."

So these things are very hard, can occur in totally unexpected situations, and I'm not at all surprised that a company like Google comes out with something like this.

nickpsecurity• 10 years ago

I know it expects cooperative behavior. However, I wonder if the protocol design could be used in a setup where an embedded firewall did the rate limiting and mediated traffic from the possibly-malicious host. Boeing already has rate-limiting in their embedded firewall but cheap or OSS project could use a OSS rate limiter prebuilt to save time.

As in, is this protocol inherently cooperative or could an implementation have checks/controls added?

rixed2• 10 years ago

Rate limiting looks outdated. Suppose you can spawn new servers (and turn down unused ones) fast enough to react to demand, and have a good LB that sends the traffic where you have capacity, then you do not need to limit anything until you reach full capacity, in which case you probably want to degrade (by query class and/or client class) rather than limit anyway.

Rate limiting is an inefficient way to distribute a service, that makes sense only if you preallocate your resources and have queries with predictable cost. Let's use this technique only as a bug-prevention tool not for resource economy, as organised scarcity is likely as inefficient for the data center as it is for the distribution of goods :)

greenleafjacob• 10 years ago

If you degrade by client or query class then in the case of just one class of query or one class of client you would have an unfair distribution, because one client could consume many resources and others would be limited to whatever's available on the margins.

I would like to see an adaptive system that, when resources are scarce, pushes towards a more equitable distribution.

Google Doorman: Global Distributed Client Side Rate Limiting

Related Stories