What I Wish I Had Known Before Scaling Uber [video]

https://www.youtube.com/watch?v=kb-m2fasdDY

by kiyanwang • 10 years ago

502 272 10 years ago

tiborsaas2• 10 years ago

"Uber is most reliable over the weekends when engineers don't change it" :)

kenrose• 10 years ago

We see this pattern at PagerDuty over the majority of our customers. There is a definite lull in alert volume over the weekends that picks up first thing Monday morning.

It's led to my personal conclusion that most production issues are caused by people, not errant hardware or systems.

asimuvPR3• 10 years ago

I've come to question releasing often as a result.

selckin• 10 years ago

Do they have the same load/users during the weekend?

DuskStar• 10 years ago

Absolute numbers of alerts are probably a lot less useful than alerts/use, or alerts/users. After all, if people use the services PagerDuty covers less often on the weekend then you might see a lower alert volume even if issues are relatively more common.

ex_amazon_sde2• 10 years ago

most production issues are caused by people

It's a well known fact, both for systems and networks.

Yhippa• 10 years ago

After working at various enterprises over the years (where deployments are slower in some cases) I've noticed you'd do a Thursday/Friday/weekend deployment, everything "looks good" and you'll still have a bunch of issues Monday morning due to users finally using the system en masse.

anezvigin• 10 years ago

Would PagerDuty consider publishing some anonymized aggregate statistics?

employee80002• 10 years ago

90% of outages are caused by configuration changes which is why change management was so hot for enterprise software.

ChuckMcM2• 10 years ago

That conclusion is well founded. We correlated issues at Blekko across a lot of different factors, the one that always held was code or configuration changes. Not too surprising in the large but definitely confirmed by the data.

hinkley4• 10 years ago

I'm bad at optimism, but maybe the takeaway here is that we're finally at the point where it's not the hardware or systems that are the biggest problem. The automation is actually working well enough that it's not the tall tent pole.

Aeolun• 10 years ago

That doesn't surprise me at all. If you're not changing software, you can't create any additional bugs.

di4na• 10 years ago

Question is : people at each level. Here most of the alerts happens at night... due to batchs...

Most of the other alerts are due to network issues, moving servers due to hardware intervention that change the topology of the network, etc etc.

mathattack• 10 years ago

Couldn't this also just as easily be a change in usage patterns?

gjolund• 10 years ago

"Move fast and break things" is the worst thing to happen to software development that I can remember.

enraged_camel• 10 years ago

I think it's OK to move fast and break things when you're in the initial product development stage, where you need to develop features quickly and try lots of new stuff to see what sticks. Once you have paying customers though, it may be time to slow down and start caring about stability, at least for your main branch.

adekok2• 10 years ago

I suspect that the issue isn't the engineers. It's the changes.

A system running just fine will usually continue to run just fine. Until it's changed.

fail2fail2ban6• 10 years ago

Or until the power is cycled. That's why it's best to do the crazy monkey or whatever it's called, walk into the production offline datacenter, online for testing (you do have geographically redundant datacenters right?) and pull out a half dozen random NIC cables. You can simulate with software tools but nothing beats the real test.

Tenhundfeld• 10 years ago

I hear what you're saying, in theory. In practice, a system is always changing – regardless of whether developers are deploying code changes.

Some events (or "changes") I've seen in last year that caused a system to stop running fine:

- analyst ran query that locked reporting DB, which had ripple effects

- integration partner "tweaked" schema of data uploaded to us

- NIC started dropping packets

- growing data made backup software exceed expected window, locking files

- critical security patch released (didn't cause a problem exactly, but did change the system from having no known security holes to having a big one)

On and on. So, yeah, I'm not disputing the idea that changes are often the source of problems. I'm just saying that any moderately complex system is constantly encountering new events whether or not the developers are making changes.

This isn't a rebuttal of your statement so much, as it is rebuttal of a common attitude I see in business folks. A lot of non-technical executives seem to have the mentality that software is "done" after it's built, which is naive IMO.

Active systems require active maintenance. You can avoid a lot of problems with intelligent architecture and robust instrumentation/monitoring, but at the end of the day systems rot and will eventually stop running fine.

And even if your perfectly planned and architected system runs in total isolation on a private server where security isn't a major concern, you'll still build up technical debt if you aren't routinely upgrading major libraries, etc. You'll get to a point where you want to implement some feature, and while there are 3 excellent OSS libraries for your platform to accomplish it, none of them is compatible with your 5 year old version.

You probably know all of this and might even agree with most of it. I just had a visceral reaction to the idea that "a system running just fine will usually continue to run just fine," and felt compelled to respond. :)

Note: most of my experience is with public or enterprise web applications. I imagine other types of systems have other problems.

mshenfield• 10 years ago

Facebook noticed the same thing.

http://www.darkcoding.net/software/facebooks-code-quality-pr... - "Exhibit C: Our site works when the engineers go on holiday"

jackweirdy2• 10 years ago

Hospital mortality rates in each department are also lowest when there's a conference for that specialty nearby. The doctors all go there, few/no routine surgeries are done that day, far fewer people die.

weberc2• 10 years ago

"Move fast and break things" https://xkcd.com/1428/

samstokes2• 10 years ago

"A ship in harbor is safe — but that is not what ships are built for."

crpatino2• 10 years ago

Are you implying that software purpose is to provide developers with puzzles to tinker with... in detriment of their paying customers?

kaspm• 10 years ago

Does anyone who has worked in software for any period of time not realize this? Computers are dumb they (generally) repeat the same tasks reliably _compared to humans_. Occasionally there are system failures but most failure are caused by code or configuration changes, especially when they interact with inputs and systems in untested ways.

amelius3• 10 years ago

I didn't see the video. But given that a cab-service has a natural sharding point (i.e., per city), I don't get why scaling is such an issue.

xapata4• 10 years ago

Some cities are bigger than others. That's a heavily imbalanced sharding key.

amelius3• 10 years ago

Why is that a problem? You could count several smaller cities as a single large city, as far as sharding is concerned.

latch• 10 years ago

Which isn't necessarily a problem. Just put X shard on the same machine. If the largest city can be handled by a single server, I generally agree with your parent.

I've thought about this problem before, both for a related problem space and with friends working in this specific space. The short version is to create a grid where each square holds car data (id, status, type, x, y, ...) in-memory. Any write-lock of concern is only needed when a car changes grid, and then only on the two grids in questions. This can be layered multiple levels, and your final car-holding structure could be an r-tree or something.

The grid for a city can be sharded across multiple servers. And, if you told me that was necessary, fine..but as-is, I'm suspicious that a pretty basic server can't handle a tens of thousands of cars sending updates every second.

Friends tell me the heavy processing is in routing / map stuff, but this is relatively stateless and can be sent off to a pool of workers to handle.

tiborsaas2• 10 years ago

Cab service on a planetary scale :) They have 2000 engineers, 800 microservices and 8000 GIT repositories. Does it still seem trivial to you?

EDIT: to downvoters: why shoot the messenger? :)

cheez• 10 years ago

Sounds like they have 2000 very individualistic engineers with that level of insanity.

esrauch• 10 years ago

I'm curious how a company ends up with 10 git repositories per microservice, and 4 repositories per engineer?

aristus• 10 years ago

Sounds like they should sit down and re-think why they need one service for every two engineers (!), and ten repos per service (!!).

moron4hire• 10 years ago

This leads to an interesting perspective, that there is a hard limit on how much traffic your services will ever have to handle, and that limit is probably a constant expression on the planet's population.

But I can't imagine how you could possibly need 8000 git repositories unless you're massively over engineering your problem. Project structure tends to reflect the organizations that build them.

JoelBennett• 10 years ago

This seems almost the exact opposite of what I've heard of Facebook (i.e.: they have a massive repository).

anentropic• 10 years ago

2000 engineers sounds like overkill for a mobile app

nkassis• 10 years ago

Not really what the video addresses for the most part. Mostly talking about their development process and impact of breaking up their application into 1000s of microservices. This is more to handle their development team scaling, rate of change etc...

mason55• 10 years ago

I think that only parts can be sharded by city. Anything having to do with users has to be global (because I can visit different cities). I suppose you could try to "move" users from shard to shard but you'd still want a globally unique ID for each user (so you could run cross-city analytics). It feels like you'd be building so much orchestration around making the global parts work in sharded manner that you might as well make the whole thing work globally.

amelius3• 10 years ago

you'd still want a globally unique ID for each user (so you could run cross-city analytics).

Analytics can run in batch mode. Also it is read-only operation. So it is far simpler than designing a distributed database.

As an aside, why would Uber want to reinvent a distributed database for storing user information?

jfoster• 10 years ago

To me, it seems the best thing to do in that scenario is taking the huge "shard by city" gain when it comes to rides. Then dealing with the global nature of users by having global user DBs sharded by user ID.

hueving4• 10 years ago

Everything related to driver selection and real time tracking is very specific to one location. The server responsible for tracking drivers in San Jose doesn't need to reference rider locations in NYC under any circumstance.

The things you highlighted are very light load comparatively. Sign in, load account info, then dispatch to location specific shard.

mranney3• 10 years ago

Scaling the traffic is not the issue. Scaling the team and the product feature release rate is the primary driver.

fitzwatermellow• 10 years ago

Link to Slides here:

https://gotocon.com/dl/goto-chicago-2016/slides/MattRanney_W...

Any speculation as to why Uber doesn't just want to use something like Netflix Eureka / Hystrix instead?

mranney3• 10 years ago

The Netflix stack has a lot of assumptions about Java, and most of Uber is written in languages other than Java.

buzzdenver• 10 years ago

Did I miss what WIWIK stands for ?

jjbiotech2• 10 years ago

What I wish I knew

mawburn• 10 years ago

I can't be the only one who keeps reading it as: Whitest Kids U Know.

paukiatwee• 10 years ago

Just to confirm, 1000 microservices in this case is 1000 different apps (e.g.different docker images) running simultaneously? 1000 microservices in this case not 1000 microservice instances (e.g. docker instances)?

If it is 1000 microservices as in different apps, then they must have at least 2000 running apps (at least 2 instances per app for HA).

Maybe uber only have 200 "active" microservice app running at the same time where each microservices have N running instances.

I just cant imagine running 1000 different microservices (e.g different docker images, not docker instances) at the same time.

employee80002• 10 years ago

Yes, and much more than 2 per instance of a service.

mranney3• 10 years ago

Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.

jrochkind13• 10 years ago

that sounds nightmarish

niccaluim• 10 years ago

Good god, what on earth for?

mtrpcic2• 10 years ago

Could you give some information as to what the breakdown of functionality is for those services? I can't fathom 1700 different and unique pieces of functionality that would need to be their own services.

user59944614• 10 years ago

>> Our number is closer to 1,700 now, but yes this means 1,700 distinct applications. Each application has many instances, some have thousands of instances.

Time to forbid adding more stuff and start cleaning.

    (12) [...] perfection has been reached not when there
        is nothing left to add, but when there is nothing
        left to take away.

https://tools.ietf.org/html/rfc1925

BinaryIdiot• 10 years ago

Whoa. That's is an insanely large amount of applications. I'm assuming that's essentially one function per microservice which is one of the huge do-not-do of microservices as it's just a complete waste of time, resources, etc.

I would love to hear a breakdown. This sounds like a nightmare to maintain and test.

cagataygurturk2• 10 years ago

Maybe my ignorance, but do they have services like left-pad-as-a microservice? I can't understand which 1000 microservices you can derive for a cab-renting application.

0xmohit4• 10 years ago

  left-pad-as-a microservice

You can also have another service to monitor the left-pad service.

oldmanjay• 10 years ago

I'm not sure if it's ignorance that leads people to dismiss things they don't understand so much as it is a coping mechanism similar to that which gave birth to various religions. Ignorance just lets it thrive after the birth.

nichochar2• 10 years ago

I used to work in startups, and wondered this all the time. Then I joined a big valley tech company, and now I understand.

It's because they hire smart and ambitious people, but give them a tiny vertical to work on. It's a person level problem, not a technical one in my opinion.

I think you solve this by designing your stack and then hiring meticulously, instead of HIRE ALL THE ENGINEERS and then figure it out (which is quite obviously uber's problem)

andrewvijay2• 10 years ago

Highly recommended video. Lots of stuff he spoke are very relatable. Like having many repos , storing configs as a separate repo, politics by people, having a tracking system

babyrainbow• 10 years ago

Sorry, don't think so. I went through the slides and didn't find anything really interesting.

softawre3• 10 years ago

You went through the slides but didn't watch the video? If so, you can't really make the argument that it isn't interesting.

andrewvijay2• 10 years ago

I'm recommending the video explicitly. Not the slides!

gtrubetskoy• 10 years ago

I think the world of service architecture is roughly divided in two camps: (1) people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and (2) people who (having gone through pains of (1)) realized that strong schemas, things like Thrift, Protobufs, Avro are a good thing, as is SQL and relational databases, because rigid is good. (Camp 1 is more likely to be using high level dynamic languages like Python and Ruby, and camp 2 will be more on the strongly typed side e.g. C/C++, Go, Java).

KayEss• 10 years ago

I still kind of think you can have great REST and JSON and make full use of a great RDBMS.

REST/JSON with strong schemas (and loose where that makes sense), using Postgres and C++.

fail2fail2ban6• 10 years ago

REST/JSON makes sense between a human backed browser and a middle tier; between machines, SOAP is the best contract first technology, or at least a custom XSD backed XML schema.

iamnothere2• 10 years ago

Is your first example really "naive" though? In my experience, loose, flexible schemas and dynamic languages are very well suited to rapid early-stage development, much more so than rigid languages and schemas.

Sure, in the long term things should be refactored, structured, and optimized. But if you do that too soon you risk locking yourself out of potential value, as well as gold-plating things that aren't critical.

mtberatwork• 10 years ago

Sure, in the long term things should be refactored, structured, and optimized.

How often does that really happen though? Once you've amassed enough technical/data debt, resistance to refactoring increases until it never happens at all. Having well defined, coherent data models and schemas from the start will pay off in the long run. Applications begin and end with data, so why half-ass this from the get go?

mixmastamyk• 10 years ago

This is the key point I think. Flexibility is very important when a project is immature. And as it matures it benefits from additional safety. Too bad so few tools support that transition.

conover• 10 years ago

I'll take gross generalizations for $500, Alex.

eric_the_read• 10 years ago

Amusingly, I've just finished trying to convince a team using C in their firmware to use Protobufs; they were trying to convince me to use JSON.

Oh, and my servers are in Ruby.

seangrogg2• 10 years ago

Protobuf v3 allows use of JSON and gRPC for Node.js handles JavaScript objects quite handily. Might not be mutually exclusive.

nodja• 10 years ago

I don't agree with this. I'm just a small developer working for a medium-sized company, but I'm a python guy and I love RDBs and REST/JSON.

Sometimes it's easier for me to alter the JSON payload a certain way in the frontend, then the python backend handles it and saves it to the DB.

The RDB helps not to stray too far into crazy-land, while python and JSON gives you the flexibility to prototype and experiment.

ci5er3• 10 years ago

In the absence of a good reason not to use Python (and there are many case where one should not), I use it quite a bit myself. And usually (absent good reason) I use it with PostresQL. SQLAlchemy is a wonderful thing.

Granted, I am neither a database nor ORM savant, but I find that it makes explicit almost as easy as implicit - but with more safety! I haven't seen that elsewhere, but I haven't looked very hard either. I have heard claims that Groovy/Hibernate do this just as well as well, but it isn't clear to me that this is completely true.

sidlls3• 10 years ago

Camp 3: those of us who think "shove something, anything that 'works' out as fast as possible" is a recipe for disaster, irrespective of the transport format or service infrastructure.

I suppose folks in this camp would overlap significantly with those in camp 2, though.

moefogs• 10 years ago

That's a really silly (and in my experience) inaccurate comment.

I like schemas ... most of the time! I like Avro ... some of the time! And JSON some of the time. And I write mostly in Python.... and Scala.

The world is not so black and white.

StreamBright2• 10 years ago

The third camp is the guys who know that you have to use schemas when it makes sense. :)

huntedsnark• 10 years ago

I don't understand the conflation of Rest/JSON with NoSQL. The most popular frameworks for the dynamic languages you cite all use SQL with strong schemas out of the box. Come to think of it I'm not sure why we're even associating Rest with JSON specifically or any content type for that matter when one of the major points of Rest is content-negotiation via media types. You can even implement a messaging concept with Protbufs in a strictly "rest-like" fashion.

oftenwrong• 10 years ago

At one time I implemented services that honoured the robustness principle - "be liberal in what you accept, and conservative in what you send". However, I have found that if you are liberal in what you accept, other services can end up dependent on quirks of your liberal acceptance. More recently I have become a believer in the "be conservative in both" strategy. Rigid is good.

wolfpwner• 10 years ago

But you are oftenwrong.

adekok2• 10 years ago

people who still naively think that Rest/JSON is cool and schemas and databases should be flexible and "NoSQL" is nice and

Yes, and no.

Yes: REST / JSON is nice. I've used them widely as a kind of cross-platform compatibility layer. i.e. instead of exposing SQL or something similar, the API is all REST / JSON. That lets everyone use tools they're familiar with, without learning about implementation details.

The REST / JSON system ends up being a thin shim layer over the underlying database. Which is usually SQL.

No: databases should NOT be flexible, and "NoSQL" has a very limited place.

SQL databases should be conservative in what they accept. Once you've inserted crap into the DB, it's hard to fix it.

"NoSQL" solutions are great for situations where you don't care about the data. Using NoSQL as a fast cache means you (mostly) have disk persistence when you need to reboot the server or application. If the data gets lost, you don't care, it's just a cache.

fail2fail2ban6• 10 years ago

SQL databases should be conservative in what they accept. Once you've inserted crap into the DB, it's hard to fix it.

You can make your schema very light and accepting almost like NoSQL which is how you get into the situation you described; the solution is to use stricter schema. That and it helps to hire a full time data engineer/administrator.

Using NoSQL as a fast cache

I'd rather use caching technology, specifically designed for caching, like Redis or Varnish or Squid.

teddyc• 10 years ago

"Once you've inserted crap into the DB, it's hard to fix it."

Agreed, it is hard to fix. NoSQL databases can be really hard to fix when they are full of crap, too.

solatic• 10 years ago

You're missing the point where you have a flexible, schemaless design while you're specing out the service and seeing if it's even necessary, viable, and will be needed for a long time for a long-lived system.

Properly encoding rigidity through type systems, SQL checks/triggers/conditions, etc is hard. It takes a really long time to really iron out all the string and integer/double/long typing out of your system, let alone do it in a way which matches up properly with your backing datastore. Once you've got it set up and nailed down with tests, then you're golden, but that's a cost that is usually not worth paying until long-term need is determined.

novaleaf• 10 years ago

Camps 1 and 2 are not mutually exclusive (well, except your inflammatory "naively" comment).

Rest/JSON is a well understood, broadly adopted, low friction RPC format.

NoSQL is not always MongoDB (for example, Google Datastore is ACID compliant), and schema enforcement via an ORM layer I would argue is actually a good thing, as it provides schema validation at compile time.

mrinterweb• 10 years ago

Rest/JSON is not a RPC. Technologies like Thrift can be used for RPC. It depends on how you define low friction for rest + JSON. JSON is schemaless, and that can be great for prototyping, but as soon as deployed services get out of sync, in terms of how they communicate, it becomes more of a burden than an advantage. Thrift, protobuf, avro, can enforce schemas and can raise exceptions for communication mismatches so less defensive programming is needed checking json responses. For internal service communication, I really think using a schema enforcing communication protocol is a good thing.

prodigal_erik• 10 years ago

The longer a database has existed, the more likely somebody in the company wrote something crucial that accesses it without your knowledge and without going through your ORM (usually because your ORM isn't implemented for the language they're using, or it emits bad queries for their use case). Sanity checks that aren't enforced by the database can't be relied on to be up to date or even happen at all.

maxlamb2• 10 years ago

Although companies like Snowflake are bridging the gap between the two camps, with products that can natively ingest JSON and Avro, and makes it queryable via SQL allowing for joins etc.

fail2fail2ban6• 10 years ago

It's possible to use SQL to retrieve data from a text file; that's not the same as using SQL backed by an actual relational database engine (and a well designed schema) that has been perfected over many decades.

taf2• 10 years ago

the short summary of what you describe is...

1. new business evolving quickly to meet and discover the product that fits the market they are chasing.

2. established and optimizing for a possibly still growing market but very well established set of features and use cases. They can take longer to deliver new features and can save lots of money by optimizing.

borlum• 10 years ago

Spot on

Osiris• 10 years ago

I've worked on both types of projects. My previous project used Scala on the backend and Node for the public API. Thrift was used to maintain persistent, reusable, connections between the public API and the backend, and to enforce strict typing on message values.

What I really liked about Thrift is that all I needed to know how to use the service was the thrift definition file. It was self-documenting.

My current project uses HTTP and JSON to communicate from the public API to the backend services. There is significantly more overhead (latency and bandwidth) and no enforced document structure (moving toward Swagger to help with that).

HTTP+JSON is great for the front-end where you need a more universally parsable response, but when you control the communication between two systems, something like Thrift/Protobuf solves a lot of problems that a common with REST-ish services.

opendomain• 10 years ago

No No No. This is like asking which is a better tool - a nail or screw? Since everyone is so familiar with SQL, they are avoiding very good reasons to use NoSQL.

'NoSQL' is just a broad term for datastores that do not normally use standard Structured Query Language to retrieve data. Most NoSQL do allow for for very structured data as well as some query languages that are similar to SQL.

BigTable (Hadoop, Cassandra, Dynamo) and block stores (Amazon S3, Redis, MemcacheD) are absolutely critical to cloud services. Json tuple document DBs are needed for mobile and messaging apps. Graph is for relationships, and Marklogic has an awesome NoSQL Db focused on XML.

Full disclosure: I am the founder of NoSQL.Org - but I also use multiple relational SQL databases every day.

herval• 10 years ago

(3) people who ponder what's the best solution for a given situation? Sometimes you want dynamic languages, or schema-less data. Sometimes you want rigid interfaces and verifiability.

Eg rolling out a public API in Thrift/protobuf will severely difficult its adoption, whereas Rest/JSON is pretty much the standard - but then building microservices that communicate to each other in Rest/JSON quickly leads to a costly, hard to maintain, inconsistent mess.

We're obsessed with "one fits all" absolutes in tech. We should have more "it depends" imho

dorianm• 10 years ago

I got told by an Uber engineer that's because of their hyper-hyper growth their tech is basically the biggest mess possible.

That's also why that makes it an interesting place to work at and helped them achieve this growth.

Personnaly I think this should be made into a global app with no geo-fencing (e.g. Available everywhere basically).

StreamBright2• 10 years ago

Absolutely amazing to watch. I think most of the big companies (Amazon, Google) already have solutions for these issues like: limited number of languages, SLAs between services, detailed performance metrics and the ability to trace microservices.

Roritharr• 10 years ago

I'm not a big green-IT guy, but always pushing your systems close to its load maxima and then backing off with the test traffic as real traffic comes in feels like an enormous waste of electricity.

taeric5• 10 years ago

Almost as much as the overhead of splitting everything to a thousand microservices...

petetnt• 10 years ago

Really enjoyed this talk. Our services don't quite (yet :)) run at that scale, but many of the issues mentioned have already peaked at some point. It's also good to have (more) validation to some choices we have made in the past, are currently making or are thinking about making in short term future.

agentultra4• 10 years ago

What I Wish Small Startups Had Known Before Implementing A Microservices Architecture:

Know your data. Are you serving ~1000 requests per second peak and have room to grow? You're not going to gain much efficiency by introducing engineering complexity, latency, and failure modes.

Best case scenario and your business performs better than expected... does that mean you have a theoretical upper bound in 100k rps? Still not going to gain much.

There are so many well-known strategies for coping with scale that I think the main take-away here for non-Uber companies is to start up-front with some performance characteristics to design for. Set the upper bound on your response times to X ms, over-fill data in order to keep the bound on API queries to 1-2 requests, etc.

Know your data and the program will reveal itself is the rule of thumb I use.

rando832• 10 years ago

I've never heard the term over-fill data, and google seems to have no idea of it in reference to software.

agentultra4• 10 years ago

Ah... it's almost the reason for GraphQL.

Basically if you're building a hypermedia REST API you return an entity or collection of entities whose identifiers allow you to fetch them from the service like so:

    {"result": "ok!"
     "users": ["/users/123", "/users/234"]
    }

The client, if interested, can use those URLs to fetch the entities from the collection that it is interested in. This poses a problem for mobile clients where you want to minimize network traffic... so you over-fill your data collection by returning the full entity in the collection.

    {"result": "ok!"
     "users": [{"id": 123, "name": "Foo"}, {"id": 234, "name": "Bar"}]
    }

The trade off is that you have to fetch the data for every entity in the collection, the entities they own, etc; and ship one really large response. The client would receive this giant string of bytes even if the client was only interested in a subset of the properties in the collection.

GraphQL does away with this problem on the client side rather elegantly by allowing the client to query for the specific properties they are interested in. You don't end up shipping more data than is necessary to fulfill a query. Nice!

... but the tradeoff there is that you lose the domain representation in your URLs since there are none.

threeseed6• 10 years ago

I have no idea why people keep thinking microservices is all about scalability. Almost like they've never worked on a problem with them before.

Microservices is all about taking a big problem and breaking it down into smaller components, defining the contracts between the components (which an API is), testing the components in isolation and most importantly deploying and running the components independently.

It's the fact that you can make a change to say the ShippingService and provided that you maintain the same API contract with the rest of your app you can deploy it as frequently as you wish knowing full well that you won't break anything else.

It also aligns better with the trend towards smaller container based deployment methods.

agentultra4• 10 years ago

I have no idea why people keep thinking microservices is all about scalability.

It's an aspect. It's often the beginning of a micro-service migration story in talks I've heard.

Microservices is all about taking a big problem and breaking it down into smaller components

... and putting the network between them. It's all well and good but the tradeoffs are not obvious there either. Most engineers I know who claim to be experts in distributed systems don't even know how to formally model their systems. This is manageable at a certain small scale as some computations take 35 or more steps to reveal themselves. But even the AWS team has realized that this architecture comes with the added burden of complex failure modes[0]. Obscenely complex failure modes that aren't detectable without formal models.

Even the presenter mentioned... why even HTTP? Why not a typed, serialized format on a robust message bus? Even then... race conditions in a highly distributed environment are terrible beasts.

[0] http://research.microsoft.com/en-us/um/people/lamport/tla/am...

You just have to know your data. The architecture comes after... and will change over time.

yummyfajitas• 10 years ago

You don't need to build a distributed system for that. Just build the ShippingServiceLibrary, let others import the jar file (or whatever) and maintain a stable interface.

I wrote about this a couple of years ago in more detail, so I'll just reference that: https://www.chrisstucchio.com/blog/2014/microservices_for_th...

HN discussion: https://news.ycombinator.com/item?id=8227721

marcosdumay3• 10 years ago

Because what you described is "encapsulation", and microservices is a very heavyweight and expensive way to get it.

duaneb3• 10 years ago

The main benefit of microservices is not performance, it is decoupling concerns. As scale goes up, performance goes down and decoupling gives a better marginal return: it's often easier to work with abstract networking concerns across a system than it is to tease apart often implicit dependenicies in a monolithic deployment.

Of course, this is context sensitive and not everyone has easily decouplable code bases, so YMMV. But not recognizing the myriad purposes isn't useful for criticism.

jfoutz• 10 years ago

I think it's worth noting, it doesn't just split the code. It also splits the teams. 5 engineers will coordinate automagically, they all know what everyone is working on.

At perhaps 15, breaking up into groups of 5 (or whatever) lets small groups build their features without stomping on anyone else's work. It cuts, not only coupling in the code, but coupling in what engineers are talking about.

The two sort of obvious risks, those teams probably should be reshuffled from time to time, so code stays consistent across the organization. If that's done with care, specific projects can get just the right people. If it's done poorly, you sort of wander aimlessly from meaningless project to meaningless project.

The other one is overall performance and reliability. When something fans out to 40 different microservices, tracking down the slow component is a real pain. Figuring out who is best to fix that can be even worse.

rhizome2• 10 years ago

a better marginal return: it's often easier to work with abstract networking concerns across a system than it is to tease apart often implicit dependenicies in a monolithic deployment.

Do the complexities introduced not have a cost, i.e. are those marginal returns offset by the choice in the first place? Call it "platform debt."

admiralhack_• 10 years ago

Great video. I went in expecting it to cover mostly the technical side of things. Instead Matt gave a great overview of the team / engineering organization dynamics to watch out for when adopting microservices. (I particularly liked that he pointed out how developers may write new code / services instead of dealing with team politics.)

eternalban2• 10 years ago

I think these "alarming trends" are highlighting that operational complexity is an easier pill to take than conceptual complexity, for most workers in the field.

Microservices address that gap.

And in the process the field is transformed from one of software developers to software operators. This generation is witnessing the transfer of the IT crew from the "back office" to the "boiler room".

majormajor2• 10 years ago

Do you think that's it's that people would rather deal with operational complexity, or just that it's easier to not think about operational complexity early on ("running locally, with low volume, everything works together great!")?

Personally, I vastly prefer debugging and building on non-microservice architectures than on something split willy-nilly into dozens of services (because most implementations I've seen don't do microservices with clean conceptual boundaries - it's more political/organizational divisions that determine what lands where, not architectural concerns).

user59944614• 10 years ago

>> Do you think that's it's that people would rather deal with operational complexity, or just that it's easier to not think about operational complexity early on

That's one way to scale development projects.

Have multiple teams of dev who work on separate stuff. They can develop in their corner however they want and that makes them happy. The final thing is a clusterfuck of services with little cohesion [YET they ALWAYS manage to put the shit together in production in a mostly working state].

The alternative implies to have a consistent and cohesive view of the components. For that, you'll need to have people with experience in architecturing scalable and sane production systems [so they understand what are the consequences and tradeoffs of ALL their decisions]. I know of very few people on the planet who can design systems. (We're talking past unicorn-rarity here). Plus, the developers must actually listen AND care about the long term maintenance AND the people involved (i.e. not want to do shit that will hit either them in 6 months or the team in the next office).

The amount of collaboration AND communication AND skills required to operate this strategy is beyond the realm of most people and organisation. There are very few individuals who can execute at this level.

eternalban2• 10 years ago

The inherent greater complexity of designing extensible and cohesive systems turns away a large subset right away and it never gets to a comparative matrix to inform the decision making process. It's been reduced to "monoliths bad, microservices good" level of thinking.

I honestly think that the rise of microservices as "the technical cure for cancer and everything else" is a very interesting surfacing of the various systemic disfunctions of our beloved software industry. The disfunctions have been present from day 1 but the environment has somewhat radically changed.

threeseed6• 10 years ago

You seem to be missing the point though. Most developers working on microservices projects don't have to deal with the operational complexity. They just need to work on their microservice and don't have to know what is going on elsewhere.

Personally I much prefer microservices for debugging. You can quickly identify which one is the problem then test the APIs in isolation pretty quickly. Sure beats having to wait 20 minutes for a monolithic app to build.

mstade• 10 years ago

I quite enjoyed watching this. My takeaway isn't so much that this is a critique or endorsement of microservices, but rather just a series of observations. Not even lessons learned in many cases, just honest pragmatic observations. I like how he doesn't seem to judge very much – he obviously has his own opinion of several of these topics, but seem to let that more or less slide in order to not get in the way of these observations.

Good talk, will likely watch again.

nichochar2• 10 years ago

Honestly I don't think the problem is microservices. I mean everything he brings up is true, but it's more of a "how you do microservices" issue.

I used to work in startups, and overall was impressed with velocity. Then I joined a big valley tech company, and now I understand.

It's because they hire smart and ambitious people, but give them a tiny vertical to work on. On a personal level, you WANT to build something, so you force it.

I think you solve this by designing your stack and then hiring meticulously with rules (like templates for each micro service), instead of HIRE ALL THE ENGINEERS and then figure it out (which is quite obviously uber's problem)

inthewoods3• 10 years ago

I found this video so super interesting and yet frustrating for completely personal reasons: the company I work for used to sell a tracing product that was specifically designed for the distributed tracing problem and handled all of the issues he highlighted - trace sampling, cross-language/framework support built in, etc. It was/is based on the same tech as Zipkin but is production ready. Sadly, he and his team must have spent a huge amount of time rolling their own rather than ever learning about our product. Now, it still might not have been a good match, but man, the problems he mentions were right in the sweet spot of what our product did really, really well.

dllthomas• 10 years ago

Sure, but was it Invented Here?

inthewoods3• 10 years ago

Yes, I think that is the primary problem - they invested in Zipkin. At the end of his talk, he mentions the idea that you should prioritize core functions and use vendors for everything else. But he's right that some person created that code - and thus it becomes difficult to get them to switch. And instead it becomes a huge source of technical debt for a company that is non-core to their mission.

jasonshen• 10 years ago

This is why sales and marketing is so so important and yet overlooked/undervalued. You might have a great product that's perfect for your customer, but you also have to convince them to use it and pay for it.

inthewoods3• 10 years ago

True - and marketing to developers is very challenging - a lot of noise to get above.

gb123• 10 years ago

Hey, Matt Ranney, I used your node-pcap library to learn how to parse PCAP :) Did not know you worked for Uber, thanks!

iamleppert2• 10 years ago

It amazes me they have 1,700 services. It would be hilarious satire to actually see the description of each. And the debugging scenarios he listed make it sound like they have very poor engineers working for them. Who on earth lets an application get to prod that iterates through a list of items and makes a request for each thing?

When did we loose our heads and think such an architecture is sane? The UNIX philosophy is do one thing and do it well, but that doesn't mean be foolish about the size of one said thing. Doing one thing means solving a problem, and limited the scope of said problem so as to have a cap on cognitive overhead, not having a notch in your "I have a service belt".

We don't see the LS command divided into 50 separate commands and git repo's.....

venantius• 10 years ago

While I'd concede that 1,700 services is a lot, and probably a function of the fact that they're hiring too many engineers too quickly (there's no possible way to indoctrinate people into sane and standardized engineering practices at this sort of hiring growth), I don't actually think what he's describing is that unusual for a company at Uber's scale or experiencing Uber's rate of growth.

I work at Airbnb and while we have many fewer engineers and services than Uber does, most of the issues he talked about in the talk resonated with me to one degree or another.

iamspoilt• 10 years ago

I couldn't agree more.

willejs2• 10 years ago

Saying something like '...they have very poor engineers working for them.' is pretty unfounded, and naive. It's easy to say that having 1700 services is overkill from our point of view, but we don't know the complete architecture, problems being solved and environment that they operate in.

One thing to possibly consider is that once you have set up tooling for a platform; logging, request tracing, alerting, orchestration, common libraries, standards and deployment. Deploying and operating new services becomes relatively straight forward.

Saying that, 1700 is a lot, which makes me intrigued to see inside uber.

bartread• 10 years ago

Yeah, that's completely ridiculous.

I'm currently working with a very (very) large UK-based retailer, with various outlets globally, and I can tell you from first hand experience that their selection process is ruthless. There are no, or at least very few, lamers in the tech team.

And yet, when you look at the systems, and the way things are built, some of it just seems crazy. Only it's not, and for many reasons, of which here are only a couple:

- The business has grown and evolved over time - with significant restructuring occurring either gradually, or more quickly and in a more intentionally managed way; systems have grown and evolved with it, often organically (which is something I see at every company I've worked with).

- Legacy systems that make it really hard to change things, and migrate to newer and better ways of doing things because they're still in production and depended upon by a host of other systems and services.

- The degree of interconnectedness between systems and services is high across the board; this isn't a reflection of bad design, so much as a reflection of the complexity of the business.

In any large organisation these kinds of things are apparent. I would imagine that in a large organisation that has become large very quickly, like Uber, if anything the problems would be magnified.

To say that they have poor engineers is therefore very unfair: these are people who would have needed to ship RIGHT NOW on a regular basis in order to facilitate the growth of the business. That's going to lead to technical compromises along the way, and there's little to be done to avoid it. There's also a difference between being aware of a problem and having time, or it being a priority, to fix it (e.g., iterating and retrieving).

iamleppert2• 10 years ago

This wasn't a derivation from the repo count, it was in response to the debug part of the talk where he was talking about fan out. Some app was iterating through a list of things and for each thing was making a network request, instead of figuring out what it needed up front. It's either complete laziness or incompetence, probably the later. I assume with any company that has that many services their culture is partly to blame, too.

It's interesting because this kind of technical debt will eventually open the door to competitors once Uber is under more financial pressure...i.e. they have burned through all the investor money.

stephansta• 10 years ago

When did we loose our heads and think such an architecture is sane?

It's not sane. That's the entire purpose of the talk.

What I Wish I Had Known Before Scaling Uber [video]

Related Stories