Domain-Oriented Microservice Architecture

https://eng.uber.com/microservice-architecture/

by amadk • 6 years ago

130 109 6 years ago

eng.uber.com

Twisol3• 6 years ago

Depending on what angle you come at this from, you could say that DOMA groups services into clusters ("domains"), as Uber has done here, or that services always should have been domain-driven, and DOMA welcomes the networking layer inside the bounded context as well.

I've spent a lot of time trying to understand when a microservices architecture makes sense, what the caveats are, and what philosophy one should take to building services. All the material I've read seems to point in the direction of services being ideally coupled to domain boundaries.

It seems to me that Uber's services proliferated beyond the framing of bounded contexts, and DOMA is their attempt to reign it back in again. I think it's an excellent strategy, and arguably a very good approach for other companies who find themselves in this position.

I don't think DOMA is a good place to stay at. The network should only be tolerated as long as it provides benefits that outweigh the costs. "Monolith" is not synonymous with "poor design". Seeing that these enclaves of services sitting within a domain depend on eachother in the way the OP describes, it really makes me think that they'd find further benefits by expelling the network from each domain.

adamkl2• 6 years ago

The network should only be tolerated as long as it provides benefits that outweigh the costs.

I agree. There should be no need to use network calls to enforce interface boundaries if you have a cohesive bounded context.

Or to be snarky about it, welcome to 2004 Uber! Eric Evans sends his regards!

gen2203• 6 years ago

Unfortunately, picking and enforcing bounded contexts is hard work in a big organization, even with strong code review processes.

There will always be people who don’t want to respect where the boundaries are drawn (not in a constructive, “it could be better” kind of way, but in a “but this works, too” kind of way). If a group of such people get together, microservices compartmentalizes their capacity to drag down the ship, so to speak. I think this risk compartmentalization is a benefit that must be weighed against the costs (in terms of latency, maintenance of shared libs, opentracing, etc). These days the costs are vanishing, as tools are quite good and becoming easier to manage.

All that said, if you’re a small team of senior engineers working with a shared mental model, a single binary with internally-bounded contexts works really well and I agree with you, having seen it done well.

blunderfel• 6 years ago

I'm a big fan of the modular monolith pattern. I usually make domain modules as independent as possible and invert dependencies for web and persistence layers. If you design well you can break off domain-based services whenever the advantages warrant it.

gen2203• 6 years ago

If you ever have to start a big company from scratch, this is how to do it.

There aren’t any major drawbacks to this model when a business is young (first couple years). The downsides appear when you have different parts of your application with very different load requirements.

It also takes a lot of discipline to write code this way. Without strict code review and more experienced hands, the bounded contexts fall apart.

One of the advantages of the microservices model is it limits the damage people can do. :)

It forces a bounded context on a team of engineers and says “hey, play in this sandbox and follow these SLAs. If your internal designs are awful, good luck.”

theptip2• 6 years ago

As far as I can tell from reading Evans' DDD, there's nothing forbidding network calls inside a service. For a trivial example, you make network calls from your API to your DB. And also to your Redis cache, and then if your service runs async/periodic tasks, to your task queue, say your RMQ & Celery instances.

So to me the OP reads like "we're coming up with some new terminology for a bounded context, and also defining how those contexts should be allowed to layer in order to simplify/control failure modes".

The layering stuff is more interesting than Uber's rediscovering bounded contexts, though it's definitely interesting that they have come into agreement with Evans (and the rest of the DDD community) on the "Service == Bounded Context" principle.

Twisol3• 6 years ago

As far as I can tell from reading Evans' DDD, there's nothing forbidding network calls inside a service.

You're quite right.

[...] defining how those contexts should be allowed to layer [...]

This, however, seems to go against the spirit of things. There is a consistent "ubiquitous language" within a bounded context, where domain terms are concrete and unambiguous. (Or rather, the context disambiguates the language.) The concept of "layered contexts" seems to neuter the concept. Does each layer successively disambiguate the one above? Or does it add new terms that didn't exist?

The layering here sounds much more technically-motivated than domain-oriented. And my argument is that the networking internal to their "domains" is largely an artifact of having build DOMA out of a plethora of disorganized microservices. Doubtless there will be some necessary networking remaining, as you remarked on, such as between processors and databases. But the origin here suggests most of it is left over from what came before.

recursivedoubts• 6 years ago

Seems to me that big progress will be made when we can decouple system boundaries from network calls.

bob10292• 6 years ago

I still think the approach of baking the entire enterprise into 1 gigantic binary is the best. Stack traces are so much easier to work with compared to distributed logs and side effects.

We are already at a point where you could hypothetically do this for a reasonably-sized organization. A 64 core CPU can support a huge number of clients. Stuff like .NET Core scales really well if you want to build something complex like this. One big binary that occupies an entire physical host is an extremely compelling development model. Literally everything becomes a direct method invocation. You can also have type enforcement and atomic releases for the entire enterprise. Also makes a monorepo an obvious choice for source code management.

jzoch• 6 years ago

This falls apart when you care a lot about availability and therefore want a particular service to be stateless. Then you add some feature that needs a tiny bit of state and suddenly you can't scale up and down as easily and need to start paying attention. This rolls forward into a big ball of mud - a monolith can be tricky too at scale

jen20• 6 years ago

Indeed - this is the premise of the "4+1 View Model of Software Architecture" paper from 1995 [1].

[1] https://www.cs.ubc.ca/~gregor/teaching/papers/4+1view-archit...

Twisol3• 6 years ago

Thanks for the paper. Where do you find that it discusses decoupling the concepts of network calls and system boundaries?

I must be honest: aside from the context in which it was linked, I came away terribly unimpressed by this paper. I found myself disagreeing on a number of points. Having only spent half an hour on it, I can't claim to have a major problem with it, but I wouldn't couch my architectural work against it.

sdan• 6 years ago

Basically clumping microservices

deltaeerie• 6 years ago

I believe this is only the first point. It's also establishing a hierarchy of these "clumps" (layers), strict APIs representing a single clump for other clumps to consume (gateways), and a pattern for clumps to extend functionality of other clumps without polluting their models (extension architecture).

dkersten2• 6 years ago

Somewhat off topic but maybe someone can enlighten me:

Uber has grown to around 2,200 critical microservices

Even thinking about the very largest systems I’ve worked on, I can’t think of what could possibly be split into 2000 separate individual services. What are these thousands of microservices and how micro are they?

nevir• 6 years ago

I'd be curious how that compares to Amazon's count of services (my guess is 10k+)

beastcoast• 6 years ago

I’d say we’re closer to an order of magnitude larger When you consider all the different businesses and domains. My org alone has about 1000 services. But we have massive scale and the employees to support that number of services. You also won’t find one architecture; it runs the gamut from lambda to REST/RPC to pubsub, event bus and workflow architectures. Typically new orgs start out with a few larger services that get broken out over time, as the complexity and number of employees continues to grow.

drol3• 6 years ago

I am working for a company that has moved from 3 to 15 backend developers and 25 to 50 services over the last year.

The amount of services is really quite meaninglessness compared to the amount of developers. As with anything the efficiency good down as the amount of people goes up. There is no way that 2000 people can agree on anything. So most likely there is lots and lots of overlap.

You could probably refactor some of this duplication out, but by the time you would be done new duplication would have emerged. Keeping many people in sync is just difficult :)

Anyway the point is that 2200 services is meaningless without telling how many employees are working on said services :)

eikenberry• 6 years ago

Remember that those aren't 2200 separate things going on, but 2200 instances of the services. I'm sure that includes very large clusters running many copies of the same service.

wikibob• 6 years ago

That is an incorrect interpretation.

I assure you there are /significantly/ more than 2200 instances of the services running.

They do indeed mean 2200 separate deployable services.

Ozzie_osman• 6 years ago

Not at Uber, but I've heard a service there can be as small as a database table or two with some CRUD. Or a lambda-style function or two that provides some simple logic when queried or aggregates from a couple other services.

Not saying that's a good way to build systems but it's definitely one way.

dkersten2• 6 years ago

Who has 2000 database tables though? Or 2000 lambda functions for that matter.

daxfohl• 6 years ago

For every important service there is the current one, the old one that is still in use, the previous previous version that one dumb team still hasn't migrated from, the next gen prototype, the service that deploys the service, the service that monitors it, the data analysis tool, data migration services, the intern project, several upstream services that continuously test the integration points, blah blah blah.

Then you look at the verticals and there's maps, payments, analytics, hosting, marketing, security, identity, partnerships, third party integrations, and a million other things you don't think about.

It's pretty easy to get into those numbers if you want zero downtime and code your own verticals, and have the resources to do it.

jeswin• 6 years ago

I don't find this surprising at all. Uber operates across 5 dozen countries (or more) each with different regulations and offerings.

For instance, in India you get autorickshaws (called "Autos" sometimes) in addition to cabs as a seperate option. Autos have different rules, billing, driver compliance and safety standards. There are probably a bunch of services around Autos.

Similarly, safety regulatons differ in each country, and in some cases teams are forced to act quickly. Having them in separate microservices again makes sense. Same goes for offers, eats etc.

2000 services is not inconceivable for a company operating across the world.

etxm4• 6 years ago

pico services.

jmchuster3• 6 years ago

Hmm, would it be easier to imagine PayPal or Stripe requiring a lot of services? Given its volume of payments, Uber has basically had to re-implement PayPal internally, custom-tailored to its own flow. Companies at this size/scale basically contain multiple whole companies within them. Think of like all the SAAS services that your company relies on, and then imagine that your company now owns all of them and all the services that come with them.

clintonb• 6 years ago

Uber uses Stripe and PayPal to process payments. They don’t need to reimplement either, and doing so would be a bit of a waste.

It’s worth pointing out that not all of the 2200 services may be user-facing. Some may be internal, such as admin tooling or CI services. That said, 2200 seems like a lot!

xcskier56• 6 years ago

While I work on much smaller codebases and so don’t have any particular insight into how DOMA will work for such large organizations, the piece that I really liked here is how the separate their different general layers of services.

The key to any service being usable outside of the exact context it was first written in is to ensure no product specific business logic is added to it.

By splitting out infrastructure from general business from product specific services, you can do a much better job of understanding, and therefore controlling where where product specific logic is allowed. This in turn will make your lower level services far less coupled to the exact context they are first used in.

For smaller orgs, this is by far the more useful information, rather than how to deal with 2200 microservices.

nfoz• 6 years ago

I don't like that they define microservices as components communicating via RPC, because this precludes other models like pub-sub.

harel• 6 years ago

I'm not entirely sure why this is sold as innovative new approach. Over the last few years, I've seen and developed micro services that are indeed cut down to specific functionality, and I've done ones where the domain was the scope. The latter has proven to produce bigger services, but a more maintainable ecosystem. Also, the division for micro services should also be dictated by the context of the business, and not just the current whims of the devs.

kopos• 6 years ago

Feels like a lot of old knowledge was discounted / ignored and the mistakes are being rediscovered.

Feels a bit like the Hexagonal Architecture being rediscovered.

Micro service with half life of 1.5 years? Does that mean that enough planning is not being done? Or leadership failure at software planning level?

Collaboration between teams at scale is very hard but that is what the leadership layer is for - to collaborate more not build more micro services.

sixdimensional2• 6 years ago

Or how about plain old onion architecture... even older than hexagonal.

This is literally scaled/distributed domain-driven design (DDD).

I have felt strongly that folks got so caught up in the hype of yet another new thing they forgot how to extend what came before.

It feels like a reinvention of existing ideas at larger scale, and a pattern we keep repeating.

I'm not complaining, of course - new things are possible and being learned through this innovation. I do feel we should be more careful on the cutting edge, to see how it relates to where we came from.

Put another way, reinventing the wheel is not pointless if you come out with a better thing, or better wheel. But don't forget what was good about the previous wheel before you throw it out?

staticassertion2• 6 years ago

I'm confused. You're saying that people are reinventing existing ideas at larger scale, but you also are saying that people don't know how to extend what came before? Those sound like the same exact thing.

pjmlp7• 6 years ago

It means CV driven development for selling conference talks, blog posts,....

tannhaeuser• 6 years ago

I have to agree. Articles about software architecture, figuratively speaking, read like the result of playing connect-the-dots using the buzzwords du jour, as thrown into the mix by cloud providers to advertise solutions to problems no one is having. It's not an entirely new phenomenon, but consumerism, big media dynamics, advertising to clueless decision makers, and self-fulfilling resume padding in IT seems to be the norm this decade.

levischoen• 6 years ago

Yup. So micro services of micro services. Clearly defined interfaces. Single entry point for other services/domains to interact with. seperation of concerns. The fundamental law will always be true : https://en.m.wikipedia.org/wiki/Fundamental_theorem_of_softw...

etxm4• 6 years ago

I’m honestly not trying to be snarky here but I didn’t know there was any other way to design microservices (with intention) that weren’t domain oriented.

timclark• 6 years ago

I’ve seen one service per database table, and have had architects seriously think that a table equals a bounded context. Then they try to work out how to do two-phase commit across micro services.

etxm4• 6 years ago

Oh that. I’ve had to merge services like that.

People are always grabbing nouns and thinking they’re microservices.

fma• 6 years ago

I'm on the same boat.

"Previously product teams would have to call numerous downstream services to leverage a domain; they now have to call just one....Furthermore, we were able to classify 2200 microservices into 70 domains."

So they went from 2200 to 70 microservices? One extreme to the other. The answer is somewhere in the middle.

sagichmal6• 6 years ago

In other words, organizations adopt microservices for an operational benefit at the expense of performance.

...no, not at all. There are a couple of operational benefits, but vastly more drawbacks, and on balance microservices are phenomenally harder to operate than monoliths.

Organizations adopt microservices when the logistical overhead of coordinating teams against a monolith becomes so large that it starts affecting product velocity. Microservices are a way to release that organizational/logistical friction, at great technical cost.

With that said, the domain-oriented bounded context is indeed the right way to think about service delineation.

scarface745• 6 years ago

The last company I worked for was very much an “API first” company. We were a B2B company and our APIs were used by our relatively lightly used website. But they were also used for sporadic and large ETL jobs when files from customers came in and most importantly we sold access to our APIs that were used for our customer’s mobile apps and heavily trafficked website.

We went microservices for the canonical reason you should - to be able to release and scale independently.

sagichmal6• 6 years ago

That reason is oft touted but experience has pretty unambiguously revealed that it never pans out in practice. I mean you do get that benefit but the price you pay for it is absolutely disproportionate if that's the only thing you want. Opting in to microservices purely for presumed technical benefits is absolutely a mistake.

rumanator3• 6 years ago

Organizations adopt microservices when the logistical overhead of coordinating teams against a monolith becomes so large that it starts affecting product velocity.

That's certainly one of the operational limits, but arguably not the most important one.

You distribute your system so that parts of it may scale independently, for example. Traffic fluctuates along time and the only option you have to scale your system is adding nodes as you go.

sagichmal6• 6 years ago

The ability to scale services independently is not as impactful as many people believe, and changing your entire architecture for no other reason than to get that specific benefit is a mistake in the vast majority of cases.

Microservices are a solution to organizational problems, not technical problems. On balance they create far more technical issues than they solve.

systematical• 6 years ago

I'm looking forward to the article about the problems DOMA introduced for them and what they came up with next. Wonder if we will see a full circle back to monoliths.

I know I'm being snarky and I have used micro-services myself, but only when it was smacking me in the face as the best tool for the job. Is the fools-gold rush still on to do everything as a micro-service from the get-go?

Side note, I can't wait to hear my boss use "DOMA" in a meeting in the coming months. FML.

javajosh• 6 years ago

Is the fools-gold rush still on to do everything as a micro-service from the get-go?

Yes.

staticassertion2• 6 years ago

I'm a fool who has done microservices first and is very happy about it. Just want to throw that out there into the sea of negativity I see towards microservices.

random_kris• 6 years ago

I was hired and was told I'll be building ONE of the microservises for their upcoming kubernetes platform. A year later I have written 5 microservises and I am responsible for managing all 5 of them. It is hell.

rumanator3• 6 years ago

Wonder if we will see a full circle back to monoliths.

For global-scale web applications? Obviously you won't. High-availability, low latency, resilience, scalability, performance. You don't get any of that by running your app on a single box. That ship has sailed two or three decades ago. Physics establishes all the limits, not software architects.

Distributed system critics, where they fixate on trendy microservices architectures or poopoo other suggestions like DOMA, should take a step back and look at themselves and what they are actually complaining about. Yes, a solution with no moving parts is simpler than a solution with some moving parts. But have you really noticed what problems are being solved by adding these pieces?

inopinatus6• 6 years ago

Monoliths don't run on a single box. My entire current business runs on a single monolithic application, and it's running across multiple AZs with five different instance roles (consumer site, API endpoint, admin site, background jobs, and reporting), sized by memory demand and contention efficiency, then scaled horizontally according to workload demand.

They are all, however, running exactly the same code, just in different configuration. I'd say it's roughly speaking 80% core libraries and the remaining 20% varies by role.

I have a command-line control & observation tool that comes in bin/ of the same repository, and it is again wrapped around the same code besides.

That's the modern monolith in production.

pjmlp7• 6 years ago

Yes, doing distributed systems programming since around 1996.

List of stacks I have used in some form since then, raw TCP/IP for in-house RPC protocol, SUN RPC, RMI, COM/DCOM, XML-RPC, SOAP, CORBA, REST, WCF and apparently gRPC is the new fashion.

At the same time, I also done modular development with teams responsible for modules, where the language features for creating modules, defining interfaces, and use binary dependencies are taken into use.

What I usually see with most "distributed systems" is that they are used as a physical solution for teams that never written a proper module in their life.

If monoliths with total lack of modularity are hard to debug, spaghetti network calls are even less fun.

pjmlp7• 6 years ago

Sun RPC marketing message, "The network is the computer".

So yeah, we keep going at this, and then people discover that monoliths written in a correct modular way, with libraries, happen to be easier to debug and reason about without a network in the middle.

Main problem seems to be that not many developers bother to read about modular programming, large scale development (like Lakos books) and what features their language of choice offers for such endeavours.

yjftsjthsd-h• 6 years ago

Alright, you've piqued my interest; any pointers for where to learn this stuff, or should I just Google all those terms?

playing_colours• 6 years ago

Can you recommend amytging to read? Lakos seems to be C++ oriented.

eternalban• 6 years ago

Are you sure that was just a marketing line for "RPC"? I thought the scope was broader.

IMO Micro-Services were addressing the skill gap in designing comprehensive schemas, not so much the object layer between user and data. So not modular "programming" but rather "modular design".

sroussey• 6 years ago

Uber thinks they only have that many services? Yikes, double, or triple the number... several years ago. Maybe they have fewer now?

vmurthy• 6 years ago

I recently read Domain Modelling made Functional [0] and one of the under-recognised benefits of using Domain Driven or oriented designs is that (if done right), business workflows and processes become well explained for developers and minimises surprises from business closer to release dates. For e.g., an order processing system will have lesser assumptions by developers and tech leads because someone with enough knowledge of the business process from order processing will be able to guide you.

[0]https://pragprog.com/titles/swdddf/

ianamartin• 6 years ago

I despise Uber as a company, but I've always loved them as a technology group ever since I used to go to their meetups in NYC. You can always count on Uber to throw a great party and also come up with the worst possible technology.

If you want to know how not to do things, Uber is a very good place to look.

hinkley• 6 years ago

And all the people who worked there will consider the fact that they did 1) something and 2) were successful means that that 'something' had anything to do with it, and therefore we should keep doing it.

Not that this is unique to Uber, mind. I just described half of my coworkers too.

exnuber• 6 years ago

They say they draw inspiration from DDD and CQRS, yet seem to miss one crucial factor. And no, I'm not talking about bounded contexts. They still focus in re-use of commodity rather than addressing why their microservice approach didn't work. Their strategy for defining how functionality moves down the stack is not bad, but it's not anchored towards the right thing. It's the consistency boundary (or aggregate root if you will). It's the circle you draw around your business rules and say "this group needs to remain consistently enforced". This defines your unit of scale, not re-use. So while they are headed in the right direction, I fear that they are still missing a fundamental piece of information to steer on.

I was employed by Uber (I quit after a couple months), and the idea's that they now hint towards were largely rejected by the engineers, and that was less than a year ago. Uber is just in the position to throw a large sum of money at making wrong decisions and getting away with it, because it's not their money. It's VC funny money.

DISCLAIMER: Yes, I did indeed create this account to be able to reply anonymously.

pgt• 6 years ago

Microservices are about scaling teams, not tech.

keyraycheck• 6 years ago

as Uber grew from 10s to 100s of engineers with multiple teams owning pieces of the tech stack, the monolithic architecture tied the fate of teams together and made it difficult to operate independently.

I think this is key sentence. Microservices does not make a lot of sense for teams of 10s, while are a great tool for team of 100s.

kerng• 6 years ago

Very interesting and seems like a logical next step with things get a bit too messy with micro services.

One thing that was not touched on - how is security done in DOMA?

* Is auth happening at domain gateways or at each service?

* Similarly for encryption, does it terminate at Domain GW or at each service?

alpmeow• 6 years ago

Hopefully both

SergeAx• 6 years ago

I felt bad for Uber engineering force while reading whole article. They swamped themselves into that microservices hell due to pressure for teams velocity and TTM, and created enormous tech debt in the process. Now they are trying to patch it up with ridiculous means like putting JSON inside of protobuf messages. I really hope this article could be a warning for management of growing startups for what could happen if you squeeze features out of your team, but managers usually speed-reading those articles just to order "do like Uber did".

stevenalowe• 6 years ago

Nice to see Uber rediscovering domain-driven design principles.

mbrodersen• 6 years ago

If you can't make a monolith work, then you will fail even harder trying to make a Microservice Architecture work.

Domain-Oriented Microservice Architecture

Related Stories