Microservices Ending Up as a Distributed Monolith

http://www.infoq.com/news/2016/02/services-distributed-monolith?utm_source=infoq&utm_medium=popular_widget&utm_campaign=popular_content_list&utm_content=homepage

by adamnemecek • 10 years ago

154 51 10 years ago

infoq.com

austingunter• 10 years ago

You can also view the original talk at www.microservices.com/ben-christensen-do-not-build-a-distributed-monolith

wcummings• 10 years ago

I've worked on what I think could be called a "distributed monolith" and its not bad, services and clients can share interfaces, common build and deployment infrastructure for everything etc.. You still get the benefit of being able to scale out, separate failures and deploy components easily, w/ the benefits of standardizing around one language.

maxxxxx• 10 years ago

We do the same. A monolith gets copied to all stations which then run different parts of it. That way they all have the same infrastructure code available.

pm903• 10 years ago

How do you deal with services that use different versions? Tight coupling means that to make changes to Service A you have to wait for Service B to make the changes...which can be a loooonnngg time if the team looking after Service B is different from your own.

jonesb6• 10 years ago

Isn't that what's supposed to happen? You don't distribute small systems, you distribute big systems, otherwise the benefits of microservices are way fewer.

bkovacev• 10 years ago

People jump on the microservices bandwagon way too often. I'm glad that Jan pointed out that not everything that is broken down is a micro service.

yahyaheee• 10 years ago

I love this article. Microservices are as much about coupling as they are about scaling. It reminds me of the recent notion of disposable code. We need to get more comfortable with the idea that decoupling likely has more long term benefits than reusablilty.

deepsun2• 10 years ago

I don't understand how Guava couples my services, could anyone explain me?

Let's say I have service A that uses Guava v18, and service B that uses Guava v19. Neither of them care what other one use, all they see is RPC (or HTTP) calls.

hatchoo• 10 years ago

I don't get it either. If I use Spring WS to provide the endpoints, how does it make it a monolith? It's always using the right tool for the right job but if one set of libraries are the right tools, why not use it across everything where appropriate.

jupp0r• 10 years ago

One of the benefits of microservices are your ability to choose the right tools for each of them instead of a common denominator. I don't know about Spring WS, but if it makes harder for ie a Go microservice (or whatever makes sense for your domain) to talk to your service then I would consider it a liability.

skybrian• 10 years ago

That assumes they were written independently. But if your services have a large heap of shared libraries in common, they all need to work together. Then when you upgrade Guava, there may be a lot of code to fix.

deepsun2• 10 years ago

Thank you. But if my services have a large heap of shared libraries in common, it's by definition a monolith. With Guava, or without. I shouldn't call it "microservices architecture" in the first place.

What I always meant by "microservices" is whatever servers talking to each other by some kind of RPC, e.g. Protobuf in Google (HTTP+JSON works as well if you don't operate terabytes). Then it doesn't matter what Guava version, or even what programming language my services are written in, as long as they support RPC format (e.g. Protobuf).

So I conclude that original post author does microservices wrong. Or just don't understand what he's talking about. Sad to see it on InfoQ.

alcima• 10 years ago

I think his point is that if ALL of your services have to use Guava (perhaps because you included some functionality that only Guava provides) then you have coupled your service to Guava. You can't use nodejs to build a service.

Randgalt4• 10 years ago

I was the guy at Netflix that had the task of moving the Company from Guava 10 to Guava 11. Guava 11 was backward-incompatible with Guava 10. On the Platform team (produced libraries the entire company consumed) we could only move to Guava 11 if EVERYONE moved to Guava 11. The only alternative would have been to shade Guava 11 into the platform library which wasn't pleasant.

BurningFrog2• 10 years ago

That sounds like a real impractical way to run a platform team.

Moving the entire company codebase at the same time may be possible with a dozen developers, but when you reach Netflix size it seems impossible.

What am I missing?

foota2• 10 years ago

What is shading a library?

fs111• 10 years ago

Friends don't let friends use guava.

Randgalt4• 10 years ago

I should mention that the reason this was such a huge problem was that our dependency system at the time gave each group the option of using the "latest" version of the platform library instead of fixed version. Everyone used it. If, instead, people had locked to a known version of the library it would've been so much easier to upgrade.

platform• 10 years ago

I have used Apache Storm and Erlang OTP. And cannot still grasp how micro-services are different from Storm's bolt's or Erlang's processes.

While subjective, for a JVM-based solution stack, Apache Storm appears to be more elegant, complete and efficient than Linkerd, as an example.

May be advantage of microservices vs Storm becomes more apparent with scale, and I just simply had not had the experience at that level.

kashif• 10 years ago

Something I posted to clarify microservices and SOA - https://medium.com/@kashifrazzaqui/will-the-real-micro-servi...

agentgt6• 10 years ago

Message Bus != SOA. What the prior poster is alluding too is the Actor pattern and/or Message passing pattern.

My problem with microservices is that your often picking subpar solutions/technologies particularly if low latency is a priority. For example HTTP is not always the best protocol and Request/Reply is not always the best messaging pattern but this is what most people equate to microservice (ie HTTP 1.1 REST a lightweight list of servers (etcd, zookeeper, consuler) or a load balancer).

ultimape• 10 years ago

This is really good. I also think about it in terms of ants and distributed systems.

Thanks for sharing.

im_down_w_otp• 10 years ago

A microservice would be more like a well-structured, logically-bounded Erlang OTP Application (as in literally the Application pattern provided by OTP), than a single process.

You could run the OTP Application, and a bunch of others, locally. You could stick some on a different machine/cluster. Everything is nice and encapsulated and distributable.

The only issue is this would still violate the concept in the OP because it forces everything to either behave like an OTP Application or at a minimum to behave like an Erlang node.

agentgt6• 10 years ago

I find this pretty hilarious coming from the guy who wrote Hystrix of which was extremely tightly coupled to another library called Archaius of which pulls a gazillion other dependencies (that being said I still highly respect Ben).

I had to spend a couple of days decoupling the mess [1].

The other annoying thing with Hystrix is that its basically Request/Reply pattern except its even worse because it uses inheritance (command pattern) and a massive singleton (HystrixPlugins). Furthermore although it uses a streaming library (rxjava) it sort of disingenuously provides very little actual support for streaming.

You don't need to have separate executables to make things decoupled but rather a good build system and architecture. Some of the ways are avoiding Request/Reply, singletons, and the command pattern.

We achieve fairly good decoupling by using a language agnostic custom message bus (it's over RabbitMQ which is AMQP... tons of clients) but you could use an actor framework or as another poster (@platform) mentioned Storm and Erlang process if you want to stick to one language.

[1] https://github.com/Netflix/Hystrix/pull/1083

ryeguy• 10 years ago

What does the custom message bus offer over rabbitmq? RMQ seems pretty fully featured already.

agentgt6• 10 years ago

The custom part isn't really custom. The custom part is just using RMQ consistently.. clients follow some guideines. Its basically saying we support Cap'n Proto, JSON, and Protobuff for the message body (one day we will pick one but alas...) and do pretty much zero routing (each message type goes to its own queue).

However we do do some endpoint routing independent of RMQ where if the client pushes a message to the "bus" and we detect the same client can consume that message we will sometimes avoid sending it over RMQ (ie local publish) (basically a performance enhancement to encourage bus usage for decoupling while avoiding the network hop for low latency).

By using protocols like AMQP and serialization that are language agnostic and focusing on doing very little routing we could switch to zmq, kafka, HTTP2 or whatever is in vogue if we wanted to.

With REST you have serious contract complexity: URIs, HTTP headers, POST form parameters, HTTP method, query parameters and the HTTP body. With an async message bus The message is the contract.

fritzo• 10 years ago

Do you have some good examples of why not to use the Request/Reply pattern? I've been trying to understand why some distributed dataflow systems prefer pushing rather than pulling.

gizzlon2• 10 years ago

Let's say you want to push data from the server to a mobile/web app in "real time". Examples I've come across lately would be a chat application and real time map updates (a la Uber).

Now your clients could either: (1) Pull the server every 0.5 second, or (2) Open a connection that stays open and let the server push messages across the connection.

Request/Reply and pulling is not bad in itself, but in some cases it has two downsides: (1) It delays everything because you can only pull so often. (2) I's bad for performance and scalability because all those pulls have to be handled, even when there's nothing to return.

Now you want to create a backend that also avoids pulling in the relevant places.

agentgt6• 10 years ago

There are myriad of reasons (I can't find a comprehensive link) but I will try to list one or two. Some even have mixed feelings since Request/Reply is so ubiquitous and very easy to understand/implement that it makes up for it.

Request/Reply is obviously not good if your problem is inherently push based ie Pub/Sub. This is because of performance reasons (could be argued) but also because Req/Rep does not fit naturally to pub/sub problems (you can think of pub/sub problems as live streams of data like stocks or chat systems).

Request/Reply pattern requires very smart endpoints. For some reason this is extolled heavily in the microservice crowd because it avoids single points of a failure. However smart endpoints need to deal with server discovery, timeouts, circuit breaking, metrics, backpressure and much more. This logic often gets put into a library and suddenly becomes part of ALL clients which as the presentation mentions is bad (this is what Hystrix does to some extent). For example having all your clients depend on ZooKeeper to find some servers is pretty coupling (this is what Archaius does).

That being said the above can be mitigated by making sure communication doesn't absolutely rely on the endpoints having the same intelligence.

Message Buses like RMQ avoid this issue because the pipe is smart. Your clients don't need to have the above logic which makes implementing clients in many languages far easier... but at a big cost: single point of failure and an extra network hop (broker).

Like smart endpoint problems smart pipe problems can be mitigated as well (e.g. RMQ has cluster support).

We use a mix of both patterns and in both cases have not-so-dumb endpoints and not-so-dumb pipes.

watty• 10 years ago

"Do as I say, not as I do". It looks to me as if Ben bundled Archaius several years ago and just recently made it a "soft dependency" which isn't required. Couldn't you say he's learned that a "gazillion other dependencies" are bad over the past several years and has learned enough to share his knowledge and implement it?

And then critiquing other parts of his code which have nothing to do with his talk is completely unnecessary.

agentgt6• 10 years ago

And then critiquing other parts of his code which have nothing to do with his talk is completely unnecessary.

It's completely relevant as its many reasons why Hystrix is very coupling which is exactly what was discussed to avoid in the presentation. I think Ben would agree and I'm sure he has learned from his experience.

The tone might sound disparaging but as I said I truly hold both the library and Ben in very high regards (otherwise why would I fix it). I suppose I have a weird since of humor because I thought hey.. I just fixed that library that had these problems and the creator is talking about those problems :) .... NOT Ben is a moron and should have done it right.

eikenberry• 10 years ago

You could be more generous in your reading and assume he has learned from past errors (hystrix).

paulddraper• 10 years ago

There's compile-time DRY, and then there's run-time DRY.

* Compile-time DRY allows me to change my code quickly and reliably.

* Run-time DRY allows me to deploy services independently.

Sometimes these objectives are at odds. Sharing code among my services, can make my code less repetitive at compile time, but it can also lead to duplicated execution at run time. And vice versa.

The important thing (besides reading the latest blogs and following the latest tech trends, of course) is to figure out what your objective is.

For example, at my company, we want (1) to improve the ease of code development, and (2) to increase production stability for critical functionality.

So we have microservices that allow us to prevent non-critical issues from taking out critical functions. And we use extensive share code for HTTP, JSON, templating, logging, some business logic etc. in order to make development easier. We don't care about decoupling releases or segmenting production ownership. In other words, run time DRY isn't important to us. Our services are simply ways to intelligently distribute execution across limited resources.

Assess your objectives, and then choose the most solution that will meet them. Never do the reverse.

If you need separation of concerns and information hiding at runtime, this article has some good tips.

tunesmith• 10 years ago

So, if someone creates a DropWizard endpoint, and someone else creates a Spring @RestController endpoint, and someone else creates a Scala Play endpoint, then the problem becomes that someone that knows Spring but not Scala might find themselves tasked with supporting the Play codebase, which is scary. So then the employer seeks to standardize on Spring, but that's apparently bad because if every new endpoint requires Spring then that's a Distributed Monolith...

benjaminwootton• 10 years ago

A lot of Microservices advocates talk about abandoning DRY to reduce code coupling.

I think that's completely impractical and that we should still make heavy use of shared abstractions in modules and libraries.

If I have a routine in two application services, e.g. a currency converter, and that code has to be changed, I have a few options:

With code duplication (Abandon DRY):

1. If the domain demands, I have to change the routine in multiple services and build, test and deploy both of those services in lockstep.

2. If the domain allows, I can change it in one service and build, test and deploy that one service in the short term. The other service will likely need to be updated at some point in the future. This is hard to manage and brings duplication of effort.

Without code duplication (DRY):

3. If the domain demands, I have to change it in one shared library and then test and deploy both of the dependent services in lockstep.

4. If the domain allows, I can change one shared library and build test and deploy only the one service that I need to change in the short term. The other service will likely need to be deployed at some point in the the future.

Shared libraries are orthogonal to the question of loose or tight coupling. We can have no shared libraries but still have two services which are tightly coupled and chatty. That is the situation we need to get away from, and it is acheived through good DDD and service boundaries, but really doesn't seem to have much to do with shared libraries.

pm903• 10 years ago

I think the problem comes in when the shared library is used to provide some kind of baseline service, so that if changes are made to the library and 2 services using them are out of sync, then chaos will result.

The solution you advocate is correct. Service boundaries need to be well defined and well separated.

ahallock2• 10 years ago

so that if changes are made to the library and 2 services using them are out of sync

Wouldn't versioning help with this?

iand• 10 years ago

I think the article misses some important context about scale and economic efficiency.

When you have 10,000 engineers then decoupling everything behind protocols and formats makes a huge amount of sense because the coordination costs are astronomical.

When you have 10 engineers then you need to standardise tooling and libraries because the costs associated with diversity in the codebase swamp the coordination costs.

neuroid• 10 years ago

I had a quick look at the transcript of Ben's talk, and it seems that one of his main issues with microservices which share the same code base (and therefore dependencies) is the overhead of upgrading their dependencies. I do understand that in some cases this might be problematic. However, if the microservices are written using the same language (or runtime), there is a huge benefit of reusing some abstractions. These might include things like validation schemas, serialisation formats, etc. These are the things that most likely would need to be created anyway for every microservice written in a new language. As usual, there has to be some balance in the amount of coupling between microservices. But there is nothing inherently wrong with reusing code between them. I think the most important factor in a microservice ecosystem is having well defined protocols and APIs.

Microservices Ending Up as a Distributed Monolith

Related Stories