Benefits of a Monorepo

https://pspdfkit.com/blog/2019/benefits-of-a-monorepo/

by ingve • 7 years ago

142 110 7 years ago

sagacity• 7 years ago

Hmm, if you're working on a single product (like the author is), then of course it makes sense to stick them in the same repository. But isn't this just common sense?

I mean, if you have a single product split over many repositories then you will run into the issues mentioned by the author: A change will invariably involve multiple repositories. Again, to me this seems to be just an indication that you are, in fact, working on a single product.

TickleSteve• 7 years ago

This is obvious to me (a monorepo advocate) but most people these days tend to organise their repos based on module/implementation lines rather than on product boundaries.

My motto is: "What you release together, you should version-control together".

Conan_Kudo• 7 years ago

I'm a multirepo advocate, but I suggest splits similarly.

Maybe it's my old-timey way of releasing software, but because I do releases as RPMs, it bothers me less if you do things like have your base library, your backend services, and your frontend in the same repo. If they need to be able to be deployed independently, I just split them up as subpackages.

I guess most language package managers don't really have such a concept, so people are optimizing for that...

ctroein89• 7 years ago

I've learnt that for "What you release together, you should version-control together", the phrase "released together" needs to be defined as "what gets globally updated at once". If you provide anything for customers to implement, you cannot update those globally at once (because you need the customer to update), and should think about using a multi-repo set-up.

My experience is that testing the interactions of historic versions of different components in a mono-repo against each other is difficult. With a multi-repo set-up, one can checkout to whatever historic version one needs to test. That makes it trivially easy to set-up testing that allows one to test historic versions of components against each.

adrianN• 7 years ago

I like having one repo for the project ("What you release together") that tracks subprojects as submodules.

lukasLansky3• 7 years ago

It's a common sense, but it's really hard to argue against super-fine splitting. Having repo per module seems like stronger separations of concerns from a distance.

Of course, what in the end really decides about the strength of your concern separation is the shape of your CI, but alas, repository granuality is just much more visible then CI checks.

royjacobs3• 7 years ago

I would argue that if you have clear separation of concerns you wouldn't need to make changes in multiple repositories all the time.

dalore• 7 years ago

Sounds like you need/could split it up into core/engine functionality, and then implementation of that core in different environments (i.e. a ios and android one).

That makes sense to me, to have 3 repositories. Common code, perhaps even in your own dsl. And then implementations done in the various platforms.

enraged_camel• 7 years ago

What about a single product with multiple components though? Let's say you have an API and a browser client (SPA). Should you keep them in the same repo, or separate them?

So far the only good reason I've had for separating them has been related to deployment concerns (e.g. your API is deployed to AWS and your browser client is on Cloudflare).

royjacobs3• 7 years ago

If you have an API and a SPA that's tightly coupled then, yes, same repo.

If your API is a separate product with a separate release cycle, proper testing and backwards compatibility checks, then different repo.

viraptor• 7 years ago

then of course it makes sense to stick them in the same repository. But isn't this just common sense?

It depends. Are you planning more projects which may share functionality? Are you going to release within a company with other teams that may want to integrate? Are some components managed by dedicated teams who may have their own cycles of work? Are you planning to split off a specific service soon?

Or to quote risky business: what is the problem we're trying to solve?

dkersten2• 7 years ago

... may ...

If something only may be the case, then I'd say its premature effort to plan for something that may or may not happen. Better to do the simplest thing now and then refactor when the need arises.

Ensorceled• 7 years ago

I decided to read this expecting to see an explanation as to why my DRF backend should be in the same repository as my React Native frontend, and instead found a discussion on why my React Native code shouldn't be broken up into iOS and Android.

Not sure this is what people mean by a monorepo ...

shibel2• 7 years ago

I tried deploying my DRF backend[1] + Nuxt.js[2] demo-app on Heroku[3] as one repo but it was a NIGHTMARE. Yes, it sucks to have to open both projects in my IDE in development, but it's nowhere near the difficulty I had with the mono-repo. Plus, I can find things much faster within each project when they are split.

[1]: backend → https://github.com/SHxKM/django-vue-ssr

[2]: frontend → https://github.com/SHxKM/django-nuxt-ssr-front

[3]: live-app → https://django-nuxt-ssr.herokuapp.com/ (on a free dyno, could crash on first load)

nicoburns• 7 years ago

I've settled on a many-repos-in-same-parent-folder pattern, which means that I can open them all in the same editor window if I want to.

I've also forgone proper IDE "project" in favour of opening the directory in my editor on an ad-hoc basis from the command line (e.g. `cd frontend && subl .`). It's not perfect, but it avoids the issue of having too many windows, or having to fumble through IDE project switching dialogs.

PunchTornado• 7 years ago

monorepo means that the whole product is inside a single repo. Like google and fb do it.

amelius5• 7 years ago

What tools are people using to manage multiple repos? A search reveals many, but what is the latest and greatest?

pjmlp7• 7 years ago

NuGET, Maven, whatever is the package manager for the language in question.

The outcome of multiple repos should be modular libraries.

Cthulhu_• 7 years ago

The outcome of multiple repos should be modular libraries.

Yes; also they should all be treated as separate, independent products. Where multi-repos go wrong is when a whole product spans multiple repos, and a feature of the product depends on features being added or altered in multiple projects.

Ideally, the split should be so that nobody has to work across multiple repos to finish a feature.

The github / NPM ecosystem is IMO a good way to look at it; libraries that are all independent of one another who have a (usually pretty good) adherence to semver.

Orphis• 7 years ago

Unless you're a library vendor, the outcome of multiple repos is most likely a single final application. Or a few applications.

amelius5• 7 years ago

Sounds good in theory, but I'm wondering: your edit-test cycle now becomes edit-install-test?

aldanor• 7 years ago

conda is not bad as a generic package manager for this sort of stuff (packages don't have to be Python although that's what it was initially aimed at).

amelius5• 7 years ago

NuGET, Maven, whatever is the package manager for the language in question.

Sometimes a repo contains code written in multiple languages.

bluGill6• 7 years ago

We wrote one in house, it works well and has useful features that nothing else has. Today I'd use conan.io, and add those useful features to that project. Our internal system works well so it is hard to find justification to tear it out, but the downside is no tool/IDE understands it, and so you can't use your favorite IDE unless your favorite IDE is the one obsolete version that we have modified to understand our tool.

zb• 7 years ago

Zuul (https://zuul-ci.org/) is an open source CI/CD system that does speculative execution of CI on changes across multiple repos. So if you have a change in one repo that depends on a change in another, you can test both together before merging either (and it enforces merging in the correct order).

IME it solves the same problem that monorepos are intended to solve, but without any of the downsides.

ivan_gammel3• 7 years ago

What's more, same result can be achieved with practically every modern CI server with minor admin effort.

is0tope2• 7 years ago

I've been an avid user of docker-compose for web app type projects. In this case, I've found putting everything in one repo perfectly manageable. Usually this means the backend, frontend, nginx and any other random microservices are just in separate folders in the base repo along with a compose yaml file. I gather that if you are using more complex CI/CD tools this becomes more complicated...

jacques_chester3• 7 years ago

This makes CI test runs slightly longer, as they first need to do a fresh checkout.

-depth=1 ameliorates this somewhat (though plays poorly with tag-heavy workflows).

bostonpete• 7 years ago

I read this, and I still don't know what "many benefits" they're referring to. I basically read one in the article -- a monorepo avoids the pain associated with making changes to multiple repo at once, which was happening frequently. Does the author make reference to some other benefits that I just completely missed...?

mark_l_watson• 7 years ago

When I was a contractor at Google I was surprised by the mono repo, but really liked it. Scaling issues aside, it makes sense having all dependencies in one repo.

I thought of having a mono repo at home because many of my projects are inter related but instead I spent some time with a library setup. For Common Lisp, I used a pattern in one of Zach’s blogs to have one central repo for all of my personal Common Lisp code that is configured as a root for Quicklisp. Any of my code can be quick loaded no matter where I am working. Really simple but it makes it possible to write small utilities/libraries and reuse them for whatever I am doing.

I sort of do the same for Python, writing a setup.py for each small project or library to install globally, including appropriate executable scripts. This also allows me to combine small things easily. I am not yet set up to do this yet for my Haskell projects but it is on my todo list.

dkersten2• 7 years ago

That's what I do too. I have a git repo with submodules for each library and service that's related to the overall thing. So, while I'm working on it, locally, it almost looks like a monorepo, but the way things are built, run, deployed and stored in remote git is as separate repos and artifacts.

If changes regularly involve multiple submodules, then I know that the boundaries are wrong and refactoring is needed.

petters• 7 years ago

I merged all my personal C++ projects into one monorepo and I really liked it. Especially useful with C++ since setting up the build system is so cumbersome.

baybal2• 7 years ago

It could well read like "The Many Benefits of Using Repo Modules"

To me it feels like a misguided attempts to work around the need of having a package manager

Why not simply build your SRPM or DEB package repo for code? Worked like magic for me on four of my past workplaces.

This also works towards building your culture of stable releases instead of non-stop development and hotfixes.

satya71• 7 years ago

It sounds like this person would have benefited from using Subversion or Perforce. Subversion 1.12 has alpha support for a limited local branch (called Checkpoints).

notacoward7• 7 years ago

These benefits do exist, but drawbacks still exist too. The "constantly updating, constantly recompiling" problem can get pretty bad when there are many developers. Developers either lose time recompiling more than they should need to, or keep working in out-of-date trees and dealing with trickier merges. Multiply by the number of developers in a large org, and that can be a lot of lost productivity.

The thing I don't see discussed often enough, in all of the "monorepos are good/bad" debates, is that they require a certain discipline. When one commit can span multiple components, it's easy to add or ignore bad coupling between those components. That contributes to the rebuild treadmill for everyone, and is bad design in a bunch of other ways as well. IDEs make it worse, because they obscure the boundaries between components, but everyone ends up using an IDE because their predecessors made it very difficult to follow the flow of control across components otherwise. So you end up with one big ball of mud. With separate repos this is discouraged, because violating the rules of good design causes immediate pain. With a monorepo the pain must come in the form of stronger reviews, to discourage bad coupling even though it's easy. Unfortunately, the very same companies that have embraced monorepos have also abandoned separation of concerns as a design principle. :(

ETA: please read bunderbunder's comment (https://news.ycombinator.com/item?id=19796960) as well. It's an insightful take on some of the same issues.

Waterluvian• 7 years ago

I like your comment about commits spanning components. Having many repos forces my team to think about how to stage changes that span many repos.

_ea1k• 7 years ago

TBH, I'm not convinced that a great effort to think about "how to stage changes that span many repos" is always (or maybe even often) a benefit. It seems like the simpler approach of just making it easy to make multimodule changes is a benefit of the monorepo.

jbwyme• 7 years ago

This is a concern with monorepo but you can still enforce the boundaries programmatically. For example you can add some simple, custom lint rules that prevent cross-boundary imports.

Another thing you can do is have your CI only check out specific directories when running tests using something like git’s sparse-checkout.

ceronman2• 7 years ago

There is available tooling to help you with that, not only has to be discipline.

A monorepo oriented build tool such as Bazel will make sure that you don't compile more than needed. This is extremely efficient. In fact, in my company with a few hundred projects we tried it and it was 10 times faster than our previous Maven based workflow. These tools also help you with dependency management so you keep your code organized. The killer feature is that you finally have true Continuous Integration because the build tool will figure out any dependencies of the code that you change, rebuilding it and testing to make sure that nothing is broken.

notacoward7• 7 years ago

A monorepo oriented build tool such as Bazel will make sure that you don't compile more than needed.

Not at all true in my experience. I use Buck, which is very similar. It constantly recompiles stuff it doesn't need to by any standard. I've seriously considered avoiding Mercurial bookmarks in favor of manual patch management to see if I can avoid triggering the cases where it gets stupid. Maybe it's a problem specific to Buck, but my conversations with peers at other companies suggest not (and only a tiny percentage of developers at any of these companies get a choice of which tool to use).

The other issue is that "more than needed" is subject to more than one interpretation. There's "more than needed" as code was actually written, and there's "more than needed" as code should have been written. Tooling can help with the first but not the second. If developers are sloppy about how their header files are structured, then a change to one that's widely used will make it "necessary" to recompile a lot of code - including code that didn't actually use the part that changed. Oddly enough, though, that "necessary" recompilation wouldn't occur if those same header files were split up. (BTW some languages make good design more difficult by requiring private implementation details in public header files, but that's a separate conversation.) Tooling can optimize what it can see, but what it sees might already be a tangle of dependencies added only because a monorepo made it easy.

lmm6• 7 years ago

I don't trust incremental builds (whether via Bazel or any other tool) for an actual deployable build - I've seen them go wrong too many times.

For the local edit-test cycle incremental builds are great, but any decent IDE will do them in the multirepo case just as in the monorepo case.

closeparen2• 7 years ago

Your language and/or build system should provide visibility controls to limit what can be coupled. Package-privacy exists for a reason.

Yes you can post a PR that relaxes someone else’s control but they can block it.

dfabulich• 7 years ago

In my experience, the first time the team tries to separate concerns, they don't necessarily fully understand the problem, and build out the wrong separations.

Multirepo teams live with the wrong modularity boundaries longer than they should, because fixing the design ("violating the rules," when they're bad rules) causes immediate pain. In monorepo it's easier to do the refactor in an atomic commit.

It's all part of the sandboxing cycle. https://xkcd.com/2044/

bluGill6• 7 years ago

Summary for those who can't be bothered to read: we had two repos and copy/pasted code that needed to be in both instead of adding more repos. That copy/paste had pain and we solved it by going to one repo.

They could have solved the pain by adding more repos as well. They never considered this as far as I can tell.

underwater• 7 years ago

They specifically mentioned breaking code outbid to more repos:

We had three choices: ... Extract Core out of PSPDFKit-Android and again deal with fragmented pull requests.

jorams• 7 years ago

That is a very uncharitable summary. The article discusses codebases being developed together or growing closer together over time, causing feature changes in one to need changes in another. It doesn't mention copy/pasting anything. They solved it by moving everything into a single repo.

Adding more repos would once again split up pull requests for features that touch multiple codebases, which is the reason they merged two codebases into one repo in the first place.

bluGill6• 7 years ago

I will admit to being uncharitable.

I am assuming they did copy/paste coding to get features into both. It is possible that they actually did write everything twice, but that is generally such a stupid idea I can't believe it (except for UI - it makes perfect sense to write a the UI for android and iOS completely separate - the logic behind the UI should still be common, but the UI elements/APIs are different enough that common probably isn't worth it)

Adding more repos should not split up pull requests. If it does that is a function of bad architecture.

rkangel2• 7 years ago

Side note: blog posts and news articles that don't display the date are very irritating. There has been a lot of discussions about monorepos, I want to know if this is something new adding to the debate or something old. If it's a news article, I want to know how up to date it is. All I get from this is the slug saying '2019'.

bunderbunder3• 7 years ago

For my part, I've worked with two long-lived monorepos so far. Both were quite small, all told, but, nonetheless, in both cases I saw the monorepo largely serving as an incubator for bit rot.

All those direct interdependencies, combined with the impossibility of just using package versioning to allow a library's consumers to upgrade at their own pace, meant that developers had a strong disincentive to do any cleanup that might result in a breaking change. They'd tend to prefer patches instead. So the long-term trend was for everything to eventually become a tangled mass of patchwork.

Google has entire teams devoted to combatting this problem. I'm not sure most companies can (or want to) afford that.

This is one of those spots where I think it's wise to be cautious about blog-driven development. By analogy, I've seen experienced woodworkers have fun with mocking the DIY furniture projects that come up in places like Pinterest. One common criticism was that, since wood expands and contracts with changing humidity, but only in certain directions, if you put it together in certain ways, you end up with a product that will literally tear itself apart over time. Of course, this isn't apparent from the Instructable article, because it's a slow process, and all those pictures were taken when the piece was still brand spanking new. It's also not something you're likely to see written in the comments section or anything like that - the people I've known who know this sort of thing aren't generally the type to write Instructables articles or maintain their own DIY blogs.

I suspect something like this also happens with any decent sized codebase: Code is a living thing that expands and contracts with changing business requirements. Over time, it will twist itself out of shape. Weak coupling and fragmented codebases are two different (though related) ways of trying to limit the damage from this effect. They both bring their own challenges, though, and aren't perfect. Monorepos don't have the same drawbacks, but that doesn't mean they don't have any. One, I think, is that you're left directly exposed to the full strength of the forces that cause bit rot.

If you're disciplined enough to recognize this and treat it as another incentive to attack bit rot head-on, then you'll probably end up in a better place for it. If you think monorepos are a panacea, though, you're setting yourself up for the kinds of failure that people generally don't like to blog about.

notacoward7• 7 years ago

an incubator for bit rot

Genius phrase. Overall, that might be one of the best comments I've ever read on HN. The DIY furniture analogy is a good one. There's so much focus on what seems to work today, and so little awareness of what affects long-term maintainability. I see this at work too, everyone chasing "impact" for the half even when it screws us over any longer time scale. Understanding how code evolves over time is an essential part of being a competent developer. It's much more important than any in-the-moment coding test.

wcdolphin• 7 years ago

Can you help me understand what bit rot means as you are using it, and why it is different in a mono repo vs an equally large multi-repo system?

bunderbunder3• 7 years ago

Just cherry-picking one rather extreme example: There was a method in a core library that had an argument that was not used. Few people knew this, and it was one that was called at least once by most services.

When I discovered this, I asked the module's maintainer about it. He explained that it used to be used, but requirements changed so that it wasn't needed anymore. He decided to leave it anyway, though, because the alternative would have been an immense effort to track down all the call sites, remove the argument, then trace all the resulting dead code, and remove that as well. It would have eaten up a week, I'm sure. Everyone had bigger fish to fry, and always would have bigger fish to fry.

As I understand it, Google gets around that problem by having a group of people where frying fish like that is their primary responsibility.

solarengineer• 7 years ago

Approaches such as Fan In supported by GoCD are of great help when working with monorepos. See https://docs.gocd.org/current/advanced_usage/fan_in.html

robbrit3• 7 years ago

One advantage that isn't mentioned in the article: code examples. As a monorepo grows, it's more likely that someone else using the repo somewhere has attempted to do the same thing that you want to do, and you can use tools like searchcode to easily find usages and see how they did it. Often this works better than documentation, which for internal projects tends to be non-existent or out-of-date.

One disadvantage: a lot of IDEs don't cope well with monorepos as they try to index the entire thing.

tracker1• 7 years ago

One of the projects I'm on at work just switched several repos into a mono repo. The issue was that there were often related changes to the Database, UI, Configuration and API projects... Those four in particular were almost always released for 2-3 of them for a given feature. Coordinating releases to dev/qa servers even was a bit of a pain. Merging them removed most of this.

There are several smaller/micro services that are still separate, however. They're generally less effected by the coordination of db/api/ui changes.

mikewhy• 7 years ago

The CI story has been my biggest gripe with monorepos. I'm surprised that no separate tool has emerged, like the "custom Jenkins scripts" mentioned in the article.

Maybe I'm missing something, but couldn't you get pretty far with something like:

    # This tells the tool where to look for changes
    packages_folder: ./packages

    # List dependencies. In this instance, if any files in `shared_api` change, frontend and backend need to build.
    dependencies:
      - frontend:
        - shared_api
      - backend:
        - shared_api

    # Definite build order
    build_order:
      - shared_api
      - backend
      - frontend

    # Somehow find these scripts in each package subfolder and try to run them.
    scripts:
      - build
      - test
      - deploy

The tool itself would see which files have changed between two git revisions, figure out what packages need to be built, and run the scripts. But I'm probably missing a million and one things.

kevinsimper• 7 years ago

I really wish there existed more tools, it seems like there is a gap for some tooling.

It would be a simple one that could check the git log and see what files has changed since last time.

Or check the hash of each folder and compare it to previous commit.

If you use Docker, you can leverage that docker will take a hash of the context and if the images already exist on the machine use the cache if nothing has changed for those files. That is also quite effective +1

Benefits of a Monorepo

Related Stories