Monorepos and Forced Migrations

https://buttondown.email/j2kun/archive/monorepos-and-forced-migrations/

by zdw • 5 years ago

101 78 5 years ago

I thought google hasn't use perforce for years but something called piper[1]. Aren't migrations good on some aspects such as force engineers to stop using insecure libraries? I would say you could do internal migrations without having to change external apis, maybe google should care about the customer more? or is there some other reason for these external forced migrations?

[1]https://research.google/pubs/pub45424/

Normal_gaussian• 5 years ago

Aren't migrations good on some aspects such as force engineers to stop using insecure libraries?

Sure, but how many times have you deployed an update that you were sure was security? Most libs I work with are 90+% features or ease-of-use releases, the security is hand wavy to do with dependencies, or a very rare "oops".

mattnewton• 5 years ago

This is mitigated somewhat by having integration tests (which shine in migration-like situations like this despite their many other problems). You can write tests that defend you from some of the breakages at Google and guarantee they will be run.

dekhn4• 5 years ago

that's correct, the monorepo stopped being served by perforce many years ago (the efforts to keep it scaling were heroic). Instead, google made something which appears to be API-compatible with perforce (so the tooling continued to work) but is backed by Google technologies with the source hosted in the datacenters (which is kind of a circular situation).

orf• 5 years ago

Isn’t that circular dependency kind of risky? What if a bad deploy knocks offline something required to complete the rollback process for that bad deploy?

I’m sure there are multiple layers of redundancy, but it seems like you could get into a Facebook style situation where you need to go open a (virtual) cage somewhere to flip a switch.

dietr1ch• 5 years ago

I thought google hasn't use perforce for years but something called piper[1].

Yes, but for I guess the author wanted to keep things simple for outsiders.

Piper is supposedly not much different than Perforce. Forgetting implementation details is just a centralized version control system that holds a monorepo and everyone* just test and build from trunk/master/head.

joshuamorton3• 5 years ago

This sort of mixes up a number of issues:

The whole git/mercurial thing is, as I understand it, related to the extensibility of the systems. Mercurial is hackable and extensible in ways that matter, git isn't (or at least wasn't)

The one version policy only really matters for third party dependencies. If you're making a change to a library that is entirely within the monorepo, you can do a three phase migration (add the new functionality, migrate users, deprecate the old functionality). Google is exceedingly efficient at this, and the vast majority of the time these migrations can be done without any effort by local owners.

For third party deps, you can't usually do that, but there's a workaround[2].

I also think that this post and the one here[1] are amusingly both on the front page at the same time. This one decrying the exact sort of mandate based approaches that "make SRE scale".

Also worth mentioning that

The person making the new API is expected to change client code to match, but is not responsible for ensuring the change does not break the client.

Is just explicitly untrue. There are policies that are clear about this (this generally comes down to "we cannot break your tests, but if it isn't tested, we can't know we broke it")

[1] https://news.ycombinator.com/item?id=28825352

[2] https://opensource.google/docs/thirdparty/oneversion/#tempor...

dekhn4• 5 years ago

google employed the folks who wrote the git clients and it was always technically possible to have implemented the monorepo with a git client, but one of the developers (the author of jgit) was opposed to this and prevented it from happening (mainly because his requests for headcount were always denied).

As the ex-maintainer of numpy and scipy third_party at google, I really appreciated the one-version-policy. It was the only way to sanely handle upgrades of thousands of different codes (and google has thousands of different codes that depend on numpy). It certainly helped us managed the complexity associated with mixing various versions of numerical libraries.

joshuamorton3• 5 years ago

google employed the folks who wrote the git clients and it was always technically possible to have implemented the monorepo with a git client

I think we're talking past each other a bit here. Piper's implementation could have been git-compatible originally yes (I think this is what you're saying?). The thing this article complains about is the turndown of the unofficial, unstaffed (20% at best) git-wrapper client around piper. That was a hacky not-fully-git-like but still well loved (I used it for a while!) tool. Doing it right would have been a huge undertaking and, as I understand, would have required building a totally new thing, while mercurial was hackable and extensible and could be used as a base.

dan-robertson• 5 years ago

I think it’s possible for two things to be true:

- monorepos are a good way to organise software, especially if there are many internal libraries or dependencies

- It is bad if your dependencies are always churning and breaking and disappearing underneath you

It sounds like the actual problem might be that there are too many dependencies or that they change more frequently. I think the OP was suggesting that the latter of these is the case.

joshuamorton3• 5 years ago

I would agree with this for the most part, but I'd also note that I think the case the author describes is uniquely bad and that probably necessitates some investigation. IDK, maybe their org doesn't follow the rules, or maybe the product they're responsible for doesn't have enough unit tests, so breakages can slip through, or maybe their (or my) bullshit radar is tuned badly and one of us is out of touch with how most people experience the churn.

(I should also mention that there are recognized issues with the cost of churn at Google, but this isn't really one of them)

jalk• 5 years ago

It sure is nice that google has tooling for updating code in their monorepo. I just wish that they would remember that those tools do not extend outside of google - Protobuf team I'm looking at you. I have spent countless hours in protobuf JDK dependecy-hell. So much for being dilligent updating schemas in a backwards compatible way, when the library itself will just kick you in the balls if there is an update.

dehrmann2• 5 years ago

And you'd think they'd separate major versions by package so they can coexist (a la commons lang), and you can just pin each major to the latest. Nope. Breaking changes across minor versions.

ec1096852• 5 years ago

It's fun to see an insider's view. This presentation from their SRE book paints a very different picture: https://www.youtube.com/watch?v=SqU8TZDnFFA

Especially around who the burden falls on when a backwards compatible change needs to be made.

Agree with the point in the post that their mentality leaks in the way they support products.

ec1096852• 5 years ago

I meant backwards incompatible change.

lamontcg3• 5 years ago

This is really overreaching philosophically when it suggests that external software is "freedom" while monorepos aren't.

What the author seems to be objecting to is that teams that manage upstream deps make breaking changes and those changes become interrupt-driven work.

From the perspective of the upstream devs that makes their lives tractable, because people get off of old versions and they don't have to support it, and security teams do not have to audit old code, and that prioritizes burning down tech debt and not putting it off endlessly.

The work that the author is complaining about is work that probably needs to happen one way or another. The monorepo as a forcing function to burn down tech debt seems to actually be working there. It would probably be a 10x bigger nightmare of years of tech debt if they didn't do that.

There might be tweaks that could be made to the process so that there were windows every year where breaking changes could be made so as to batch up the breaking changes coming down the pipe -- and require a variance if someone needed to push an emergency breaking change due to an external requirement.

bradleyjg• 5 years ago

Maintaining backwards compatible isn’t impossible or intractable. It requires thought upfront and then imposes constraints going forward. Two things many programmers don’t like, but are nonetheless generally good things.

A company shouldn’t make it impossible to make breaking changes but it shouldn’t make it easy either.

Semver is a good tool to build norms around. If your two year old project is either 0.x or 11.x you should probably be a bit embarrassed.

lamontcg3• 5 years ago

Sure but that has nothing to do with monorepo-vs-manyrepo. Those are good things to do in any case.

And I suspect you're going to find that inside of Google these are 10+ year long projects which were not designed well enough for that (because honestly nobody does that). And the breaking changes are most likely largely all well intentioned. Without at time machine its just work that needs to get done in one way or another.

SemVer also has its problems, and is not a panacea, for example:

https://hynek.me/articles/semver-will-not-save-you/

And I'm positive Google is aware of SemVer.

erik_seaberg• 5 years ago

You don't have to maintain the old code forever if you implement the old API using the new API. If you find you can't do that, your dependents can't use it either.

humanrebar2• 5 years ago

It's worth noting that the one version policy is a technical requirement for linking compiled languages together. Yes, there are organizational reasons to require exactly one version of a project. But compiled languages, especially C++, will do Very Bad Things if you have versioning inconsistencies. It's possible to be intentionally ABI compatible in certain ways for certain lengths of time, but scaled over time, number of concurrent versions of a project, and number of downstream uses, you're just waiting to be burnt hard.

That being said, it's possible to do version pinning carefully at the executable level, just so long as each executable has exactly one version of each dependency in its dependency graph. But then you have a combinatorial explosion of dependency sets to evaluate and support. Theoretically possible, but not very practical. ABI stable languages and languages that ship as source code have a much smaller analysis space since compatibility can be assumed in many more situations.

bombcar• 5 years ago

You also have to worry about programmers being api compatible- if someone moves from project A using lib v1.3 they may not realize project N is using lib 1.4 with some differences.

anothernewdude4• 5 years ago

You can pin multiple versions of the same dependency, just not with the shitty ldd you're thinking of.

Or avoid dynamic linking altogether.

Rust lets you have multiple versions of the same crate living in the same executable.

humanrebar2• 5 years ago

Statically linking doesn't necessarily fix this. Nor does ensuring only one version is actually linked.

The problem starts when different dependencies have different opinions about ABI-important details. Like layout of data structures, layout of v-tables, use of concurrency primitives, or even stack utilization.

I don't have as much depth of knowledge in cargo (rustc probably can't avoid these problems). From what I can tell, it prefers resolving dependency sets per executable, which means it would be opting for the combinatorial explosion of compatibility sets problem. That's a valid tradeoff to select, though I am extremely skeptical that an organization with 30k Rust projects would find that dependency juggling is a non-problem.

dehrmann2• 5 years ago

I can’t help but imagine whether Google Cloud’s low market share is related to this. If the internal culture is that clients must act multiple times a month to deal with impending breaking migrations, at least some of that attitude likely falls on external customers.

Not GCP, but Google runs some of its open source projects like this. I've been burnt by breaking Guava and Java Protobuf changes multiple times. Especially with Guava, I strongly recommend that library developers avoid it.

jrockway2• 5 years ago

I think this is an article about writing tests. Some code the author depended on changed the behavior of their app, while they were focused on releasing a critical fix. They deployed the critical fix and noticed the change way too late in the process, thus wasting an entire workday getting it fixed. Well, that's what tests are for -- so that when some person you don't know edits a dependency, they get a message "you can't check this in until you debug this code you are affecting". Maybe you don't need that for your one-person weekend project, but at a company with tens of thousands of developers working in the same repository, it can sure prevent a lot of problems.

anothernewdude4• 5 years ago

Some guy who updates the dependency isn't fixing your tests. They're breaking whatever and not caring.

Jensson4• 5 years ago

If they break your tests at Google then you just roll back their code. After you've done that a few times they will fix your code or learn not to break your tests. This isn't hostile, this is what the coding guidelines tells you to do, if someone broke your tests you roll it back.

adrianmonk• 5 years ago

The article didn't mention the big advantage (in my mind) of forced migrations: the migration actually gets done.

I've worked at places that have done it both ways (forced migrations vs. ways to do migrations later when convenient), and when you can do the migration later, it often ends up being much later, sometimes infinitely later (i.e. never).

Do you want all the pain of migrating now? Or do you want to spread the pain over a longer period of time?

If the breakage is due to incompatibility (the API behaves differently, etc.), then breakage that the migration caused is probably still going to happen if you do the migration later. So you're not reducing the total amount of breakage/pain.

And if you spread it over a longer period of time, you are increasing the maintenance burden on the team that needs to provide compatibility with the old and new ways.

tehjoker• 5 years ago

I don't think they're wrong about the inefficiency of it all being psychically damaging though. I wonder how many dependencies the average project has and how many projects each engineer has to get a rate of breakage that high.

lmm• 5 years ago

Probably the technically best client I worked for had a 6 month rolling deprecation cycle. Every library must release at least every 6 months (or, in the case of an external library that hasn't made a release, the dedicated internal maintainer for that library must assert that the library is still being maintained). If you depend on something that's more than 6 months old, your build fails.

Everyone was forced to do their migrations (exemptions would generally be approved but only once). But they had fair warning and could schedule it to fit in with the rest of their work.

anothernewdude4• 5 years ago

Or the migration doesn't get done and Google kills yet another product.

ramraj073• 5 years ago

It looks like OP is mad that they have to spend 2 DAYS a month running down permissions or overrides due to broken dependencies. I mean they are supposed to do what else exactly? Just heads down keep writing code and nothing else?

The question is if they are being overworked for this or if theyre mad that their cushy well paid perk filled 9-5 job is inconveniencing them twice a month to get out of their seat. Like what?

The only times I’ve seen that work is when absolutely no one actually runs the code. If you want to write code at places where it touches billions of people (for better or worse), then this seems like a better way to do things than what other companies seem to do.

cphoover4• 5 years ago

This doesn't really address the problems OP brought up... like centralized authority, rebuilding for rebuilding's sake, and priorities interrupted because of sporadically imposed deadlines from other teams... OP mentions a timeline for fixing another critical issue was delayed because of an immediate requirement for upgrading a dependency. I can see how that could be frustrating, especially if not communicated with due-notice ahead of sprint/agile planning.

Just because OP is payed well doesn't mean they can't point out frustrations borne out of what they view as inefficient process.

ramraj073• 5 years ago

I’m not convinced that it’s inefficient though. Google is quite likely the company that’s running the most amount of code in the most amount of computers compared to ANY company in the planet, and their system works. I’ll be curious how Microsoft works especially with internal APIs because they’re kinda the one competitor in terms of mammoth code bases and are famous for keeping backwards compatibility for external interfaces at the least.

Minimally they also use monorepos, so there’s that.

brigade• 5 years ago

I would be annoyed if 10% of my development time was consistently wasted because no one cares about building a foundation that doesn’t shift like quicksand.

Jensson4• 5 years ago

Alternative is spending 50% of your time in meetings where you have to motivate what the code you write should look like in order to avoid people spending 10% of their time writing migration code.

And it isn't like those with dependencies has no recourse, if they think one of the dependencies breaks too often they just remove the dependency and live without it. The library maintainers of course wants to avoid that so they have strong incentives to not break API's. And they can't break API's without first notifying or fixing their users either since then their code will just get rolled back, at Google it is typically much more work to break an API than for the downstream dependencies to fix those breakages.

skissane4• 5 years ago

Rather than be able to pin something to a version that works, the culture is that first someone breaks you (or says, “this might break you, please figure out if it will and act accordingly”), then you have to dig around for the change that caused it, find the right person to approve a rollback, and then figure out how to prevent them from breaking you again

Shouldn't the person/team who makes a change take ownership of fixing any breakage it causes, rather than passing the buck to their customers?

It sounds like the real issue isn't anything as technical as monorepos-vs-polyrepos or software versioning policies, but rather a management and culture problem?

cgrealy2• 5 years ago

Shouldn't the person/team who makes a change take ownership of fixing any breakage it causes, rather than passing the buck to their customers?

That doesn’t feel scalable to me. I’d imagine some dependencies at Google run to hundreds of clients.

To take an OSS view, some packages run to hundreds of thousands. Imagine if the maintainers of react had to fix all their clients.

ratorx• 5 years ago

With React, it’s not possible because there’s no systematic way to change client code. With a monorepo, it’s more tractable.

It’s still not scalable in the sense that the downstream can make any change they want. They are restricted to changes they can either manually or automatically apply to client code in a reasonable amount of time. But that’s the trade off for being a heavily depended on library (which a lot of the time is an explicit decision).

account42• 5 years ago

Sounds like a very good motivation to keep breaking changes to an absolute minimum.

cphoover4• 5 years ago

I think if you are going to critique the explicit strategy of "One version" and forced upgrades, an acknowledgement of the many issues with lazy or incremental upgrade strategy implicit to semver is also warranted. One of my biggest gripes about semver, for instance, is how often supposed non-breaking changes in semver do in fact break client/consumer functionality. Semver as far as I can tell is generally managed by people and is a potential point of human induced error. Perhaps this is mitigated through tooling and integration testing in sophisticated organizations, but in my experience it is broken in much of the OSS projects and less savvy organizations I have worked with.

Still, I find this writeup to be a good glimpse into culture at Google that one engineer found to be, at least, more complicated than the rose-colored glasses through which we usually read about Google's engineering practices. This criticism also mirrors what people have said about Google's tendency to kill off their products that people depend on, which causes reticence on the part of users to adopt new products. (related: https://killedbygoogle.com/)

skissane4• 5 years ago

One of my biggest gripes about semver, for instance, is how often supposed non-breaking changes in semver do in fact break client/consumer functionality.

Something I've seen happen – management issues a directive, "Every team must use SemVer it is our standard". Now every team has version numbers 1.x.0. Almost nobody ever bumps the major version above 1 – people just make breaking changes on the minor version. There is no way to enforce the requirement to bump the major version, so no way to tell if people are doing it when they are supposed to (I know there is some tooling to detect backward incompatible API changes, but a lot of places don't use any such tooling, and even if they did, it will always be incomplete, there will always be backward incompatibilities it won't be able to detect.) So long as every team's version numbers look like SemVer, management can be told "Directive implemented, everyone is using SemVer now"

cphoover4• 5 years ago

Yea or projects that are forever on 0.x.x pre release so any change can be breaking according the spec (as it is considered experimental) even after having many adopters. I can think of at least a few OSS projects like this.

markbnj• 5 years ago

> I can’t help but imagine whether Google Cloud’s low market share is related to this. If the internal culture is that clients must act multiple times a month to deal with impending breaking migrations, at least some of that attitude likely falls on external customers.

We've been GCP customers for five years and I can't recall offhand an instance of a forced migration due to a breaking change. They give a lot of lead time and plenty of notices, and then continue to support the old thing for a good period of time. It makes sense to me that they would treat outward facing APIs differently given that they span many organizations by definition.

altmind• 5 years ago

https://news.ycombinator.com/item?id=24165445

caller9• 5 years ago

https://cloud.google.com/blog/topics/inside-google-cloud/new...

flerchin• 5 years ago

I've been pushing our org towards this for my entire career, and will probably continue to do so. These are problems I aspire to have.

marginalia_nu• 5 years ago

The wisdom of monorepos is highly contingent on what sort of operation you're running. I've worked in organizations where some higher up read some tech blog that praised them as the salvation of programmer kind and pushed them through, but it turned out to be a really bad fit for the stuff we were doing.

If you have something you might characterize as one large suite of applications, they do bring benefits.

If you have many independent products, as is common in the B2B-space, especially with several versions in production at different customers that have different customization demands, it can be an absolute nightmare. Team A and D urgently required feature X as some regulation went through in their market and their customers had gambled it wouldn't, now Team B, E and G can't produce anything meaningful for six weeks because they need to build a downstream workarounds to circumvent feature X because their customers aren't allowed to have it because it violates GDPR.

whoisthemachine• 5 years ago

This sounds similar to an experience I had at a much smaller, less mature company, but without a monorepo. We did, however, have a "platform" team which created many packages which all of the other teams depended on which were often forced on the teams. Unfortunately, the reliability of their packages varied greatly, so unless your applications had great integration tests (our team's did), you would take the update and find out in production it caused you issues.

I think another poster [0] hit the nail on the head when they identified the problem as likely poor dependency behavior.

[0] https://news.ycombinator.com/item?id=28846514

underwater• 5 years ago

Google's architecture isn't designed for individual developer productivity. It's a global optimisation. Google appears to value continual evolution of products, infrastructure and code, and the monorepo successfully achieves that.

In my experience the alternative tends to lead to divergence of codebases and products. We've all seen companies that have some great products counterbalanced with a long tail of products that behave completely differently, or are orphaned, or take decades to move to new infrastructure or integrate with other products.

kohlerm• 5 years ago

In big companies you will need some global coordination unless you have lots of disconnected products. Trying to build something integrated and ignoring thar global coordination is needed just does not work

anothernewdude4• 5 years ago

A thought comes to mind. How many of Google's dead and discontinued products are down to the increased burden of maintenance Google push on them.

Jensson4• 5 years ago

Every service is a huge maintenance burden if you take security seriously.

jarofgreen• 5 years ago

So if I worked at Google and in my library/service provided a versioned API what would happen? Would I be told off? Do people do this?

Eg, /v1/dosomething, I need to make a breaking change but I can still provide the old service too so I leave v1 alone and instead add /v2/dosomething

eterevsky3• 5 years ago

The upside of having a monorepo with just one version of everything is that all the code is compatible with each other and easy to understand across projects.

Among other things it prevents diamond-dependency problem and other dependency issues. Google binaries are already regularly hitting the upper size limits for linker. If you needed to link 5 versions of each library it would greatly exacerbate the problem.

Having a monorepo also greatly benefits maintaining security. You only need to fix a security issue in the trunk, and the fix will automatically be applied in every binary that uses your library upon its next rollout.

all to optimize consumer behavior to boost ad revenue.

I've been working for Google for more than 12 years and never ever have I thought about optimizing ad revenue. Not just that, but I also never heard about this consideration in any meeting that I attended over the years.

Did I mention the report was commissioned, in part, by Google? (End of Aside.)

I don't see how this in any way invalidates the conclusion. It's not like this report was commissioned for marketing purposes.

note git is not allowed at Google

This is very misleading. All Google open source projects, including Chromium, AOSP and all of github.com/google use git. What is not supported is a git frontend to the internal VCS. This is mostly due to the fact that support of any additional tool required staffing, so only a limited number of configurations/tools can be supported.

disgruntledphd22• 5 years ago

What is not supported is a git frontend to the internal VCS. This is mostly due to the fact that support of any additional tool required staffing, so only a limited number of configurations/tools can be supported.

It's super weird to me that Google wouldn't staff this. Given how much of the external world uses git, I would have thought that keeping new hires productive would have made this staffing requirement a no-brainer.

But what do I know, I don't work for Google. I'd be super interested if anyone knows why this is the case, though.

eterevsky3• 5 years ago

I'm also slightly disappointed by the decision not to support git, but honestly it doesn't really affect productivity.

You have two interfaces to the version control system: one native, that evolved from Perforce, and the second one based on Mercurial, which is pretty similar to what can be done with git.

At some point there was a decision to be made whether to support Mercurial or git, because they have very similar feature sets. I don't quite know why they made the choice they made.

To clarify, all of this only affects the client side of the VCS. It is not really possible to use either git or Mercurial for repository storage without some major trickery.

gipp• 5 years ago

There was at one point a git interface. It was extremely cumbersome to use compared to the perforce/mercurial ones and was eventually deprecated. I can't speak to how much of that is staffing limitations versus underlying technical reasons for poor compatibility.

At any rate, it's extremely simple to move to Mercurial from the git world, particularly with trunk-based development.

Jensson4• 5 years ago

Likely this person hasn't worked in a large old codebase outside of Google. Migrating dependencies is a large part of the maintenance work in every large old codebase with lots of dependencies.

The last two words are the crux of this issue: monorepos deliberately centralize power.

No it doesn't, it democratizes decisions. In a monorepo when someone breaks an API you have the power to roll that breakage back. In a typical environment you will have to deal with that breakage sooner or later anyway without any say on the matter. So a monorepo makes it much harder and costly for library maintainers to break downstream dependencies, while in a typical environment they can break their API every new commit and create a ton of work for everyone else without a second thought.

"But I don't have to migrate without a monorepo!"

Yes you do, but without a monorepo the requirement to migrate wont come from other engineers, but will come from some guy high up who demand that now everyone must switch to at least version X to reduce security issues and the huge maintenance burden of maintaining Y separate versions at once. So if you think that engineers in non monorepo environments doesn't spend a ton of time migrating dependencies then you are wrong. They migrate just as much, but they happen in bigger chunks and gets way more complicated.

kitd• 5 years ago

So a monorepo makes it much harder and costly for library maintainers to break downstream dependencies, while in a typical environment they can break their API every new commit and create a ton of work for everyone else without a second thought.

I worked on a microservices project where downstream projects put "contract tests" into upstream dependencies. Ie, you can change your dependency if you want, but it has to match the contract/API we agreed with you or your change doesn't go in.

videlov• 5 years ago

At my previous job I was part of the team managing a decently sized monorepo. In my observation the discussions of monorepo vs. multiple repos are often emotionally charged. I explain this to myself with the argument of independence and isolation that is often presented, advocating for multiple repositories.

In those discussions I often bring up that it is important to consider the engineering organization as a whole and think of it as a system in it's own right.

Others have already highlighted the monorepo benefits associated with managing dependencies. When it comes to (forced) migrations - we are all familiar with the accumulation of technical debt at organizations over time. I hypothesise that absence of (forced) migrations plays a role in causing it.

In my perspective the majority of monorepo shortcomings are related to tooling. On the build side I have found Bazel to be a fantastic piece of software. The source control question is another story. This was actually one of the original reasons for me to co-found a devtools company - Sturdy (YC W21)[0] - taking a step back and thinking over the developer experience.

[0] https://getsturdy.com/

Monorepos and Forced Migrations

Related Stories