Josh: Just One Single History

https://josh-project.github.io/josh/

by rapnie • 2 years ago

91 35 2 years ago

josh-project.github.io

ixxie• 2 years ago

How does this improve monorepo workflow?

rapnie2• 2 years ago

See the use cases: https://josh-project.github.io/josh/usecases.html

rapnie2• 2 years ago

Repo link: https://github.com/josh-project/josh

jotaen• 2 years ago

Josh combines the advantages of monorepos with those of multirepos by leveraging a blazingly-fast, incremental, and reversible implementation of git history filtering.

It might be just me, but I have some serious trouble to understand what this actually is, and how I would use it specifically.

From the homepage, I understand that this is some sort of utility around working with git histories in mono repos. But is Josh a client-side CLI tool? Is it a convention for structuring mono repos? A web service? A complete VCS? Or some combination of all that?

p_l3• 2 years ago

It works by creating a proxy git remote which transparently rewrites history in ways that keep both sides in sync.

digdugdirk• 2 years ago

Transparently in the "I don't have to worry about it" way?

Or transparently in the "when something goes wrong I'll have no way to tell what happened" way?

alkonaut2• 2 years ago

This is a great idea. Where do apply this though? The reason I use git at all is because of the support in other systems. It's not the best VCS, it's the best one that I can create a new repo for in Github or Azure Devops, or that I can link with issues in some issue mgmt system.

If I'd need to self host a git repo, it would immediately lose most of its value. That's not to say it's useless longer term though. Other good extensions to git like Git LFS were also once in that situation, but wide adoption quickly meant that it's now a (mostly) integral part of git and every hosted git repository supports it. This would be very useful if it was adopted in the same way by the big repo hosters.

k8svet2• 2 years ago

The site pretty clearly explains that the split repos are usable as normal git repos. And indeed I have a project I follow by cloning a josh managed repo. I didn't even know it was a josh repo until months after starting.

alkonaut2• 2 years ago

Yes that I think is a given if there should be any traction. The big question is: how do I (or how long until I can) create a repo like this at the various git providers. Because to a rounding error, people don't create or host their own git repos. They are hosted on github, devops, etc. Not because it's better or because it's hard to self-host a git repo but because it's convenient and more importantly because any self hosted git usually means giving up important features in integration with CI systems, Issue management and so on. Offering one thing (e.g. better monorepo support) is not going to fly very well if it means giving up N other things. At that point it's mostly a tech demo like early Git-LFS was until the full support was on Github.

k8svet2• 2 years ago

Uh, what's going on with this thread? I'm pretty sure I didn't write this comment an hour ago (how it displays) and it didn't appear at the top of my comment history? Instead, appearing as "a day ago" in my history further down.

aseipp• 2 years ago

It doesn't manage a Git repository. It's a stateless proxy that sits in front of your existing Git repository. You just run it in front of your existing infrastructure. That's about it.

wbl• 2 years ago

I was unaware that people who used perforce and git would go decide that git needed to work like perforce.

skybrian• 2 years ago

To sum up, it looks like this is a server that provides smaller views of a large monorepo. A simple case is checking out a subdirectory of the repo and just getting the files and history for that subdirectory.

They also advocate using these views to do builds. That way, your build can't depend on any files that are outside its view. It's a kind of sandboxing for builds. [1]

In particular when using C family languages, hidden dependencies on header files are easy to miss. For this reason limiting the visibility of files to the compiler by sandboxing is pretty much a requirement for reproducible builds.
With Josh, each deliverable gets its own virtual git repository with dependencies declared in the workspace.josh file. This means answering the above question becomes as simple as comparing commit ids. Furthermore due to the tree filtering each build is guaranteed to be perfectly sandboxed and only sees those parts of the monorepo that have actually been mapped.
This also means the deliverables to be re-built can be determined without cloning any repos like typically necessary with normal build tools.

Bazel users might find this familiar. It does a similar thing by running build tools in a temporary directory that contains symlinks to your actual source code.

But it's not something you actually need if you're not using a monorepo, since your package manager won't pull in other packages that you don't depend on.

[1] https://josh-project.github.io/josh/usecases.html

scubbo2• 2 years ago

It's _amazing_ how much work monorepo fans put into "making a monorepo operate like a polyrepo", when they could just...

danpalmer• 2 years ago

I sort of get it. I don't think anyone coming from a multi-repo world really understands the full implications of a monorepo until they've worked in a large scale one (e.g. those at Google, Meta, or other places), so I think they try to keep certain kinds of practice that would ideally change in a monorepo world.

And I also don't think Git scales to that (Microsoft just about made it work with a custom filesystem, Google and Meta both use custom source control, Meta's is based on Mercurial), and therefore it's often necessary to split the monorepo into many repos and there exist a lot of tools for treating a set of Git repos as one Monorepo.

scubbo2• 2 years ago

I don't think anyone coming from a multi-repo world really understands the full implications of a monorepo until they've worked in a large scale one

That's entirely fair. My sole experience is the one black-sheep monorepo at my own relatively-recently joined company, which is nowhere even close to approaching true large scale.

Genuine question, though - what _are_ the advantages, as you see them (you didn't explicitly say as much, but I'm reading between the lines that you _can_ see some)? Every positive claim I've seen (primarily at https://monorepo.tools/, but also elsewhere) feels either flimsy, or outright false:

* "No overhead to create new projects - Use the existing CI setup" - I'm pretty confident that the amount of DX tooling work to make it super-smooth to create a new project is _dwarfed_ by the amount of work to make monorepos...work...

* "Atomic commits across projects // One version of everything" - this is...actively bad? If I make a change to my library, I also have to change every consumer of it (or, worse, synchronize with them to make their changes at the same time before I can merge)? Whereas, in a polyrepo situation, I can publish the new version of my library, and decoupled consumers can update their consumption when they want to

* "Developer mobility - Get a consistent way of building and testing applications" - it's perfectly easy to have a consistent experience across polyrepos, and or to have an inconsistent one in a monorepo. In fairness I will concede that a monorepo makes a consistent experience more _likely_, but that's a weak advantage at best. Monorepos _do_ make it significantly harder to _deliberately_ use different languages in different services, though, which is a perfectly cromulent thing to permit.

crotchfire3• 2 years ago

Do not try and bend the spoon, that's impossible. Instead, only try to realize the truth...there is no spoon. Then you will see it is not the spoon that bends, it is only yourself

-- what Pijul users say when they overhear git users arguing with each other about monorepos.

https://pijul.org/

Git insists that you impose an artificial ordering on commits which affect disjoint sets of files. And then people come up with heroic efforts like josh to try to bandage over that mistake. Or you could just not make the mistake in the first place...

bognition• 2 years ago

What alternative are you suggesting?

exDM693• 2 years ago

Pijul is a version control system that is patch based (like Darcs) instead of snapshot based (like Git and Mercurial).

Together with some mathematical models on the properties of patches it can deal with some merge situations that need to be resolved manually in Git.

Pijul's website (and Darcs' before that) has some theory that may be helpful if you are interested in the details.

The distinction between patch based and snapshot based is mostly invisible to the end user, except when it isn't.

Josh: Just One Single History

Related Stories