System Initiative has open sourced its collab DevOps tool

https://www.systeminit.com/blog-open-source/

by vmbrasseur • 3 years ago

69 33 3 years ago

I read their website, their Github project page, and still have no idea what it is (but I know how to install it!). I started to read their blog post but it's too long and while they claim this tool is useful for DevOps, I couldn't find any info on how it accomplishes that.

btobolaski• 3 years ago

The website seems to have more information https://www.systeminit.com/ the video is enlighting. I'm unconvinced by their product though, I'm not sure that this is a good approach.

VectorLock2• 3 years ago

I was interested in the concept but after looking at the source code and the architecture I think I agree with the "not sure that this is a good approach" conclusion.

holoway11• 3 years ago

Hey - CEO here. Happy to answer questions if y'all have any.

VectorLock2• 3 years ago

Was this something you had planned to announce prior to the whole Terraform kerfuffle and it is just unfortunate timing, or was this precipitated by that event in any way?

holoway11• 3 years ago

We've been planning to open source it this way for the better part of a year. It just so happened that we were ready to open it up right after the Hashicorp folks made their decision. No intentional timing.

You can see more on our approach here: https://www.systeminit.com/open-source/

mdaniel2• 3 years ago

Apologies if this comes across as "gift horse mouth"-y but https://github.com/systeminit/si#local-development-setup and its https://github.com/systeminit/si/blob/main/docs/DEVELOPMENT_... friend are some wowzers for just tire kicking the thing. Is there any interest or intention to offer a docker-compose.yml for a little less "I want to spend an entire day just to see this run?"

holoway11• 3 years ago

Yep. If you go to https://systeminit.com and click "Sign up", you can use our launcher to run a build directly.

lijok• 3 years ago

So, so many questions.

Let me preface this by saying - if this works well enough to give devs confidence in the tool, this could be revolutionary. It would combine implementation, documentation, visuals and deployment into a single view simple enough for everyday devs to grok.

I couldn't find any documentation - if there is, that answers these questions, please link me to them.

1. How does authentication work? Is it handled by the tool?

2. Can it support non-AWS resources, such as Grafana?

3. How does version control work? Can you review changes, roll back, etc?

4. In the demo on the main site, the security group was not attached to any VPC, what's going on there?

5. Does this remain maintainable when dealing with very large configurations, for example, a multi-account AWS setup with tens of thousands of resources. If so, what mechanisms are available in this tool to facilitate this maintainability?

Thank you

holoway11• 3 years ago

It's very early - it doesn't do much yet (but what it does do is compelling!). There isn't a ton of documentation, but we have put a lot of real user research into things.

1. Authentication works through Auth0. We're building towards multiple deployment models, where your account works across all of them. That said, it's all open source, so if Auth0 doesn't work for someone, we're happy to make it pluggable.

2. You can model anything you like. Under the hood it's a hypergraph of Typescript functions - so you would model Grafana, and then make calls to its API when actions are needed.

3. It's built in to the model via change-sets. As you do change the model, we show you whats changed, and what actions we would take. The design is heading toward letting you have comprehensive reviews based on which portions of the entire model are impacted. Roll-backs aren't really a thing in infrastructure land, but you can see old versions of the model and decide you want that to be the current one.

4. We model the upstream 1:1 - so that configuration uses the default VPC in the AWS account (which has likely been deleted if you use Terraform, for example.)

5. It's too early to have very large configurations yet. But there are techniques we can borrow from other domains - nesting, for example, or layers. We're working on the fundamentals first, and then we will deal with scale.

Great questions!

Dowwie• 3 years ago

What were the decisions supporting the project's use of Buck2? Considering your team is specializing in the domain of infra automation, compile times of all of SI's Rust must be a hot topic among the team.

holoway11• 3 years ago

We've got a fairly large monorepo, with code primarily in Rust, Typescript and PgSQL. We wanted something that would scale as we grew into things like remote execution and build farms, but that would let us be pragmatic in the meantime. Buck2 fit the bill at the time we needed to solve the problem. We also needed it to understand cross language dependencies.

One of the great things about it is that our CI system uses BXL to automatically generate pipelines on the fly from impacted code, taking into account dependencies. So things generally move as quickly as possible through the system.

It's been real work to adopt, but the upside for us has been worth it.

adamgordonbell• 3 years ago

Best Skid Row Album?

holoway11• 3 years ago

Slave to the Grind, without question.

c7DJTLrn• 3 years ago

It looks cool, but I would be hesitant to leave IaC behind. DevOps/sysadmin kind of people are infamous for (generally) not liking GUIs for a reason. We like being able to grep through a repo, revert to older versions of the code, and deploy new infra all through the CLI.

It might be adopted by people who already do ClickOps or aren't comfortable with the CLI, but I'm not sure about the rest of us. Looking forward to seeing how it progresses.

hiepph• 3 years ago

I heard in a Podcast that they are inspired by game development workflow (e.g. Unity) where multiple teams can work together in a unified way. This reduces "niche" power like CLI commands that only some of sysadmins can wield.

Personally, as an infrastructure engineer, I still prefer CLI and everything as (text) code. But I'm open to change if this "second wave" could take off.

maccard• 3 years ago

As someone who works on games professionally, the first thing people do is turn on the text support for unity assets. Unreal still uses binary assets and it's a giant pain in the ass.

The problem isn't that everything should be text, it's that there are no tools out there that work with other content types. Semantic diff tools would be incredible honestly, and text is a poor representation of code. It's just the best we have right now

jahsome• 3 years ago

I tend to romanticize CLI as a universal and think of GUIs as niche, so it struck me odd to see CLIs referred to that way. That's not a character judgement. I'm actually curious what you mean by niche. Would you mind elaborating?

holoway11• 3 years ago

Skepticism here is warranted - the history of things that look like this in our space isn't fantastic. We've put a lot of engineering into trying to create something that is a power tool, and flexible enough to solve hard real world problems. It's early, and we've got lots of work to do, but that's the goal.

nanliu• 3 years ago

Infrastructure components should still be written in code but composing multiple components (network, infra, database) is often a Byzantine maze of different tooling/language choices (terraform/pulumi/crossplane) and multiple pull requests.

If you shift to an application developers perspective, and zoom out a level most medium/large org still require interaction with multiple teams to spin up an application. Testing and deploying this stack end to end is expensive and time consuming.

I’m not sure what the optimal solution looks like, but certainly open to tools that force us to think more about building interfaces around infra components and wire it together without navigating PRs in a dozen different repos. PR should be the process to create a new version of a component, but not an API for users requesting an instance of that component.

datahead• 3 years ago

Hi holoway, I've been following your project for a couple of months now. How do you compare yourselves to upbound/crossplane?

IME crossplane has been a "much better terraform" and also borrows from some of tf's open source provider code. Seems to be one of the best IaC pattern for k8s centric shops.

Pros: - adoption of k8s core engine, state mgmt

- model everything as a CRD, consistent definition pattern both infra and app

- open source

- great UI when layered with argoCD

- declarative

Cons:

- steep abstraction learning curve (for me anyway)

- docs lacked key context for newbs (also getting way better, big efforts here)

edit: formatting

holoway11• 3 years ago

I think you hit the nail on the head with crossplane being a 'much better terraform' by design. Our goal isn't so much a better IaC / Declarative infrastructure tool - it's a better overall workflow for doing collaborative DevOps work. We think that by having an active model of your component, and tracking the resources along side, we can fix the feedback loops in a way that things like crossplane, terraform, or pulumi really can't.

Of course today it's early - so you have to look at what we're building as a foundation for the future. But it's a solid foundation to build on!

predictabl3• 3 years ago

Wait, I thought k8s and open shift had this entire service catalog concept that was basically sort of a CRD->IaaS bridge.

Did that die? Is that not even what crossplane is? Once again, I'm 6 clicks deep and havent seen a single actionable example. I wish I could get inside some people's heads and wonder wtf theyre thinking when marketing this stuff.

Edit. Wow OSB and Service Catalog are dead. Is there a obituary?

mdaniel2• 3 years ago

Once again, I'm 6 clicks deep and havent seen a single actionable example

Based on the terse reply to my question, it seems they really want you to create an account and go through their onboarding tutorial after agreeing to an absolute telephone book sized ToS. Hell, it's possible I'm violating one of the paragraphs by even talking about the process. I appreciate that's likely not what you had in mind, but it seems to be what they have in mind

throwawaaarrgh• 3 years ago

Used identical tools before. It's just another vendor lock in monoculture. Doesn't matter that it's open source. The design is such that it's all or nothing. Same as Terraform.

holoway11• 3 years ago

One side effect of our design is that, once we bring more of the bi-directional capabilities back into the product - we can track the real-world resources as they change, and update the model on your behalf. So you can keep using (or start using) whatever tool you like to make the changes, and SI will do its best to help you rationalize that. We've had this working in earlier versions, and will again in the not too distant future.

aae422• 3 years ago

None of the code that goes into the bi-directional capabilities will be open sourced though, will it?

mathfailure• 3 years ago

The interface looks sexy. The binding to paid services instead of binding to APIs of open source systems makes me just sigh and go on with my life without System Initiative.

millerhooks• 3 years ago

I got through the first half of the tutorial. I have a busy week but I was excited to see this. It’s right in line with tool I’ve been trying to cobble with distracted effort for nearly a decade. I am so excited to continue to use this tool. Y’all nailed it.

System Initiative has open sourced its collab DevOps tool

Related Stories