Habitat — A new approach to automation

https://www.habitat.sh/

by Chico75 • 10 years ago

204 54 10 years ago

otoburb3• 10 years ago

The article link points directly to the Habitat demo. For those of us that don't already know what Habitat is, a better place to start might be the about page: https://www.habitat.sh/about/why-habitat/

Habitat seems like an entirely new Chef product to enable what they're calling Application Automation.

Relevant quote: "[...] automation must travel with the application, rather than be provided by the platform. Everything the application needs, from build dependencies, run-time dependencies, configuration, dynamic topologies, deployment strategies, secrets management, security auditing, and so on belongs with the application, because we're doing it for the application. Runtime and infrastructure layers exist to support the application, but the efficient building, deployment, and management of the application are decoupled from those layers."

Incredibly ambitious.

evgen• 10 years ago

Containerization is disrupting a lot of their current market, so they need to adapt in some fashion. Given the fact that the only hammer they have at hand is ruby systems automation tooling it is not a big surprise they are aiming at these targets.

Unfortunately, IMHO, the scope is a mile wide and an inch deep in terms of what chef brings to the table here. I can walk down that list and either think immediately of another tool that is dedicated to the specific task or the needs are so slight that a few ansible or bash scripts take care of the need. Sounds like they are aiming to occupy some space in the general "tooling & orchestration" arena, but it will be an uphill climb.

24610016422• 10 years ago

This is an excellent point - kind of goes against my earlier comment. Maybe they have to move this way because that's their only choice - if they don't move into automating this, containers will disrupt them (or already has)

jacques_chester• 10 years ago

Comments based on the most trivial skimming of the docs, so I'm probably wrong about everything.

Incredibly ambitious.

It is, but ambition is necessary if you want to build distributed production systems. BOSH[0] does something similar in providing an all-or-nothing approach to configuring distributed systems built out of heterogenous software components spread across multiple machines.

Note here that the key is the heterogenous part -- if you have an undifferentiated flock of software, you can rely on a different abstraction to deploy them (anything from PaaS to a hand-rolled thing, though I don't recommend hand-rolling this stuff). But very few distributed systems are designed to rely on emergent behaviour. In fact that's usually what we're trying to avoid.

Where they appear to differ is that BOSH is quite strict about lifecycle -- you can't "truly" update configurations on-the-fly. You update the relevant YAML and then ask BOSH to update the world for you. It seems like a hassle, but if you couple it with git and Concourse pipelines, you get a complete, externalised history of every system configuration change you've ever made.

We already know that this top-down model of heterogenous distributed systems configuration management works better than trying to extend it from the machine up (as Cfengine and its various successors tried to do). BOSH has been deploying large systems for years and years. It was originally designed to deploy and upgrade Cloud Foundry without downtime, something it does with aplomb.

What I really like about Habitat over BOSH is the way it exposes the available knobs and dials. That kind of interactive discoverability is super helpful sometimes.

I was, however, struck by this claim:

This architecture allows us to avoid the production cliff problem: the closer you get to production, the more difficult it is.

This is literally the problem that PaaSes solve for you. I've seen companies without PaaSes, with home-rolled PaaSes and with Cloud Foundry. In the latter case, the biggest obstacle is usually getting VPN access. And there are other PaaSes apart from CF. Habitat seems to be doing a kinda-sorta PaaS here (note the "Supervisor"), but without most of the feature plumbing that makes PaaSes operationally useful: routing, logging, service injection and the like.

Disclaimer: I work for Pivotal, we do the majority of the engineering on both BOSH and Cloud Foundry.

[0] https://bosh.io/

willejs2• 10 years ago

This looks really interesting. Im waiting for 8:30PDT for the full announcement though!

Im curious about how it works under the hood, it seems to solve 'all the problems' ;)

My skepticism comes from some of chefs past product releases, which have seemed rather, half baked.

Products like chef metal (provisioning), pushy (push jobs) or delivery...

This is in no way negative as I am a long time chef user and fan of chef.

onion2k• 10 years ago

As nice as the 'console' on the page is, the fact the window gets cleared when you click to go to the second step makes it look like it lost what you did on the first page. To me that makes it look like it's broken - the steps should maintain the state and build on each other, not start over each time.

sctb• 10 years ago

We updated the link from the live demo page (https://www.habitat.sh/try/) to the homepage.

24610016422• 10 years ago

I love the Chef ppl, but...

Is it just me or are there a ton of other folks doing this? The idea of introducing yet another layer of stuff is scary, and there's lots of work left to do on Chef. In a lot of ways, they feel just like Nomad (coincidentally also on the front page)... I wish they would spend some time on their core products versus expanding into an overfilled market.

rmoriz• 10 years ago

That's what I thought, too. Chef (software/ecosystem) needs a bit more love IMHO.

However it shows that Chef (Company) is still able to design, build and ship a new product, as an organization. That's a very positive sign. Until recently I thought when comparing Chef with Hashicorp: Hashicorp is so much faster in building and releasing usable stuff but they seem to feel the pain in growing too fast and too broad in terms of products.

So, in the end: It's a great thing, just also don't forget to put more love into Chef (software) and don't dissipate developers/time/budget :)

_greim_2• 10 years ago

I'm not sure what I'm looking at. What is being "automated" here? The headline and home page don't give much context.

otoburb3• 10 years ago

The main points around automation seem to be[1]:

* Flexible versioning and configuration rollout

* Ability to deploy anywhere

* Ease of management

Concretely, they're touting their gossip-based configuration and topology management layer as a way to send configuration updates to service groups, and then letting the service groups apply the changes to all the group members. Contrast this with a more traditional way of propagating changes across a network by enumerating every single host/VM/container. I'd imagine another large component of the application automation would come in the form of tooling allowing the ability to deploy anywhere (bare metal vs. VM vs. container vs. IaaS/PaaS providers), but haven't seen that just yet. Maybe it's being covered in the live announcement.

[1] https://www.habitat.sh/about/why-package-automation-with-app...

_greim_2• 10 years ago

So but am I automating devices in my home/building IoT-style? Or am I automating deployments of software? Or hardware? "gossip-based configuration and topology management" means nothing to me unless I know actually what the overall context is here.

blakeyrat• 10 years ago

The two-word answer seems to be "software builds".

I am always amazed at people's capability to give an extremely complicated answer to an extremely simple question. "What is being automated here?" "Software builds."

willejs2• 10 years ago

Theres a live announcement now https://bignews.chef.io/

Cshelton2• 10 years ago

Found this elsewhere, interesting to note, Habitat looks to be built mostly with Rust

https://github.com/habitat-sh/habitat

holoway5• 10 years ago

This is Adam - Rust has been amazing to work with, and was the prefect choice for this platform. It's great.

Cshelton2• 10 years ago

You should reach out to the Rust team if you haven't already. I'm sure many would love a blog post on using Rust. I was thinking about using protocol buffers for something in Rust, it's cool to see an example of a real app using it, thanks!

bjz_• 10 years ago

Could you elaborate a bit more on why you chose Rust, and why you think it was the right choice for this project?

the_arun• 10 years ago

Is this like Docker?

radhikadesai• 10 years ago

I had the same question

holoway5• 10 years ago

Nope! You can use it with docker, and we think it makes the experience better. We love docker.

SEJeff• 10 years ago

Is there anywhere that actually lays out the tech that powers habitat? ie:

This thing uses kubernetes or this thing uses mesos This other thing uses docker/rkt/containerd/lxd This is the set of things we wrote in rust, etc, etc.

ungzd• 10 years ago

Rather like nix + consul.

vishvananda2• 10 years ago

I've been waiting for this announcement for a few weeks. Since the home page is quite lean on an actual explanation, let me give a rough summary of interesting points as i understand it. I apologize in advance if I have anything wrong.

1. It is a from-scratch source build system written in rust that borrows a huge amount from nix. This means repeatable software builds with isolated dependencies.

2. It attempts to move some configuration management primitives into the build pipeline so that the deployment config has enough flexibility for real-world use.

3. It intends to replace single host process supervision with a distributed model based on gossip.

This project is most definitely ambitious, and I'm not sure the whole goal is realizable but some aspects are particularly interesting.

1. The build process seems quite useful for building containers, especially if it gets some traction. The main drawback of nix is that no one knows how to use it. If this gets the backing of the hordes of chef-aware system administrators, we may have a good solution for isolated repeatable builds, especially if they focus on deterministic reproducible builds[1] as well.

2. Distributed systems are notoriously hard to configure in a general way. It seems like every company has their own magic combination of config management, monitoring, and scripting for keeping systems like zookeeper running. Its pretty clear that we don't have the right primitives for sharing this work. I don't think a container management system has the right primiteves for this either, because I don't think you can solve for all distributed systems in a generic way. You really need custom logic for the particular software you are trying to distribute. I'm not convinced that a gossip-based process-supervisor is the best approach for this, but it could give us a place to start collaborating on these primitives.

[1] https://wiki.debian.org/ReproducibleBuilds

davexunit3• 10 years ago

1. It is a from-scratch source build system written in rust that borrows a huge amount from nix. This means repeatable software builds with isolated dependencies.

Except the builds aren't at all repeatable in a meaningful way, judging by the example code on the home page. The 'do_build' function runs 'npm install'. This means that 1) the build container has network access (i.e. it's nondeterministic by definition) and 2) the declared dependencies do not fully describe the dependency graph. So, I really don't see how much it could possibly borrow from Nix if it threw out the most important part of it.

vishvananda2• 10 years ago

They used npm install in their example code? Well that was a poor choice. Having dealt with npm and node apps in my own build system, I have not found a good solution for deterministically building node apps. Does nix have a solution for this? Thankfully the prepackaged software with reasonable source build systems don't do craziness like this. See the redis package for example:

https://app.habitat.sh/#/pkgs/adam/redis/3.0.7/2016061320525...

GrinningFool2• 10 years ago

I think the point is more that it's completely under your control. The application build would be done however best suits your infrastructure and requirements.

Using npm doesn't demonstrate repeatability, but you could just as easily build a given application from fixed, verified source in this step.

softawre• 10 years ago

You always have the option of running a local npm cache/registry. We do for npm and artifactory.

Not exactly the same as vendoring dependencies, but it's a decent compromise in some situations.

mbrock• 10 years ago

Thanks for trying to explain. So many times when I visit these websites advertising developer-oriented tools I just click around trying to find an actual explanation of what the thing does and how it works, only to close the tab a couple of minutes later with no clear idea.

canterburry• 10 years ago

Hmm, I very much like the concept but when I see the need for supervisors and a habitat service, it makes me wonder if this is really what is says it is.

If I need to pre-install all these services on my hosts before any of this will work, then how is that really different from uniformly deploying all my applications on an application server such as WebSphere? An EAR too contains all my configuration data, my bindings, my security settings etc. So...if you uniformly run WebSphere everywhere in your enterprise, it sort of accomplishers the same thing.

HOWEVER, if the habitat configuration is capable of talking to my F5 load balancer to create a resource pool, self register my app with my corporate DNS, create a DB schema and populate it with my tables, talk to my networking equipment and open all relevant firewall ports and do so when the application is deployed to a new environment, as well as undo all these actions if it's ever moved elsewhere, then you have piqued my interest...

GrinningFool2• 10 years ago

There's no pre-installation - the supervisor is bundled with the application deployment itself.

HerpDerpLerp• 10 years ago

Can I also buy a sofa? http://www.habitat.co.uk/

albasha• 10 years ago

Ah, another limiting abstraction layer to learn and to fight.

Bash is plenty good.

davexunit3• 10 years ago

But all of the package recipes are just Bash scripts, which I think is just terrible, btw. Using a real programming language would have been a much better choice.

holoway5• 10 years ago

We struggled with this! I think we found the right balance, but time will tell.

zeveb• 10 years ago

Using a real programming language would have been a much better choice.

The problem is that everyone disagrees on what a tolerable real programming language is. Some refuse to use Python; others Ruby; others Scheme; others Lisp.

At least every can tolerate shell more or less.

hrayr• 10 years ago

Home Automation is what first came to mind when I read the title. With this in mind, reading the blurb on the website sounded like, oh they're talking about building home automation software...

"build automation, IT automation, deployment automation, etc" would have saved me some confusion. Or simply compare yourself to other solutions out there will help me understand what category this new piece of utility and name fall into.

reitanqild• 10 years ago

Yep. Would made perfect sense for the name as well.

johnbellone• 10 years ago

The GitHub project: https://github.com/habitat-sh/habitat

mwcampbell• 10 years ago

I wonder if this project is doing too many things. It includes a package builder and a supervisor. Is there any benefit to integrating the two?

For a project that's smaller in scope but addresses the part about packaging the automation with the application, Joyent's ContainerPilot [1] looks interesting.

[1] https://www.joyent.com/containerpilot

usaphp• 10 years ago

That jumping header is quote annoying, I guess the developer did not calculate a proper padding to accomodate for a sticky menu once it becomes fixed positioned

BHSPitMonkey• 10 years ago

How does this compare with Heroku's buildpack system?

rmcdermo• 10 years ago

I watched the videos, did the tutorial, read the docs, read the tech site stories but I still don't see why I would use Habitat over the current Docker workflow and ecosystem (including third party) for building, shipping and deploying applications. Both are a type of packaging that puts the application, its dependencies and configuration in a bundle that can run exactly the same everywhere. Docker has Dockerfiles and docker-compose.yml files, Habitat has plan files (seems like a shell script, Dockerfiles and compose files are much simpler). Docker has a central repository/registry "docker hub", habitat has depots. "docker run redis" == "hab start core/redis", etc... etc...

The docs/articles say habitat apps can run in containers, but why bother if your app can already be run in a container that achieves the same thing with even greater isolation (cgroups). Why put a self contained app in a self contained container application?

I'm not trying to put down Habitat, I'm trying to understand its role. As I currently see it, why would I choose this over the more mature Docker environment with a massive established ecosystem and user base?

Again I'm just trying to understand what I'm missing, if I'm missing something please point me in the right direction.

Habitat — A new approach to automation

Related Stories