Marelle: logic programming for devops

http://quietlyamused.org/blog/2013/11/09/marelle-for-devops/

by lars512 • 13 years ago

148 49 13 years ago

CraigJPerry2• 13 years ago

Prolog does seem a reasonable fit for the problem space.

That said, even CFEngine's long winded syntax (born of its commitment to "promise theory") isn't so bad. In my experience I don't feel expressiveness of the available languages is the problem most I need of solving.

The real problems not being tackled by the existing configuration management tools are testing bundles/(play|cook)books/modules, cleanly injecting metadata to facilitate reuse and the bundle debugging workflow.

atsaloli• 13 years ago

What kind of metadata would facilitate reuse?

Have you seen the latest verbose mode in CFEngine (as of 3.5, June 2013)? It's more compact to facilitate debugging.

How else could debugging workflow be improved?

CraigJPerry2• 13 years ago

Say I have a simple bundle for inserting lines in /etc/security/limits.conf, it'd be good to have the bundle agent / edit_line which describes how to make the change, completely separate from the data that describes user1 gets a hard nofile of 64k.

There's lots of ways to achieve that from just doing selective usebundles from various bundle files containing only vars to more flexible approaches; In our deployment we've provided this as part of our bundle framework (we inject the parameter data via modules protocol - so a box in location A applies different data than a box under line of business B), they both reuse the exact same (comprehensivly tested) code.

This means no code release for a data change. No need to train everyone in CFE syntax just to allow an app developer to express that user c on box d is allowed more open file handles.

The point though is that It'd be good to have as a first class language feature instead of each cfe site reinventing a separate framework for the same functionality.

Haven't had any hands on with 3.5 so not familiar with that, but better print debugging isnt where I was going with the idea. I want to be able to step through a bundle, I want to be able to break when global class X is set. I want to then be able to set class X while stepping through. Kinda like making debug logging level interative and with higher signal to noise ratio on screen.

If there was language support for unit testing my bundles I could use that to reason about what I expect to happen. Diego Zambonis book proposed one way to do that with the existing facilities but its just a proof of concept, e.g. it cant guarantee environment cleanup after a show stopping error in a test.

zimbatm• 13 years ago

The meet/met notation instantly made me think of babushka: http://babushka.me/

The base principle seems to be the same except that babushka is a ruby DSL and that the dependency solving is part of the library instead of the language.

qznc• 13 years ago

Babushka seems to be one of the inspirations. The article says "Still, Babushka made me uneasy: all this ceremony and complex templating, just to describe a few facts and simple rules?"

e12e2• 13 years ago

Very interesting. I wonder if it would be viable to build an entire package management system (like apt) in a similar manner -- and if there are any (semi-recent) improvements to prolog-like systems that might improve dependency resolution "for free"?

X43• 13 years ago

These are the package management systems that I use. A new package management systems just does this http://xkcd.com/927/ What we need is a cross-platform cross-applicaton meta package management system which comes as configuration management that evolves to a general purpose package manager without relying on any external package manager.

I'm not fanboy enough and would happily replace:

emerge, equo, apt-get, aptitude, ipkg, dpkg, zypp rpm, yum, tazpkg, pacman, homebrew

tar/gzip, slackpkg (Yeah, you, Slackware, we were friends and you helped me for long enough. You shouldn't do all my work anymore, you're old. Take a rest)

pip, cabal, bundler, gem, composer, pear, cpan, ctan, cran, maven, vundle – haven't you also a love hate relationship with these?

e12e2• 13 years ago

Also like the bootstrap.sh - very nicely structured and easier to verify than many others I've seen. I'd still prefer these scripts to output missing dependencies that calling sudo -- but otherwise very nice.

That is, rather than:

      function install_git() {
          echo 'Trying to install git'
          case $(uname -s) in
            #...
            Linux)
              if has_exec apt-get; then apt_update
                sudo apt-get install -y git
              #...
              else bail "Unknown linux variant"
              fi
              ;;
            *)
              bail "Unknown operating system $(uname -s)"
              ;;
          esac
      }

Simply aggreagate a list of missing packages, and ouput (based on a similar case-statement for detetining host):

    echo Please install missing packages:
    echo sudo apt-get install ${missing-deps}

https://github.com/larsyencken/marelle/blob/master/bootstrap...

lars5123• 13 years ago

Thanks! :)

It's a little aggressive in calling sudo mainly because I'm using it to bootstrap new systems non-interactively. I don't want it to ask any questions at all, just to go.

acjohnson55• 13 years ago

It's been a while since I've studied Prolog, but it's good to see it getting some press. I learned Prolog before ever dabbling in Scheme or ML, and it was just as much of a paradigm shift from imperative style. It's kind of like FP-style pattern matching, but taken to a whole new level, such that the pattern matching itself is pretty much the entire computational engine.

Rather than using functions, which take a number of arguments and return a result, you use predicates on a number arguments. You build your program out of clauses, which are conjunctions of predicates. A process called unification [1] finds the argument values (if any) that satisfy an entire clause. By sharing constants and variables between predicates, the valid results that unification can produce are restricted to what the logic of your program allows.

So your algorithms end up being, in many ways, description of what a valid results look like, and the Prolog engine goes out and finds them. This paradigm clearly maps nicely to domains with lots of rules, like decision engines, parsers, and such. Sounds awesome, right? One caveat is that, in practice, you as a programmer have to have a pretty good understanding of how unification works in order to write performant algorithms.

This is all from several years-old recollections and a little bit of Wikipedia to refresh my memory, so someone please chime in if I'm incorrect someplace or leaving out key concepts.

[1] http://en.wikipedia.org/wiki/Unification_%28computer_science...

mickeyp• 13 years ago

Prolog's a surprisingly good fit for many every-day tasks where people otherwise invent DSLs or poorly-defined state machines and pattern matching algorithms.

So how come we don't use it more often? Well, the reason is actually quite simple:

Prolog's incredibly difficult to reason about once you grow past trivial facts and clauses like in the OP's article -- the examples given by him would've been introduced to you in the first lesson on Prolog in, say, University. The complexities of green/red cuts, backtracking and the procedural nature of Prolog makes it a really difficult language to truly learn -- much less one to specify complex programs in.

I've never programmed in a language where 5-6 lines of errant Prolog code could stump myself, a post grad and a professor before until I did Prolog.

keenerd2• 13 years ago

It is a shame really. Declarative programming is so much simpler to think about and extremely powerful. I've been working on trying to make declarative programming more accessible and while I personally have seen 60x improvements in ability, I have been unable to convince any of my teammates to give it a whirl.

X43• 13 years ago

@keenerd try to build a DSL ontop of prolog and they'll come & play.

chubot• 13 years ago

So I'm not experienced with Prolog, but the thing that scared me off is that it doesn't appear to have any computational complexity guarantees. It seems like will scale very poorly as your data gets bigger. It's just a notation for exhaustive search as far as I know, and you have to add ad hoc cuts outside the declarative model to get reasonable performance.

I have experience with other abstractions which are supposed to be "declarative", but you really have to peek under the hood at the implementation to get them to work in production settings, and I don't like that.

calpaterson6• 13 years ago

I'm skeptical, because there is already a configuration management language that is strongly influenced by Prolog and logic programming - Puppet - and the results are not that great.

The secret of configuration management is that ordering NEEDS to be a first class feature and ordered manifests should be the default, not the option. Puppet suffers hugely from the influence from the fact that ordering is unpredictable and that you have to use arrows or requires to add order. People accidentally forget to add ordering, the resulting code works most of the time, and then one day you start running into an issue because some implicit dependency either didn't happen, or happened in the wrong order. In as much as all logic programming involves not explicitly controlling flow, I think they're inappropriate for configuration management. Sadly, I'm still stuck with using Puppet because of the organisation I work in.

Having worked with Prolog in the past, I would also suggest that an important feature of a configuration management language is that the syntax make it hard to make typos and other minor errors because the feedback loop of deploying your code against a local vagrant machine is already long enough. I vividly remember the annoyance of having missed the full stop to end a block. I'm not desperate to repeat that experience.

gwillem2• 13 years ago

That's the main reason we've switched over from Puppet to Ansible. It has less abstraction layers, but at least its execution order is predictable. The easy learning curve for new colleagues and the compact syntax are also nice benefits.

lobster_johnson3• 13 years ago

That's what I don't understand about Ansible. It's basically -- literally, actually -- Puppet reinvented with a new syntax based on YAML.

Puppet works so well, Ansible seems like a typical "not invented here" tragedy that frequently occurs in open source. Rather than coalesce around something that works and has years of experience and knowledge and bug fixing sunk into it, people create something from scratch.

To be specific, this is Ansible:

    - name: Memcached
      template:
        src=$repository_basedir/memcached/templates/memcached.conf.j2
        dest=/etc/memcached.conf
        owner=root group=root mode=0644
      notify:
      - memcached-restart

The same in Puppet:

    file {
      '/etc/memcached.conf':
        content => template('memcached/memcached.conf'),
        owner => root,
        group => root,
        mode => '0644',
        notify => Service['memcached'];
    }

It's literally the same data expressed here.

Puppet has very little abstraction. But what you do get, which Ansible has not really understood, I think, is a data model. The above results in the graph getting an object File['/etc/memcached.conf'] which can be referred to. I can do this:

    service {
      'memcached':
        require => File['/etc/memcached.conf'];
    }

This in turn creates the graph node Service['memcached']. It takes part in a larger graph that looks like this:

    Stage[main]/Node[myserver]/Service['memcached']

In fact, manifests are all compiled into a large graph like this, with interdependencies between the nodes. The logic-based, Prolog-type language in the OP's link is precisely what Puppet builds underneath.

Likewise, the graph model is extensible. I can invent a new object like so:

    define public_key() {
      file {
        "/home/$name/.ssh/authorized_keys":
          content => "$key";
      }
    }

    public_key {
      'someuser':
        key => 'some public key';
    }

    account {
      'someuser':
        require => PublicKey['someuser'];
    }

Here, the define public_key() actually defines a new type of object that can be instantiated in scripts. When one is instantiated, other objects can be depend on it. This allows me to encapsulate stuff in my scripts; rather than having other declarations depend on the internal details of how a public key is stored (ie., in a home directory), they depend on my definition.

Ansible may look simpler on the surface, but I think it's all surface. When you have learned to use Puppet, it becomes simple. Ansible looks to me like the product of a person who looked at Puppet, said something like "ugh Ruby, ew syntax" and made his own thing, without looking deeper. Simialrly, Ansible appeals to people who look at Puppet with the same reaction, again without looking deeper. The irony is that they are buying into something that is arguably poorer.

That said, I would argue that part of the problem is Puppet's documentation, which surely must drive people to "poor man's Puppet" solutions. (The cynic in me wonders if Puppet's poor documentation is a deliberate ploy on part of Puppet Labs to boost their consulting business.)

NuclearTide2• 13 years ago

I second everything said here.

In my internship last summer, I arbitrarily chose Puppet to automate the configuration of the company's developer environments. Puppet's unpredictable ordering should be a write-and-forget benefit, but in reality it's very difficult to reason about. In the end, my script imposed a top-to-bottom ordering with requires.

Also, developing configuration scripts against Vagrant is a huge pain. While `puppet parser validate` checks for syntax errors, most of the logical, dependency-based errors reveal themselves after Vagrant takes a good minute to fire up.

Anyway, hopefully my Puppet-Vagrant scripts aren't causing any maintenance nightmares...

robbyt2• 13 years ago

If reasoning about explicit order of resources is too much for you, just enable the new resource ordering feature: http://docs.puppetlabs.com/references/latest/configuration.h...

Then assign classes like: class{'ntp':}-> class{'ssh':}-> class{'other_stuff':}

problem solved?

microtonal• 13 years ago

Puppet's unpredictable ordering should be a write-and-forget benefit, but in reality it's very difficult to reason about.

But this is Prolog and ordering in Prolog is deterministic. Clause goals are proven in order. When there are multiple clauses with the same predicate, they are executed in order. If this weren't the case extra-logical facilities such as cuts wouldn't work in Prolog

lobster_johnson3• 13 years ago

Puppet has implicit ordering (for example, a file will depend on its parent directory) as well as explicit (arrow syntax, new in 3.0, I think).

But really, I don't understand the complaint. Writing explicit dependencies literally what Puppet -- or any other config management system -- is about. Puppet cannot possibly know what your intent is. So you have to tell it. Just like you have to tell the OP's system. It's simple enough.

Having used Puppet for a few years, my biggest complaint is that it's not quite modular enough. For example, in one system I have a way of declaring an "app". An app may have some dependencies (Postgres, Memcached), settings, users, etc. I have two choices, I think; either have one large, monolithic define allowing me to instantiate the app from a bunch of config:

    apps::app {
      "someapp":
        environment => 'production',
        uses_unicorn => true,
        db_name => 'someapp_production',
        db_user => 'someapp',
        memcached_servers => [...],
        # etc.

...which means that every setting has to be a variable that's passed around and used by the define. Or I can split it up into pieces:

    apps::app {
      "someapp":
        environment => 'production';
    }
    apps::memcached_config {
      "someapp":
        servers => [...];
    }
    apps::db_config {
      "someapp":
        host => ...,
        db_name => ...,
        db_user => ...;
    }
    apps::unicorn_config {
      "someapp":
        ;
    }

The problem with the latter approach is that eventually, much the configuration needs to be collected into just a few actual config files. Let's imagine that the app uses a single "app.conf". Normally you would build it using a template. But with the second approach, you would have to build it out of fragments that come from multiple defines; probably you would collect them in a conf.d type folder, and then use an "exec" definition to concatenate them into "app.conf".

Puppet is very modular, but it's a bit awkward when it comes to this level of modularity -- or I haven't found a better way.

mitchellh2• 13 years ago

I have to respectfully disagree. I try not to take sides in these issues because the software I make all works well and co-exists with Puppet, Chef, etc. But I feel I need to voice my opinion here.

I _love_ the explicit ordering of Puppet. It is one of my favorite features over other options. I'll explain why I think it is a good thing.

For credentials to add some sort of credibility to this post: I worked for 4 years as an ops person managing not small (but not big) infrastructures for multiple companies ranging from 10 nodes up to 200 nodes. Some organizations used Chef, some used Puppet, and I even used Salt once. I created and maintain Vagrant and Packer, both of which integrate deeply with all major configuration management systems, so I have to know how they work decently well to be able to maintain that.

Let's use Chef for the remainder of this post, but please don't take this post as saying "Chef is bad. Puppet is good." That is false. I'm just using Chef as an example where ordering is implicit (based on the order resources appear in a recipe and run list).

The explicit ordering of Puppet makes it very clear exactly what a resource depends on. When training junior ops, I was frequently asked "so I use requires/notify/etc. to tell Puppet what order to do things?" And I would say NO! That is the _result_, but the purpose of requires is to list what the resource has as a logical _dependency_. For example, a common problem with junior puppet devs is to say "Oh, I'm making resource C which depends on resource B and A, but since B depends on A, I'll just put 'B' in the requires list." That is _wrong_. Logically, C depends on B and A, so put both there! Don't think about ordering, think about requirements/dependencies.

Given this, Puppet is so great. You can look at any resource and say "oh hey, this depends on ONLY these list of things." Sure, the dependencies may have their own dependencies, but the resource in question doesn't care about that. Example: this file requires that this service is installed. That service might require, say, RubyGems, but the file doesn't care! Its not an immediate dependency of the file.

With Chef, on the other hand, I would see "okay, C happens after B and A, but does C _really_ need to happen after A?" I don't know. I can't answer that. It requires much deeper domain knowledge of that infrastructure to answer that question. This leads to more of a nightmare [for me] months down the road, when I don't quite remember if something needs to happen after something else.

With Puppet, as a style rule, I require all ops engineers list _all_ requirements (exhaustive require clauses) whenever they're working. Then, months down the road, you can move around anything you want. You know if you broke something because Puppet will tell you (before it runs, during the 'compilation' phase).

Personally, I prefer this sort of explicitness.

I understand that some don't like this, but I like it. And people can like it. I don't believe either way is STRICTLY better/worse. I felt my opinion needed to be shared.

lars5123• 13 years ago

This sounds like a question of how declarative your deps are, with Chef and Puppet picking different points on the scale.

I can relate that with Babushka now, we commonly rely on ordering of requirements (A <- B, C in that order!) at times when we should really have one of those requirements require the other (A <- B, B <- C). This works fine for the original purpose, but leaves minefields floating around when you want to reuse deps for another project.

lamontcg2• 13 years ago

One nit to pick here is that I'd say that Chef is 'explicit' about ordering (I explicitly say exactly what is going to happen), while Puppet I would call 'implicit' based on solving a dependency graph and a bit of random hash order evaluation.

This is actually one of the founding principles of Chef in that Adam Jacobs got tired of, in large systems, when you get one link in your dependency tree wrong you wind up with code which will pass tests and get deployed to prod then accidentally get broken the next day as the random ordering changes.

Also, Chef does have dependencies between cookbooks and we do depsolve at that level. You can even construct things so that your base role that does most of your user account creation, ntp, dns, mail config, etc -- which is most of the time orthogonal to your app -- does not need to be a dependency of your application cookbooks. If you're deploying a rails app you can make that cookbook depend mostly just on installing ruby and deploying from github and you can test that, tightly, in isolation on a vagrant VM with test-kitchen. You can also setup more complicated integration scenarios where your nginx load balancer cookbooks don't depend on your apps, and your apps don't depend on your nginx config, but you can have integration tests for your nginx loadbalancers that brings up both in a virt and makes sure that you can load balance your app -- again, though, without sucking in every person's ssh keys in the org to set it up.

And even though you're expressing dependencies, you're doing so in a array with an ordering to it, so that the dependencies reliably get solved in the correct direction, so that even if you don't specify the deps right, if you get the ordering correct it all works.

And for credentials, I started using CFEngine in 1999, I did most of the "base role" of Amazon's CM infrastructure from 2001-2006 (growing from doing config pushes to 400 servers in my department to 30,000 servers across the company), then I built a few more small CFEngine deployments of 1,000-3,000 servers before moving on to working at Opscode.

And my background with CFEngine -- which really invented the Prolog-style-Config-Mgmt that this whole article is about some 20+ years ago. And with the frameworks that I built on CFEngine I always imposed ordering so that you could read scriptlets from top to bottom and see what happened and then composed those scriptlets in runs that would execute in order. When we started out not doing that it was hugely confusing to newbies since actions would be taken in a seemingly random order in the file and that is not a natural way for people who walk into config management -- so we went out of our way to break the dependency-solving that CFEngine did and create something that ran almost exactly like Chef does (technically even more extreme, since there was no dependency resolution between "cookbooks" and while there wasn't an explicit run_list the order that files were included simply determined the order of execution).

lars5123• 13 years ago

Even in "non-deterministic" style prolog, the execution order is still fixed, and dependencies are traversed in the order you wrote them. There will be no surprises.

The missing full-stop is no different to a missing semi-colon, comma, or backslash, issues we still have in many modern languages.

calpaterson6• 13 years ago

The missing full-stop is no different to a missing semi-colon

Yeah, it's an annoyance. The difference is that the feedback loop on operations codebases is longer, so they're more annoying.

char_pointer• 13 years ago

For Puppet you can use a validation script and a git commit hook to shorten the feedback loop on syntax errors: http://spoop.net/devops/2013/11/12/validate-puppet.html

Marelle: logic programming for devops

Related Stories