Using Puppet to Automatically Configure New EC2 Instances

http://www.codelord.net/2010/12/19/using-puppet-to-automatically-configure-new-ec2-instances/

by abyx • 16 years ago

51 17 16 years ago

sammcd• 16 years ago

If anyone here could discuss the pros and cons of this compared to chef for server configuration i would be very interested.

I often spend way more time than i expect to when configuring a new server, and have recently been very interested in ways to speed that up.

spidaman• 16 years ago

Initially, it won't speed you up. Developing the right provisioning formula requires some investment of time but I think it's time you get back many times over because you don't need to follow a bunch of manual steps and making changes to the infrastructure is easier when it's all code. I prefer chef simply because it's easy to develop customizations; the DSL is just ruby. I have a pretty non-standard application stack but automating the provisioning is pretty straight forward. I can bring up the whole thing from bare machine images in minutes. Check out http://wiki.opscode.com/display/chef/Launch+Cloud+Instances+...

I don't know if there's a puppet equivalent to chef-solo but using it with vagrant and VirtualBox is my favorite way to develop chef recipes. I did short presentation about this a few months ago, see https://docs.google.com/present/edit?id=0AVIsoyl05MRmZGNwY2N...

BTW, the OP talks about needing a separate security group for the puppets but last I checked, the default security group on EC2 makes the machines you own visible to each other and nobody else's that you haven't explicitly permitted. So I think all of that was unnecessary.

lox2• 16 years ago

A Puppet provisioner was recently committed to Vagrant.

sam2• 16 years ago

The equivalent to chef-solo in puppet is "puppet apply". It's very handy for developing puppet recipes on an offline virtual machine.

lox2• 16 years ago

The huge advantage to Chef is that it's just ruby, where as Puppet has it's own syntax for recipes.

Another light-weight alternative for configuring new servers is Babushka: http://babushka.me/

WALoeIII2• 16 years ago

Puppet (as of 2.6) has a pure Ruby DSL. Puppet and Chef are rapidly converging on the same feature set, at this point its like arguing .rpm vs .deb. There are certainly differences and if you spend a ton of time with both tools you will have a favorite, but fundamentally they both serve the same purpose and will enable you to get the job done.

geekinthecorner• 16 years ago

It's amusing how every article about Puppet gets trolled by the Chef group. Is there ever a useful technology that doesn't inspire holy wars?

dkubb• 16 years ago

No. I don't believe there ever can be. For a specific technology choice to become useful -- and more importantly to remain useful -- I think it needs strong competition.

fredoliveira• 16 years ago

I for one, appreciate a pretty succinct article on setting up Puppet. Having used Chef briefly in the past (and now getting ready to use Chef - or puppet - across a brand new infrastructure setup), I came across the lack of (or just bad) documentation for Chef many, many times.

sam2• 16 years ago

I've not used chef, but I'm in the process of setting up a new infrastructure using puppet and I've found the documentation to be pretty good.

swalberg• 16 years ago

I like his way of doing the auto signing. I wrote about the topic at http://www.ibm.com/developerworks/linux/library/l-migrate2cl... and just had the new instance run the manifest locally.

The other option is to have the instance sign its own key, much the same way I have the instance set itself up in Cacti.

kljensen• 16 years ago

This is super useful thanks. I'm mystified why puppet lacks substantive ec2 documentation on their website. I would have thought this was a major use case.

WALoeIII2• 16 years ago

I wrote a pretty detailed piece on doing exactly this last summer, hopefully soon iClassify will be replaced by Puppet Dashboard, but for now I run both.

http://onehub.com/blog/posts/coordinating-the-onehub-cloud/

irons• 16 years ago

I've never used puppet before, but in an EC2 context, I'm puzzled about why one would want to use on-the-fly configuration instead of fully setting up AMIs to serve specific roles.

At first blush, future-proofing against lower-level AMI changes and maintaining instance role flexibility don't seem like a good tradeoff against increased startup time in most cases.

abyx• 16 years ago

We're just starting off with Puppet and EC2, but in general my current feeling is that I'd rather postpone maintaining images as long as I can. This is because a) I don't trust myself to keep these up-to-date, meaning I'll need configuration once the instance is up b) The cost of making a change is tiny with puppet. If you've got images you have to save them again after making the changes. That extra step currently has no value for me, when setting up a machine takes 2 minutes (which is fast enough for us)

cowmixtoo• 16 years ago

I'm going to sound like a broken record but I don't understand why bcfg2 isn't more popular.

If anything, I'm going to 'port' this post to an bcfg2 example.

Andys• 16 years ago

I've recently done a project with puppet that turned out horribly. I'll share what I discovered:

* Puppet's error messages are next to useless. Sometimes it can only tell you which file had an error, and not which line number. Sometimes you'll be left wondering where in the many files that make up a complex project there was an error.

* The only way to actually test the correctness of your puppet scripts is to deploy on real servers. Your staging environment has to be identical to production if you don't want any nasty surprises.

* Despite being written in ruby, the design doesn't follow ruby's philosophies or best practices. The config language is not dynamic or ruby-ish. Compared to ruby, you get code blowout when you can't use DRY and have to unroll loops to fit inside the bounds of what the config language can do.

* Simple things are hard. You have to write your own modules or use 3rd party modules for what I consider simple things, such as editing an existing system config file to add two lines in the middle if they don't already exist.

* There's a steep learning curve because a large project is like spaghetti code where you aren't quite sure which file to find what you're looking for. It is very easy to create dependency loops, and hard to find them, and even harder to resolve them. I ended up generating the dependency graph to a PNG file and eyeballing it, several times per day.

I got the impression that puppet is ideal for getting a "PHP+MYSQL" server up and running that requires almost no deviation from the operating system defaults. If your setup is a more complex one I'd stay well away.

Using Puppet to Automatically Configure New EC2 Instances

Related Stories