Maybe we’ve not improved things as much as we think we have with Devops?

by michaelbiven • 11 years ago

47 40 11 years ago

PaulHoule• 11 years ago

One problem is that existing devops tools are not that good.

For instance it seems insane to me that packer and vagrant exist as entirely separate products. As a java programmer I find that vagrant files are way too verbose. I am sure I could write some Ruby functions to take care of that and also to fix zillions of other things that are awful about Vagrant, but then everybody writes their own functions.

toomuchtodo• 11 years ago

Try Docker instead. I can build Docker containers that provide our devs with a full blown dev environment that mimics production, which anyone on the team can then pull down with a simple "docker pull <registry>/<container name>:latest" command.

falcolas4• 11 years ago

Docker has its own issues, and in the end is remarkably similar to Vagrant when it comes to development. Docker's biggest win on the development (not deploy) front is that docker containers are faster to set up and tear down; though with a decent Vagrant setup and a package cache, you can build a brand new Vagrant image in under a minute.

Since I'm on a Mac, and thus going to be running VirtualBox anyways, I prefer to just work with the VMs directly.

xacaxulu• 11 years ago

Vagrant is a very well developed tool and has tons of configurations available in the Vagrantfile. It does take a lot of time to get deep into it however. When coupled with Chef's Test Kitchen, vagrant is a powerful tool. Side note; they actively review and accept pull requests.

rev_bird2• 11 years ago

The Vagrant files we've been using have been whittled down to almost nothing -- we use Chef to provision boxes, so the vagrantfile is pretty much just "install Chef, forward ports, run role xxx." At that point, it just pulls the role (and cookbooks) out of the user's local chef repo -- and getting the Vagrant Berkshelf plugin running even removes the need to deal with downloading cookbooks at all.

jtreminio• 11 years ago

You're not supposed to use Vagrant for the actual provisioning of a box - use the proper tools like Puppet, Chef, Ansible, Salt, etc.

And Packer is for generating the base boxes. Packer abstracts away the virtual machine software like Virtualbox, VMWare, Parallels, etc.

davexunit2• 11 years ago

Unfortunately, developing and deploying web applications is still a total nightmare. To build anything useful you'll need about four distinct package managers to get all of the necessary dependencies and some mish-mash of configuration management and provisioning tools to develop it, and perhaps a different set of configuration management and provisioning tools to deploy it. We can do better than this!

danudey• 11 years ago

Hmm, let's see…

To build anything useful you'll need about four distinct package managers to get all of the necessary dependencies

* Ubuntu packages (dpkg via apt)

* Python packages (via pip)

* Our own software packages (via our own dependency manager)

* Custom-built software packages (e.g. nginx) or non-package software installations (e.g. activemq)

some mish-mash of configuration management and provisioning tools to develop it

We use salt for (some of) our configuration management along with just symlinks into git repositories (where possible, e.g. MySQL and others won't follow symlinks for security)

perhaps a different set of configuration management and provisioning tools to deploy it

We have a custom-built deployment system which we use to deploy git packages (including the configuration packages that the configuration files are symlinked into) as well as our actual software.

So… yeah. You pretty much hit the nail on the head.

davexunit2• 11 years ago

Thanks for the real life example.

rjbwork3• 11 years ago

Y'know what? Up until Jan last year I had never even touched a server to deploy an app, much less set up a CI/CD/DevOps environment.

Now, I'm comfortable with TeamCity, Octopus Deploy, Azure (a ton of hosted services), PowerShell, and a host of other tools.

I've set up bunch of HipChat integrations so we know exactly where in our pipleline a particular build is (code pushed, dev build/deploy, and promotions to various environments), if there was a problem with a unit test run, if there were problems with integration tests, and if there are problems in dev/staging/production with our actual live running code via HipChat integrations with our structured log processing server.

There are good tools out there and they're pretty usable - just maybe not at your price-point of free for your environment. I went from 0 to 1-click whole-environment deployments in just over a year.

moserware• 11 years ago

It's nice to hear of other people with a similar stack on HN (Azure, PowerShell, TeamCity, HipChat, and presumably C#/.NET)! It's working well for us too.

In addition, we're increasingly doing more with ChatOps with mmbot (as an Azure Web Job): https://github.com/mmbot/mmbot

One nice thing about mmbot is that if you use the DLL mode instead of the CSX mode, you get a great IDE debugging experience which makes it even faster to create meaningful ChatOps extensions. I've found it to be a more pleasant development experience that Hubot with CoffeeScript.

rjbwork3• 11 years ago

Hah, we also use mmBot, but while it was our original plan to have it be our "DevOps 1 stop shop" we ended up shying away from that idea in favor of a more hands off approach. We still use our mmBot for a lot of back-office querying type operations, but decided that, at least for the moment, TC/Octopus will be our way forward.

mordocai• 11 years ago

I don't know if you read the same article I did.

The author specifically said that we'd come a long way on tools but not made much headway on the people issues related to Dev/Ops.

Your comment does not appear to be adding anything to the article besides another anecdote that the tools are good (and if that is what you meant it to be, then I misread the tone of your comment).

rjbwork3• 11 years ago

My intention was more, "I went from 0 knowledge at the beginning of last year, to setting up a full devops pipeline on my own at the beginning of this year. What's your excuse?" But I wanted to highlight the tools I'm using as well.

Spooky23• 11 years ago

Of course we haven't. In many cases, DevOps == have developers push code to production platforms.

IMO, there's been a lack of attention to the engineering side of the platforms themselves. In the places that I've seen that yammer on about devops, I see a lot of agile projects where version 1.0 works great, but the ongoing maintenance of the platform (whatever it is) is a complete nightmare.

sgoings• 11 years ago

I think it's worth reflecting that not every improvement initiative (in this case: see DevOps) across the whole world is moving as one large entity.

The ending line: "Otherwise in my opinion it feels more like we’re treading water and not making the advancements we could be making." sounds like an existential fallacy to me - it has been my experience that although the "DevOps vision" has not changed significantly, the actual practices and evangelism of DevOps have been infiltrating and transforming the way organizations think about software engineering.

sanderjd• 11 years ago

Basically my thoughts: 5 years is not a very long time! It isn't treading water, it's taking time to mature and penetrate. It's actually taking root surprisingly quickly, and only looks slow from an early-adopter's perspective. If those talks from 2010 and 2011 were given to substantially the exact same audience as that talk from 2014, then yeah, maybe I see the point that things are just being re-hashed with little progress, but I doubt that's the case.

Giving the same exact talks to an ever-expanding audience over 5 years is already a win.

whistlerbrk• 11 years ago

This recent trend of articles wrt to devops it making me want to post here.

Not trying to be too snarky, but why can't people just use shell scripts for automation?

They are easy to repair, in a language you (should) know, reproducible, and testable. They are native to your target, supremely portable, etc, etc.

cheeseprocedure• 11 years ago

Because it's not worth re-implementing the functionality available in Jenkins, Chef, etc. etc. etc.

stonogo• 11 years ago

This is a misunderstanding of the situation. Jenkins specifically has no business being anywhere near a production deployment; it is a development tool, despite "devops" people tending to use it as a web-interfaced crond.

Automation should be tooled to the production environment by people who are familiar with the production requirements and -- most importantly -- by the people who will be expected to maintain the environment and respond to critical issues.

nickysielicki• 11 years ago

Why do you think most people don't use cfengine?

fweespeech2• 11 years ago

How would you deploy a cluster with highly dynamic portions with shell scripts?

You end up with a process that pulls it from a database or something to fill out the configuration. At which point, you might as well just use a script in python or an existing system.

herge2• 11 years ago

Bash scripts (to pick the most popular shell language) are hard to write, hard to debug (no stack traces), have weird historical artifacts, like not having named arguments to functions, or [[ $foo == true ]] or a whole slew of gotchas that make writing and maintaining them unpleasant.

If your application is written in ruby or python, why should you learn another (worse scripting, but that's just my opinion) language to deploy it?

_wmd• 11 years ago

hard to write, hard to debug (no stack traces), have weird historical artifacts

Only slightly being sarcy, but this perfectly describes my experience of Salt, Ansible and Chef, but instead of a handful of 'ps' commands to figure out what's going on, you have Python/Ruby interpreters on both sides of the SSH connection, giant gumps of weird SSH command lines, a big C++ library and multitudes of TCP ports and home-grown crypto (Salt+ZeroMQ), perfectly descriptionless error messages like "Timed out", "command failed", or even worse, something you never had to deal with in shell: "template syntax error".

I have a personal axe to grind in this department (been thinking about this style of system since the mid 2000s, but never dared start yet another project), so it's entirely possible my expectations are higher than normal here. Still.. processing YAML with Jinja2? Seriously, bash or the most arcane old Makefile syntax was better than this.

chimeracoder2• 11 years ago

Not trying to be too snarky, but why can't people just use shell scripts for automation?

This is why I love Ansible.

Unlike many of the alternatives, Ansible doesn't try to invent a radically different DSL. Instead, ansible uses YAML playbooks that degrade very nicely to shell commands[0]. You can literally take almost any shell script and turn it into an Ansible playbook with a series of Vim regexes - or heck, write a simple sed script to translate it automatically.

Of course, if you use Ansible modules instead of the `command` directive, you get a lot of performance benefits and guarantees of idempotency. But it doesn't try to reinvent the wheel; it just builds on top of it.

It makes it really easy to hack together an Ansible script that is immediately understood by anyone who's used bash/sh and it makes it possible to improve upon it incrementally by replacing shell commands with the relevant Ansible modules.

I've used other automation tools, and the thing that I really disliked about them was that it was so difficult to migrate an existing workflow to a new paradigm. In 2015, systems work is still very imperative, so while declarative syntax is philosophically attractive, I still find it way more practical to use the tools that require smaller mental leaps.

[0] e.g., this thing was hacked together from a shell script, and I never bothered to clean it up, but it's still already more readable than the equivalent shell script: https://github.com/ChimeraCoder/znc-kibana-playbooks/blob/ma...

falcolas4• 11 years ago

My biggest problem, well two problems with Ansible:

The maintainer is unusually hostile to user contributions, especially when compared to, say, Salt. A colleague of mine tried to contribute a (fully functional, tested, and documented) change to the MySQL user module to support passing in hashes for user creation - the pull request was closed with a comment like "this can go in a community module". Nevermind that the MySQL user module is a core module...

It's also horrendously slow when your playbooks grow. The constant copying back and forth over SSH can be downright crippling. Just running a "green" (the server is in the target state and no changes need to be made) playbook against a local vagrant instance can take over a minute for my current monitoring server playbook. As a point of comparison, Salt completes in about 5 seconds.

That said, it's still one of the best "no setup" tools, for the cases where you can't install a minion of one flavor or another. (Yes, theoretically the salt-ssh command line tool exists, but it requires NOPASSWORD sudo on the target server to run)

falcolas4• 11 years ago

For the same reason we don't write all of our backend tools in C: there are better languages for this domain, and better libraries which we can leverage.

For example, when running Puppet or Salt, I don't have to worry about which package manager is installed, or mucking about with checksums to see if a file needs to be replaced, or templating, or variables, or...

If I'm being honest, just the thought of writing dozens of sed and awk commands to mimic the templating offered by Jinja gives me hives.

That said, after fighting for a few years with Puppet, then Ansible, then Salt, I'm thinking that bash and make files are sorely underestimated for some of the simpler provisioning tasks. Bootstrapping a Salt Minion or a salt master is a lot more straightforward in bash than it is in Salt itself.

billybofh• 11 years ago

Having been doing sysadmin duties for more years than I care to remember and having gone through the gradual process of manual configuration -> shell scripts -> cfengine -> puppet, I'd say despite problems in tools like puppet/salt/etc they are way, way more reliable and sysadmin friendly than shell scripts.

For just one example, I've recently begun deploying CentOS 7 boxes whereas previously we mostly used 6 (and some 5). Largely the existing puppet modules kept on working despite major OS changes (hello systemd!). If I'd been maintaining a set of bash (or as used to be, csh) scripts I'd hate to imagine the amount of spaghetti I'd have to have mashed up.

We've also got almost every Linux distribution out there you can think of, versions going back a decade or more, Solaris boxes, BSD's and not long got rid of our last AIX box (and I'm going to ignore the last couple of OpenVMS machines). Handling it with hand-rolled scripts was really not fun compared to puppet.

fweespeech2• 11 years ago

Congratulations on getting rid of your last AIX box. I wish I could :|

rachelbythebay• 11 years ago

Your simple script is someone else's bad day. Imagine a shell script that needs to do a dozen things in sequence. Is it just a list of 12 commands? If so, do they even have error checking, whether by "set -e" or with explicit checks? What if it does fail and it gets restarted? Are any of the steps going to cause problems if they run twice or are they smart enough to not do anything this time? How about if you have "critical sections", such that you always have to do A-B-C, and if any of them fail, you have to undo them in reverse, C-B-A? Do any of the parts need locking so that multiple instances don't run at once? Do you need timeouts on any of your calls in case something gets stuck? What cleans up the locks if you exit uncleanly? Will another instance be able to start up and continue?

You can handle all of these things in a shell script, but that's not the question. The question is: will someone who's "just writing a quick script" think that far ahead?

My experience so far suggests the answer is usually no.

It's not the language. It's the mindset.

xorcist• 11 years ago

It's certainly possible to do with shell scripts and by stringing lots of small utilities such as expect together. But I've been there, and after a few years and a handful of consultants came and went, it's all a big ball of mud. I know how many of my colleagues get spaces in file names, tmp races, and error handling right in their shell scripts, and they're not many.

What Puppet et al brings to the table is a bit of structure and best practice. After a two day introduction, anyone can start making changes to our infrastructure, and I'm pretty confident that the new hire in two years' time will understand it as well. And if I need to restore a previous environment, it's all in version control, and will work since it defines state as opposed to transforming it.

But the big win with a configuration management tool is not even in the tool itself. It's having a central database of _all_ configuration in every environment.

Suddenly you can start doing things like tell an application to connect to its auth service, and it'll know which one it is or even mock it if required. Your monitoring software knows every app in every environment, and can configure itself, so it's literally impossible for someone to forget to set up monitoring for a new service.

It's a huge win. Once you used to it, you never really understand how you managed without. But you really need to go all in on it to see the benefits. Automated systems integration is possible only when you store configuration in one and only one place.

Goladus• 11 years ago

What Puppet et al brings to the table is a bit of structure and best practice.
But the big win with a configuration management tool is not even in the tool itself. It's having a central database of _all_ configuration in every environment.

I agree that is the big win. Though in theory you don't need a configuration management tool to get it. Just sufficient discipline and attention to best-practices.

What the config management tools provide, here, is a framework for combining "site-wide" configuration, "local/discovered" facts, and "human-assigned" roles and classes and exposing it all, consistently, to a specialized scripting language.

The library of modules/cookbooks/etc. also often take care of a lot tedious logic/details that would otherwise have to be in the code (although this part can still get in the way...).

afarrell• 11 years ago

Why do I not do everything in bash?

Because the syntax for if statements and while loops is godawful. I've taught it to myself at least 30 times.

Also, there is no debugger, so I can't step through my code like I can with pub.

Also, the tooling around automated testing is worse.

xj9• 11 years ago

I'm really looking forward to Triton[1] coming out of beta. This may be unpopular/wrong, but I don't think you should have to care about what machine your app is running on in any way.

[1] https://www.joyent.com/developers/triton-faq

Maybe we’ve not improved things as much as we think we have with Devops?

Related Stories