Testing with Jenkins, Ansible and Docker

http://blog.mist.io/post/82383668190/move-fast-and-dont-break-things-testing-with

by cpsaltis • 12 years ago

161 23 12 years ago

blog.mist.io

trjordan• 12 years ago

Cool stuff! We actually use a similar setup [1], but with the additional hassle of figuring out how to handle hardware connections back to the app. Docker + Jenkins has definitely been a big win.

[1] http://www.appneta.com/blog/automated-testing-with-docker/

SeoxyS• 12 years ago

You should optimize your RUN commands. Every time you RUN in a Dockerfile, it creates a new filesystem layer. There's a hard limit (42, iirc) to the number of layers that Docker can support.

Instead of doing:

    RUN echo bar >> foo
    RUN echo baz >> foo

You could do:

    RUN echo bar >> foo \
        echo baz >> foo

jhspaybar• 12 years ago

This limit is now higher(somewhere in the range of 100), and will eventually go away. Once this limit is gone, best practices will almost surely be one command per run.

jafaku• 12 years ago

Once this limit is gone, best practices will almost surely be one command per run.

Agreed, I thought the same, but then I hit the 127 (I think) limit.

1331• 12 years ago

In my humble opinion, each layer should have a clear purpose: http://www.extellisys.com/articles/docker-layer-overload

shykes• 12 years ago

You make good points in this post. We are working on addressing some of them already. I would love to discuss this directly, you can ping me anytime, shykes on #docker / freenode.

Silhouette3• 12 years ago

At the same time, we need to move fast with development and deliver updates as soon as possible. We want to be able to easily deploy several times per day.

Genuine question, not intended as any sort of troll: what benefits would people with this philosophy say their organisation gains from routinely deploying multiple times per day?

I have nothing against better testing tools or more efficient development processes, of course, and if you have a serious bug then being able to fix it as quickly as possible is obviously beneficial. I just don't understand where this recent emphasis on always trying to move fast has come from, or what kind of management strategy someone might use to take advantage of such agility.

josh-wrale• 12 years ago

I found this article helpful in understanding the 'multiple times per day' motive.

http://martinfowler.com/articles/continuousIntegration.html

Silhouette3• 12 years ago

The motivation for continuous integration I can certainly understand. I'm sure many of us have worked on projects where someone was working on a branch for an extended period to implement some big new feature, and then ran into the mother of all merge conflicts when they finally wanted to release their changes for general use. I don't think CI is appropriate in all cases, but I find it a better strategy than mega-merges for most projects, most of the time.

It's specifically the motivation for continuous deployment that I'm curious about here. Fixing bugs quickly is obviously a plus, and I found the point 'twic mentioned about rapidly instrumenting a production system interesting. These are good arguments for being able to release an update on demand, and indeed for continuous integration as a technique to support that goal.

However, to me neither seems like a strong argument for routinely releasing multiple updates per day. I've always been a little surprised that so many web sites and SaaS applications seem to take pride in making such frequent changes, even though doing so surely incurs some risk of breaking things and typically also causes some degree of instability in the UI. I was just wondering whether anyone had experiences and/or actual data they could share about whether the practice made a measurable improvement to, for example, revenues or reported customer satisfaction -- something concrete that might outweigh the risks -- or whether it's more of a subjective preference for that way of working.

d1_mo• 12 years ago

I think the point is that deployments should be streamlined and we should not make such a big deal out of them as many of us used to.

If the code for a new feature is ready and all the tests pass, being able to deploy with the click of a button allows you to close the ticket right away and move on to the next task in line. Also, there's no need to coordinate with other developers that are about to finish their thing.

Overall, it's good for productivity and peace of mind, as streamlined deploys are less error prone.

joshdance• 12 years ago

It is all about feedback loops. If you finish a feature and no one sees it for a month because your on a monthly roll, you don't know if it really works for them. And if it comes back and needs an improvement, you have to figure out what you were doing again. Two month iterations are hard to learn from.

twic• 12 years ago

benefits

Over what? Are you seeking a comparison with releasing one a day, a week, a month, a year?

One of the responses to your question contrasts multiple-times-daily releases with once-every-two-months releases. There are certainly clear benefits there. Personally, i am less convinced of the net benefits of multiple-times-daily releases over a once-a-day or even once-a-week releases.

I work on a few apps which have fast release cycles, where we often release several times a week, and occasionally release several times a day.

One big benefit is the ability to fix bugs fast. When a customer service rep comes over and tells us about a mistake in the data we are showing our users, or a sysadmin comes up and tells us about some alerts we're causing on his dashboard, we can quite often fix the bug and release the fix the same day. That really helps keep customers and sysadmins happy.

Another benefit is the ability to get metrics into production fast. If we're making plans, sometimes we realise that we really need more information about the behaviour of the system or users to make a good decision. If that's information that can be captured in metrics or log events, we can add the metric release, and then gather information quickly enough not to completely derail our planning process.

The ability to add metrics fast also helps with tracking down bugs, because we can essentially instrument the production system on demand. For example, at one point, we had a suspicion that some service that took two overlapping date ranges as parameters was being called with date ranges that did not overlap. So we added a check for that which logged an event if they didn't, and pushed it to production. The next morning, we knew exactly how much this was happening, and which other application the was making the calls.

The flip side of this is that our frequently-released apps get essentially no meaningful manual QA, and precious little automatic testing. We have huge suites of tests which we run in CI, of course, but we don't have time to put every release through soak or performance tests.

If you wanted to make a general handwavy statement, i'd say that having really fast release cycles helps break down the wall between development and production. Which you might or might not find was a good thing.

skybrian• 12 years ago

One benefit is that more releases means fewer changes in each release, which means when something starts to go wrong, it's easier to figure out what happened.

But of course there's a limit; you don't want releases that are so quick that you don't collect enough data about the effects of each release.

mateuszf• 12 years ago

This book may be good read to you: http://www.thoughtworks.com/continuous-delivery

wiremine• 12 years ago

I've been using Ansible for a few production-related tasks lately, and think it's great. It provides the right level of abstraction, IMHO: you can crack open a playbook, read through it, and know exactly what it is doing. There's also a growing number of playbooks if you google around.

That said, the biggest downside I've seen with Ansible is reusable components. They have something called Galaxy in beta [1], which should help, although it feels a bit rough yet...

[1] https://galaxy.ansible.com/

geerlingguy• 12 years ago

Galaxy is slowly, but surely, becoming a more helpful tool. At this point, I think most people are using ad-hoc playbooks and homegrown roles. I think more people will start using (and improving) Galaxy roles as the tool becomes more mature (most of the glaring bugs are gone in Ansible 1.6, which is coming soon), and more roles are added/shared.

It's nice being able to put together a plethora of servers within seconds, and start deploying them to DO, AWS, or a local box for playing around within a few minutes! See, for example: https://github.com/geerlingguy/ansible-vagrant-examples

crdoconnor• 12 years ago

That said, the biggest downside I've seen with Ansible is reusable components

This was a major, major issue with puppet, too - although somewhat worse, in that the manifests were more unreadable.

One issue that ansible still hasn't really solved is the horrible if/then/elses you have to put in to accommodate RHEL/ubuntu/centos/whatever else differences.

I like that galaxy at least lets you specify which environment the playbook will run on, though.

mpdehaan2• 12 years ago

Check out the group_by module, as this can make those totally go away.

fit2rule• 12 years ago

Was quite surprised to see a lack of playbooks in Galaxy that have anything to do with installing/setting up the Java RE .. anyone know any reason why that may be the case?

regecks• 12 years ago

Take a look at github.com/drone/drone for an upcoming CI platform built around Docker. It's totally open source, but you can use the hosted version at drone.io as well.

Testing with Jenkins, Ansible and Docker

Related Stories