Fabric Python with Cleaner API and Parallel Deployment

http://tav.espians.com/fabric-python-with-cleaner-api-and-parallel-deployment-support.html

by tav • 15 years ago

183 32 15 years ago

tav.espians.com

jedsmith• 15 years ago

I've just started playing with Fabric, and I like your take on its functionality better. If they're resistant to merge you, why not fork into something new?

You definitely have something very useful there, from a casual read.

tav3• 15 years ago

Whilst forking is always a possibility, I'd rather contribute back to the project and help improve Fabric collectively. The community is made up of really nice guys and it'd be great if we could figure out a way of getting the various ideas to work together.

Thanks for checking out the docs and your encouraging words!

_Fil_2• 15 years ago

Jeff mentioned on twitter that he's keen on merging some of the stuff post 1.0.

rch• 15 years ago

Plus one for the task decorator.

Very nice work all around.

tswicegood3• 15 years ago

Just a quick note: these features are definitely experimental. The @task decorator has been in stable forks of Fabric for over a year at this point (this is the 3rd, if not 4th independent implementation of it). Likewise, parallel tasks have also been relatively stable for quite a long time.

jsdalton2• 15 years ago

Good to hear, hopefully the task decorator will make it in soon because I think it's a much superior API.

The little hack on Fabric I use currently takes the opposite approach -- I use an @internal decorator that hides certain functions from the task list but keeps the default Fabric behavior around for anything else. It's just a few lines of codes that mess with a private list Fabric uses to track internal functions:

    from fabric.main import _internals

    def internal(fn):
        _internals.append(fn)
        return fn
    _internals.append(internal)

tswicegood3• 15 years ago

That's an interesting idea. I like the idea of turning things off too. In some cases, that's more useful than explicitly defining a @task. Care to open a ticket on code.fabfile.org with it? Maybe we can work that in as an option in 1.1 (when @task is set to hit).

sophacles3• 15 years ago

While I like a lot of the stuff you have incorporated into your fabric, it seems to me that you are trying to take credit for things that have existed as branches to the main codebase of fabric for quite a while. Just off the top of my head: task decorators, parallel execution, better host management and command-line task listing all exist in other people's patches. If you have redone the code, good for you, but at least mention that the ideas and prototype implementations aren't yours. (I don't think a claim to ignorance of such is really that valid either, since the Redmine for the project has tickets for most of the things you've done, and has links to the branches containing them).

As to getting stuff into fabric: a very valid complaint (and one I have myself, since like a lot of people I run my own hacked version of fabric) is that fabric development moves very very slowly. There are several different feature branches in several people's code-bases that could have been released 6 or more months ago. Perhaps if you start pulling other patches that have a lot of functionality complementary to yours, you could have a product good enough to push development on the mainline too (ala gcc/egcs of old...)

bitprophet_4• 15 years ago

At this point I'm pretty sure I've learned my lesson about release timing (I've also signed on with a job which should give me more time to work on Fabric, since they use it daily). Definitely deserve all the flak I get for being slow, though.

tav3• 15 years ago

Let me get this right. Due to convergent needs, dozens of Fabric users post various ideas to the issue tracker and mailing lists. And over the years, some people hack together personal branches solving some of these issues and, despite the incredible amount of time that's passed, they fail to get their work into the mainline branch.

Eventually, one hacker comes along — who, having been a Fabric user for a long time, has independently experienced similar needs and finding no decent solution, decides to take on the task of implementing the various features himself — making sure that it's clean, backwards compatible and fully documented. And your response is to accuse him of trying to take credit for other people's work?!

I can understand your frustrations with the pace of Fabric development, but accusing another hacker of taking credit for other people's work simply beggars belief. Even more incredible is that so many people decided to upvote you! For the record, I'm not taking credit for anyone else's work. I came up with the various ideas myself and you can look at the code to see that it's not based on anyone else's work. Thank you.

StavrosK2• 15 years ago

As a side note, your blog is beautiful. Did you design it yourself?

sophacles3• 15 years ago

Dude, I'm just saying it is pretty dubious that you show up at the community, poke around the tickets, ask lots of questions, probably notice the roadmap on Redmine where lots of the existing solutions are tagged for inclusion, and after all that interaction claim you singlehandedly fixed everything wrong with the project. I'm sorry if you are somehow upset by my doubt, but you have to admit the whole thing seems just a bit "too good to be true".

As for the difference between independent implementation and code-stealing: I have not and will not accuse you of stealing code. If it came off that way, I apologize.

I will however continue to assume your rooting around other people's work gave you ideas, gave you paths for completion of tasks, and gave you enough working information on reasonable solutions that you can not accurately say you did it by yourself.

_Fil_2• 15 years ago

There is definitely some cool stuff in there. I particularly like the @task decorator, the directory support, the context and of course the parallel tasks.

Nice work tav.

askedrelic2• 15 years ago

Cool ideas. We're just now moving from a 1 machine deployment to a 4 (and probably more) machine deployment at work and I can already see the value in this.

polvi• 15 years ago

Another cool trick: Integrating Cloudkick tags with Fabric. Using the Cloudkick API - you can express things like "run certain_func on everything tagged 'loadbalancer'".

https://support.cloudkick.com/Integrating_Fabric_with_Cloudk...

We dog food this for all of our production deployments. For example, when we ship a new version of cloudkick.com, we just run "fab ship", which looks for all servers tagged "cloudkick.com" and runs the appropriate deployment script on each of them.

Tagging can be done through the CK UI, automatically with the agent, or with things like puppet or chef. This allows you to fully automate deployment and other tasks around groups of servers.

askedrelic2• 15 years ago

I figured out something like that using the Python Linode API last week: http://asktherelic.com/2011/02/17/dynamic-fabric-commands-fo...

bitprophet_4• 15 years ago

Fabric maintainer here -- Tav's been quite active on our IRC channel while developing much of this, and it's great to have more people excited about improving the tool!

As tswicegood says in another comment, some of these changes (@task, parallelism, etc) aren't unique to Tav's work, so when it comes time to merge we'll draw on the various implementations to see what feels best.

The features unique to his fork will be evaluated like any other contribution and merged in if they fit well and are generally useful (which many seem to be).

The plan right now is post-1.0 (1.0 is scheduled for Pycon or earlier) to round up a lot of major contributions and get them into core, so people like tswicegood don't have to work off of their own forks quite as much :)

simonz05• 15 years ago

I cant see how adding such a wast amount of features makes the API cleaner. You made the library more complex than what it already was. Fixing Fabric should in my opinion be done, not in a backwards compatible way, but by forking it and taking away all the crap you don't want, then build a cleaner API. Adding more features to something which already is bad doesn't make it better.

ghc• 15 years ago

This looks nice, but I have to be honest...I'm not going to switch from the version of fabric I use to this. Why? Even though I do have to deploy to more than 30 hosts at once, the thought of the time that would need to be spent verifying that there are no bugs in this would make it impractical to switch. I have to imagine there are others like me who feel the same way. It's a shame that the people this would most benefit are probably the least likely to switch.

bitprophet_4• 15 years ago

I don't believe Tav posted this as an actual fork, but simply highlighting his branches/patches, in the same way that many other Fabric users have code waiting to be merged into core.

When the time comes to merge/incorporate them, we will definitely be wanting peoples' opinions on whether they feel the changes would fit into core Fabric :)

cagenut• 15 years ago

I admit to substantial ignorance here, but can anyone who's using fabric, or for whom these patches to fabric are an important improvement, speak to any kind of pro/con comparison they did with func (https://fedorahosted.org/func/)?

I've been using func for parallel deployment to 15 - 20 servers for about a year now, but I have to admit I didn't really do any research into other options at the time. Its written by the guys who wrote yum and cobbler, which are already core to my setup, so it seemed natural to just stay in their ecosystem. Thing is I never hear about anyone else using it, so I'm curious if anyone did more diligence than I and had specific reasons of choosing one or the other.

mccutchen• 15 years ago

I had not heard of func until this thread. I chose fabric because it was the only Python-based option I was aware of.

traviscline• 15 years ago

Did some filter-branching and rebasing to get tav's work on top of current master: https://github.com/traviscline/fabric/tree/tav

encbladexp• 15 years ago

The @task decorator is a real good idea. Many good changes in 1.0.

The only thing i miss is a virtualenv context manager (with virtualenv(blupp):...).

carlhu• 15 years ago

Beautiful work. I've always felt that Fabric strikes a effective balance between simplicity and flexibility and love how you're extending it without breaking its spirit.

Fabric Python with Cleaner API and Parallel Deployment

Related Stories