Gophercloud: A Multi-cloud Software Development Kit for Go

http://developer.rackspace.com/blog/gophercloud.html

by jnoller • 13 years ago

61 27 13 years ago

developer.rackspace.com

howeyc• 13 years ago

Interesting. Browsing the code it looks like they are using closures[1] in a way I haven't seen other Go code use them. Similar to the "using" keyword in .NET or the "with-" macros (with-open-file) in Common Lisp.

[1] https://github.com/rackspace/gophercloud/blob/master/accepta...

zemo2• 13 years ago

You don't see it because it's not really idiomatic Go code. Another way to do this would be to define a type that is a function type, and then to put a method on that function type. Yes, functions can have methods in Go. This seems a bit odd at first, and it may not seem particularly important, but because a function can have a method, and an interface is just a method set, you can use a function to satisfy an interface. A more idiomatic way to accomplish the same thing would be to define the argument function as it's own type:

    type AccessFn func(gophercloud.AccessProvider)

so now AccessFn describes any function that takes an AccessProvider as its sole parameter, and returns nothing. On this function type, define WithIdentity as a method (edited for brevity):

    func (fn AccessFn) WithIdentity(allowReauth bool) {
	acc, _ := gophercloud.Authenticate(...)
	fn(acc)
    }

Inside of the function's method, you can invoke the function itself. Then you would define the actual AccessFn as some variable:

    fn := AccessFn(func(gophercloud.AccessProvider ) {
        // ...
    })

and then call its WithIdentity method:

    fn.WithIdentity(ar)

And then, if you were so inclined, you could define an interface over your WithIdentity method, so that in whatever context you're in, anything that has a WithIdentity method will do. This often (but not always) leads to code that's a bit easier to read. Nested callbacks... not always the easiest to wrap your head around.

shameless plug: I've written about this in more detail here, if this sort of thing interests you: http://jordanorelli.tumblr.com/post/42369331748/function-typ...

jnoller6• 13 years ago

I shot this to Samuel the current dev lead - thank you!

kc5tja4• 13 years ago

Goodness, I'm confused. I'm going to need a lot of time to digest this. That said, much discussion about ideal way of writing Gophercloud happened on the #go-nuts IRC channel, and nobody objected to what I have now as being unidiomatic. You're the first!

Deeply nested closures are definitely a problem, but I usually solve that by factoring closures out into top-level functions of their own.

That said, one of Gophercloud's goals is to be as idiomatic as possible. Are you available on #go-nuts if I have questions? I'd like to play with this technique some more and get a feel for it before applying it to (and risk breaking) gophercloud's existing API.

kapilvt4• 13 years ago

First there's a huge caveat to calling this multi-cloud, ie. its a library for openstack clouds only from what i gather of the src and documentation. Second, there's already a nice go library for openstack clouds that's seen multiple cloud usage (rackspace, hpcloud, openstack grizzly/folsom) in the form of http://launchpad.net/goose It has significantly more coverage of available openstack services (image, compute, storage) and more realworld usage and better testing afaics.

jnoller6• 13 years ago

No, there is no caveat; we're working on openstack first, then rackspace, then others. It's just started out obviously. We examined goose/juju and preferred to approach this via the Apache 2 license ala all of the other tooling/SDKs out there we (Rackspace) support and assign developers to.

Yes, it is currently focused on openstack clouds, but the goal is to reach parity with Fog (ruby), jclouds (Java), Libcloud (Python), pkgcloud (node.js), and so on.

The focus is a series of clean abstractions for users to be able to build tools and programs which can talk across any cloud provider easily using a common set of names / abstractions.

kapilvt4• 13 years ago

ic. Probably worth updating the readme then atm its got "Gophercloud provides a Go binding to OpenStack cloud APIs."

fwiw. goose is LGPL.

I'm at a bit of a toss up re LCD between clouds vs exposing the underlying functionality of the clouds. The LCD apis (libcloud from experience) break down with real world usage even for simple things like net security groups, or zones spec into provider specific usage. It seems like most cross cloud apps ends up needing to redefine the abstraction layer within itself rather than attempting to reuse the same LCD code across clouds at which point its not clear that the LCD layer isn't a limitation to taking advantage of a native cloud functionality.

kapilvt4• 13 years ago

sorry.. i couldn't let this go.. how is that not a huge caveat? your advocating and advertising for something that doesn't exist, and that does its existing job worse than other options.. ie a better alternative openstack client lib in go that focuses on openstack. i have huge amount of respect for you from pycon, and maybe this is your first rackspace advocacy... but sigh, all this is a call for developers afaics to work on a vaporware project afaics (going by the standard you set of multicloud). you haven't addressed my example of net security groups and how their elegantly solved by a generic lcd library.. because the fact is rackspace has no net security options in their cloud api, and really its incredibly cloud specific, i'm not talking about passing an a k/v for instance creation but actually configuring security groups (ports, etc), which is pretty basic stuff for a cloud provider.

"I hope users and consumers will keep us honest - as we said in the post, our interest is pretty much no lock-in on an API level (hence: openstack) and no lock-in on the code level."

any api is a lock-in, openstack just happens to be opensource in implementation, but per google v. oracle, there doesn't seem to be any qualms in other opensource impls using the amazon apis (cloudstack, eucalyptus) to deliver what customers want without lock-in.

microtonal2• 13 years ago

Go also shares some attributes with Erlang; in particular, its “goroutine” construct makes writing concurrent software much easier than managing threads by hand,

That's a lot of handwaving. The overlap is that both Go and Erlang have lightweight threads. But Erlang uses the (fairly) different actor model, which usages messaging between actors, supervision trees to control and monitor the life cycles of actors, remote actors, etc. Go has lightweight threads and channels to communicate between threads. Go (concurrency) shares very little with Erlang.

It's also funny that they move on to claim that Go has better concurrency support than Java, when in fact, you can use real Erlang-like actor networks in Java by adding one dependency to your Maven file. Akka actors are lightweight concurrent entities, since you can easily create hundreds of thousands or even millions of actors.

That said, it's interesting to see one API for multiple clouds.

[1] http://akka.io/

kc5tja4• 13 years ago

Both Erlang and Go's concurrency models are strongly influenced by Hoare's work in Communicating Sequential Processes. That certainly qualifies. You appear to be interpreting "some attributes" as a deep-dive analysis into the unique traits of what makes Erlang, well, Erlang. That was never the intention, and I'm not sure what motivated this.

Actor model libraries exist in various states of maturity for Go, as well.

microtonal2• 13 years ago

Both Erlang and Go's concurrency models are strongly influenced by Hoare's work in Communicating Sequential Processes. That certainly qualifies.

Erlang uses the actor model, while Go uses (a modified version of) CSP. Earlier versions of CSP were more like the actor model in that they did not introduce a level of indirection (the channel). But it's not the same concurrency model.

You appear to be interpreting "some attributes" as a deep-dive analysis into the unique traits of what makes Erlang,

What else am I supposed to do? Otherwise, it would be a statement as meaningless as 'Haskell shares some properties with Algol 68'. I am pretty sure it does.

That was never the intention, and I'm not sure what motivated this.

Erlang is well-known for its concurrency features and the reliability provided by the actor model (including supervision, etc.) (e.g. in telephone systems). It is quite easy to read the quoted sentence as an attempt to give a good reputation to Go by comparing it to Erlang. I only question if such a comparison is warranted.

Actor model libraries exist in various states of maturity for Go, as well.

Well, if they are available in various states of maturity, I am interested to know which one is the most mature. Does it cover most of the functionality of Erlang and Akka?

(The most interesting package that I found some time ago was a binding against the Erlang C nodes functionality.)

bebna• 13 years ago

And I hoped for cheap gopher:// hosting.

jnoller6• 13 years ago

Will pass to product management. Could be possible.

ggoodale• 13 years ago

Seems like you've got a WAIS to go.

gtaylor• 13 years ago

I think it's a noble effort, but I'd be strongly concerned with the ease of use/digestion/maintainability of something that strives to cover most major cloud APIs. This stuff moves fast, and I can't imagine having to try to keep up with multiple larger platforms.

While we were getting Pathwright launched, we contributed a whole lot to boto, which is primarily (but not exclusively) catered to AWS. It grew Eucalyptus (not a huge jump) and Google Compute support, but they were badly documented/tested and still seem to see much less maintenance than the AWS stuff. The situation has improved slightly as Amazon has hired a few more guys to help the maintainer make boto awesome, but they're still playing catchup with the core competency (AWS). This isn't to talk badly of boto, it's more to illustrate the magnitude of the task of keeping up with even one aggressively developing vendor.

I worry that these "mega-cloud-libs" are the wrong approach. The OpenStack stuff will probably end up with excellent support, but the others are likely to see much less attention, documentation, maintenance, help. The neglected areas of the codebase get to be embarrassing and not up to the quality standards of the primary focus (OpenStack in this case). Perhaps they end up falling behind in API versions for some vendors, limiting their utility.

This is definitely not to poo-poo the effort, I think it's a really neat ideal. Personally, I'd love to see them focus on making a kick ass OpenStack Go API with top notch documentation and tests, and leave the other services to their respective communities. That alone would be noteworthy.

jnoller6• 13 years ago

Thank you for the feedback - there is definitely a risk of lowest common denominator, and some truth that a generalized toolkit won't be as "good" as a domain-specific tool. For openstack specifically there's goose/juju right now.

We (rackspace) have had great success with Fog, jclouds, pkgcloud, libcloud, and other multi-cloud toolkits. For openstack / Rackspace cloud we also have php-opencloud, openstack.net and pyrax (which uses the python-XXXX clients from openstack core).

It definitely is a fine line to walk; a good abstraction though means that you can use the most common features easily and float through providers, but you have access to the advanced features if you need or want them.

Commonly those advanced features (on a provider level) mean a greater level of vendor lock-in which is why they may not make sense to expose in the higher level abstraction.

kc5tja4• 13 years ago

I'd also like to point out that having a base-line level of functionality often allows for domain-specific extensions to be written, either as adjunct libraries (c.f., libcloud and rackspace-monitoring libraries for Python), or as integrated extensions in succeeding versions (how Emacs evolved over time).

You're right to think it's impossible to keep up; even getting this far, I've heard rumors and tremors about up-coming V3 APIs in several areas. I'm sure by the time V3 support is contributed, V4 will be on the drawing board. Thankfully, the general availability of a V3 API does not usually preclude the continuing support for V2.

All we can do is deliver value to our users in a consumable format while trying to remain aware of any changes in our ecosystem.

icedog2• 13 years ago

Why do I have to read six paragraphs before I find out what Gophercloud is?

jnoller6• 13 years ago

We wanted to do a quick introduction to Go ideas so that those of us "not in the know" would know the why, what and additional context.

icedog2• 13 years ago

Yes, obviously you're doing an introduction to Go. But you may want to re-evaluate your description of quick.

vanderZwan• 13 years ago

by Rob Pike and Ken Thompson

Why does everyone always forget Robert Griesemer? He might not be as involved with the public as Rob Pike, but he's the guy that made gofmt and godoc (and I'm pretty sure he's also at least partially to thank for the language being specified in such a way that it is "gofmt"-able in the first place, considering he maintains the spec). That alone counts as a pretty big positive influence on the language.

gecampbell• 13 years ago

Thanks for that. I'm new to Go myself, and that's some helpful info.

Gophercloud: A Multi-cloud Software Development Kit for Go

Related Stories