Portable Cloud Programming with Go Cloud

https://blog.golang.org/go-cloud

by ArmandGrillet • 8 years ago

442 148 8 years ago

blog.golang.org

alongwittheflow• 8 years ago

Why is Google advertising its own product through official golang blog? It looks very amateurish.

spraak3• 8 years ago

How so? Go Cloud works with AWS and soon others. That seems pretty mature to me.

stirner• 8 years ago

Because it's not a Go project, it's a Google project[1]. It's like Apple posting new iOS features on the Swift blog, e.g.

[1] https://github.com/google/go-cloud

tomcam• 8 years ago

FWIW I’m the target market and delighted to learn about it on the blog.

MoOmer• 8 years ago

The examples reference the use of an Amazon Web Service product for the abstraction. I don’t see a problem here.

pkaye3• 8 years ago

The primary example they gave was on AWS.

spraak3• 8 years ago

Maybe it's a biased and distorted perception that I have, but Go seems to be the most consistent and unsurprising language I've used, and I mean this positively. I haven't written any Go for over a year since my current job mostly uses Node, but I feel immediately comfortable jumping in to the examples in this blog post. I don't think that I'm explaining what I mean very well, but it's something I don't think is so true for Node/JS and other languages I've used.

pkaye3• 8 years ago

Our company uses Go. Most people who join have no experience in Go yet usually they can get comfortable with the language in 1-2 weeks to start contributing. I know nothing about Node/JS but I'm sure that takes much longer?

nostalgeek2• 8 years ago

but I'm sure that takes much longer?

Your people can pick up go reflect package or go routines/channels and co in 1-2 weeks and start contributing? impressive... most people can pick up for loops, if,switch statements, variable and function declaration in a week in any language as well...

In fact, a lot of non programmers use JS regularly to write scripts in the browser. I imagine your people know how to program before hand... So what do you think takes longer to learn? mastering go threading model or being able to add interactivity to the DOM?

stcredzero8• 8 years ago

Most people who join have no experience in Go yet usually they can get comfortable with the language in 1-2 weeks to start contributing.

A lot of what makes anything hard to learn are the non-obvious gotchas. The design philosophy of golang is aimed at avoiding the non-obvious gotchas and the creep-up-on-you-at-scale gotchas.

aphexairlines• 8 years ago

1-2 weeks is standard for all languages though. A week or less is how long it usually takes people at work to pick up Objective C and contribute to iOS development, or pick up Scala and contribute to service development.

stcredzero8• 8 years ago

The problem with inconsistencies and surprises in languages, is that these are often the basis of conundrums that are more expensive to detect, figure out, and fix by a factor of 10X or more. I say this with appreciation and without irony: It turns out that "Blub" is actually a good thing, so long as it's done right.

All languages are going to suck, somewhere. The trick is to actually optimize a language for the difficult, expensive parts of development. Many languages seemingly optimize for writing things from scratch slickly. They're "blog example optimized." Instead, languages should be optimized for debugging, reading code, and deployment.

The fact that Go is not afraid of boilerplate is often actually a good thing. I've had to debug C++ in the confluence of 2 templates, such that there was no source code to display in the debugger. I don't expect to have this problem in Go, ever.

nine_k2• 8 years ago

There's a fine balance here.

"You can't pay people enough to carefully debug boring boilerplate code. I've tried." (Yaron Minsky of Jane Street)

While C++ goes to one extreme, not having certain expressive means (whatever they may be) has a price, and for certain kinds of problems, this price may be high.

This returns to the idea I very much agree with: "The trick is to actually optimize a language for the difficult, expensive parts of development." The catch is that difficult parts are also problem area-dependent. Go definitely found a sweet spot for certain kinds of problems, though.

sbjs2• 8 years ago

Meh, everyone has a preferred type of language they find least surprising. To many Ruby is the least surprising, and Matz used "principle of [his] least surprise" as a guiding principle when designing it, but many other people find it enigmatic and confusing. Some people find Haskell the least surprising and most consistent, others say Java is. Every language has caveats and inconsistencies, even Go. Like how Go has generics, but only for two or three built-in types. It seems to be more about the person trying out the language than the language itself.

sbjs2• 8 years ago

The article has two examples of how I already find Go to be inconsistent and/or surprising:

    w, err := b.NewWriter(ctx, "gopher.png", nil)
    ...
    _, err = w.Write(data)
    ...
    if err := w.Close(); err != nil

Why is := used on the first line but = used on the second line, and then := used again on the third line? I get that you use := when introducing a new variable, and = when just assigning, the third line is not really introducing a new variable, since err is still in scope... or is it?

Not to mention, what is the last argument `nil` supposed to represent? In most high quality Node.js libraries, functions/constructors will take an optional options object so you can omit it, or pass an object like { timeout: 300 } to it.

stcredzero8• 8 years ago

Minimalism helps. Ruby has one kind of minimalism, but fails at minimalism in some other ways. The same with Go. People who like Haskell like the kind of reasoning it facilitates. People who like Java want a managed version of C++'s "good parts."

A lot of it has to do with expectations. Going beyond the principle of least surprise, there's also a principle of "no dismay." Go is like the dependable old Corolla. It sets modest expectations, and it delivers a high degree of utility very consistently, and if you're ever let down, it's very rare. C++ can cause you utter dismay due to rather subtle slip-ups. I've only seen this level of dismay in Golang around subtle resource release issues around prepared statements.

sjellis• 8 years ago

It seems to be more about the person trying out the language than the language itself.

This is true, but Go does have a clean and mostly familiar syntax, by design: you can mostly follow along with a piece of Go code even if you haven't learned the language.

I think that what the OP is pointing at is a related issue: readability. This is explicitly a high-priority design goal for Go: lots of things in the language are designed so that developers can easily read and understand the code in the projects that they work on, and that memory use is never hidden.

paulddraper4• 8 years ago

unsurprising language

---

This is a valid Go program.

    import "fmt";

    func main() {
        fmt.Println("Hello world")
    }

This is an invalid Go program.

    import "fmt";

    func main() {
        // fmt.Println("Hello world")
    }

Surprising.

---

Creating typed general-use data structures is impossible.

Surprising.

tree_of_item• 8 years ago

Nope, you have to use imports. It's a little annoying but not "surprising" since the language shoves it in your face right away.

tapirl2• 8 years ago

neither of your example is a valid Go program.

pmontra• 8 years ago

For me the big surprise is that we still have languages where we have to write things like

  func setupBucket(ctx context.Context) (*blob.Bucket, error) {
      sess, err := session.NewSession(&aws.Config{
          Region: aws.String("us-east-2"),
      })

full of ( ) { } * and & as if we were still in the 80s. So Go is a very surprising language, but this is highly subjective. No hard feelings.

didip2• 8 years ago

For a statically typed language, the number of sigils Go have is surprisingly low.

tapirl2• 8 years ago

Most popular languages are full of (). Most popular languages are full of {}, except python. All C family languages are full of *.

I can't see any surprises here.

robbrit2• 8 years ago

You're right, it is dependent on your perspective. Go is intended to be unsurprising to developers who want to just be able to look at code and have a good idea of what it does. Punctuation is very useful to get rid of ambiguity.

Contrast this with a language like Ruby which tries very hard to make punctuation optional, and often ends up with ambiguous cases where you aren't 100% certain how the interpreter will parse some code and so you end up throwing in all the punctuation anyway.

notdang• 8 years ago

I prefer it like this instead of relying on indentation

outworlder2• 8 years ago

It is so consistent and unsurprising to the point of being boring. Which is a quality I have never thought I would appreciate, but if I am at work I do. It's very utilitarian.

I'll take Scheme for a spin if I am doing something for fun.

EDIT: %s/bored/boring/g

untog• 8 years ago

Go is certainly very easy to reckon with. But I was surprised at how little O cared for it when I tried it, given the hype I'd heard. The two things that immediately jumped out at me were error handling and generics. I understand both were deliberate choices but I'm still surprised they were made - I've been working with Swift a lot lately and it just feels like it makes so much more sense to me.

I do love how opinionated Go is, though. Every language should have a gofmt equivalent.

symlock• 8 years ago

Yes, gofmt is a wonderful since so many other languages can have very different way of writing code leading to harder-than-necessary digestion of new libs or projects.

cloverich• 8 years ago

I feel like a caveman typing out error handling code in go, but on the other hand (even as a beginner in Go) I _never_ run into unexpected conditions. There's tradeoffs of course but every time I find myself thinking about what kinds of error conditions may be happening in my Node code (where I spend most of my time), and whether I covered them all... i think I'd trade.

NiceGuy_Ty• 8 years ago

Seems like a very useful library. Any similar libraries for other languages?

spiderPig• 8 years ago

Isn’t this just centralizing the cloud provider SDKs? Instead of AWS, GCP, Azure designing their own SDKs, they have a common contract to adhere to. I’d imagine as the cloud becomes more mainstream, you’ll see similar support in Java and .Net (a System.Cloud package for example)

colek42• 8 years ago

It looks like it, but think of the power. If you write your app for S3 and Amazon raises their prices 100x you can be running on GCP in no time. Of course, you just create interfaces for your data model and implement them with your favorite service, but nobody has time for that.

gnufied• 8 years ago

There is Fog library for ruby.

ryukafalz• 8 years ago

It's a lot like Libcloud for Python: https://libcloud.apache.org/

sciurus2• 8 years ago

libcloud (python), jclouds (java), and fog (ruby) are three that come to mind.

stcredzero8• 8 years ago

I'm working on a new kind of container that allows for lambdas with very-very low latency. (Suitable for running a game loop.) This is absolutely awesome! Programming against the auto-generated AWS golang APIs is pretty painful. It works, but it's not idiomatic and overly verbose. I'm going to give it a try and see if switching to GCP is that easy.

tmpz22• 8 years ago

FWIW there is an AWS-go-sdk-2 our there that’s more idiomatic.

audi0slave• 8 years ago

Have you looked at this: https://www.usenix.org/system/files/conference/atc18/atc18-o...

didip2• 8 years ago

Even though this is nothing but a centralized client-side libraries, it allows Go developers to live in a multi-cloud environment sooner.

I am pretty excited about it.

cflewis4• 8 years ago

Hey ho. I worked on this project.

The majority of the team is in SF for Google Cloud Next to present this. I'm about to hop into the car to head up myself to watch, so I can do a little AMA throughout the day. Reply to this to make sure I see your question!

Here's a question I've already seen:

Is this a Google project or a Go project?

It is a project led by the Go team, and we are very aware of our responsibility to be cloud provider neutral. At the same time, while the project is so new and evolving so rapidly, we need to have working examples that we can change atomically in a single repository (atomic changes necessary for rollbacks and such). We don't want to have cloud provider-specific code in the Go project, so the code is currently sitting under the Google GitHub org. When the interfaces to different products stabilize, the intention is to graduate them up to the Go GitHub org. Then anyone can code implementations to that interface for whatever cloud they like in separate repos, similar to the way `database/sql` is structured. The cloud provider-specific code will always live outside the Go project.

But things are very new, so change is possible.

gant5• 8 years ago

I'm a little bit concerned because the last package that Google seems to have purpose-built for Go+Google (AppEngine?) was context, that gave us things like context.Context, a generic KVS that needs reflection, heavily relies on convention and doesn't work in linear time and "just make every function take a context and use context.TODO if you don't need it".

It also seems to be one of the biggest points of controversy for Go 2, with calls from "everything should have a context" to "context should go away".

ironjunkie2• 8 years ago

yeah, that context thing is a huge mess.

The libraries that support it have now all their function signature way longer, and it is not well understood if it is a "Request for cancellation" or something else

atombender• 8 years ago

Context is a pet peeve of mine. It's a design mistake as implemented, even if it serves an important function. It infects everything, forcing every method to pass contexts around, and it conflates multiple separate concerns (cancellation, deadlines, and key/value data).

Given that all of a Go program sooner or later will need contexts somewhere, it would be much cleaner to let goroutines have this stuff implicitly. I don't know enough about the intervals to say whether this is feasible with the current runtime, though.

rajathagasthya• 8 years ago

This looks great! Are there any beginner friendly issues for contribution? Didn't find a related tag in Github issues.

padiyar83• 8 years ago

zombiezen• 8 years ago

Hi, Tech Lead for Go Cloud here! Glad that you're excited to contribute! We don't have a tag yet since we're just starting out. The best way you can contribute right now is to use Go Cloud for yourself and file any issues that you find.

JustARandomGuy• 8 years ago

Is this compatible with the Go runtime of Google App Engine[0] currently? I was having difficulty getting the golang Google Cloud Storage client to work on App Engine several months ago.

0: https://cloud.google.com/appengine/docs/go/

merb2• 8 years ago

a lot of stuff is hard to do an App Engine Standard with Go. (especially `dep` support and I guess `vgo` aswell)

cflewis4• 8 years ago

Possibly! We're targeting Go 1.11 as we want the module support. It might work on GAE as is, but we haven't tested.

mfer2• 8 years ago

we are very aware of our responsibility to be cloud provider neutral

Yet, Azure support is listed as unplanned.

The language could mean that this is not going to be planned. The fact that y'all have not worked with Azure Go folks to get this moving is something to ponder.

A reminder that Go is a Google language and isn't vendor neutral?

skybrian4• 8 years ago

Maybe they're hoping Microsoft will do it?

vsekhar2• 8 years ago

PM for Go Cloud here (and poster of the Azure feature request).

"Unplanned" on the issue tracker just means we haven't assigned it to a sprint milestone for the team yet.

Azure and more clouds are very much in the works. As an open project, there are many approaches and we're open to all of them.

One other point: since Go interfaces are implicitly fulfilled, anyone can write and share implementations of Go Cloud for new providers or custom services without asking anyone for permission. If you take a look at the implementations we've provided, you'll find it's actually rather easy.

More to come.

thermodynthrway• 8 years ago

After Google threw half of Android into "Google Play Services" I don't believe it.

Then anyone can code implementations to that interface for whatever cloud they like in separate repos, similar to the way `database/sql` is structured

Of course they can, but you're working on a Google-centric version first right? Otherwise why would it be at Google Cloud Next?

That makes all other cloud providers second tier, and I don't think you should call yourselves "platform neutral". If you were, you would code up multiple platforms from the beginning, like is normally done for support on different operating systems for example

lazyhacker3• 8 years ago

(Eng Manager for Go Cloud) The APIs and HTTP server released today all work on AWS as well as GCP and support for Azure is a top priority but didn't make it for today.

Please see the comment from our PM regarding Azure: https://news.ycombinator.com/item?id=17604358

go-question• 8 years ago

Is there a session or talk on this at Cloud Next?

cube2222• 8 years ago

There was yesterday.

lazyhacker3• 8 years ago

https://youtu.be/_2ZwhvIkgek

justinclift2• 8 years ago

Why the MySQL support?

Seems pretty weird to include support for an Oracle offering from the get-go, especially given the bad blood between Google and Oracle.

lazyhacker3• 8 years ago

It's more important to us to address what developers want and many developers are using MySQL even on clouds.

arendtio• 8 years ago

I like to see how Go seems to getting some traction within Google. For years, I had the impression that Google was more Java/JS/Python driven and I was wondering why they weren't using their own language for more projects :-D

robbrit2• 8 years ago

Google is actually C++ centric - every other language is a bit of a "second class citizen" when interacting with infrastructure.

Go is still new and Google is "old", so there is surprisingly little Go code at Google - it doesn't make sense to rewrite critical infrastructure just because you have a shiny new language.

The derivative is positive though, and slowly Go is being used for more and more new development.

erik_seaberg3• 8 years ago

When I left, Google was nearly 50/50 C++/Java with some lingering low-volume Python. Language-neutral RPC was key to not being stuck in one language.

pjmlp4• 8 years ago

I can assure you that on Android world C++ feels pretty much "third class citzen".

pinkgodzilla3• 8 years ago

I wonder what kind of monitoring and alert system must exist if any cloud provider changes their SDK or APIs (signature, logic, capability etc.) upstream while your go binaries are in production. Did Google manage to get cloud providers on board to register and update a 1 to 1 match between the upstream api and the go cloud api?

Secondly on the article's mention on multi cloud usage, you would need to be a pretty large place to need (or even bothering to assess and convince yourself that you need) to use multiple cloud providers at once. Just learning and tweaking settings in the cloud providers GUI console is half the battle won sometimes.

Saying all this, this, together with the data portability announcement, it is definitely great for competition and going to bring great resilience in your code base to be SDK agnostic.

fishywang2• 8 years ago

I don't think cloud providers would just change their SDK or APIs without notice. It will break anyone using their SDK, not only go-cloud.

For your second point, it's normal for small startups to shop around and pick the cloud provider that's more suitable for their needs. For example on my last job at a small startup, we switched from AWS to GCP for cost reasons, and I implemented an abstraction (similar to go-cloud's) between S3 and GCS because of that.

pinkgodzilla3• 8 years ago

They never change over the course of say 3 years? 5 years? If it changes frequently that's great - someone could be put to task to monitor. If it changes very rarely, you will have a surprise outage although I am guessing debugging and fixing it would not be an issue unless you have SLAs you need to keep up with.

outworlder2• 8 years ago

Secondly on the article's mention on multi cloud usage, you would need to be a pretty large place to need (or even bothering to assess and convince yourself that you need) to use multiple cloud providers at once.

Oh, this is not the case at all. Do business with any retailer? Try telling them your stuff is hosted on AWS and see how that works for you.

pinkgodzilla3• 8 years ago

If this occurred frequently for my business, you would architect your system for on-prem flexibility and capabilities. However, I think for just "data" like S3 blob storage, this is definitely a no brainer. I would imagine for VMs its going to be a lot more difficult with load balancing, permissions and what not.

pwaai• 8 years ago

Why should I learn go and use go cloud over AWS SDK?

barrongineer• 8 years ago

They are trying to promote portability.

TheBlight• 8 years ago

Looks like they're trying to promote ease of moving customers from AWS to GCP to me but sure.

skybrian4• 8 years ago

Interesting that they built a new dependency injection framework, much like similar things in Java. I guess this doesn't replace Context, though?

https://github.com/google/go-cloud/tree/master/wire

enjoylife• 8 years ago

Doesn't replace context, and neither is it alone in terms of being a DI library, [1].

[1] https://github.com/uber-go/fx

latchkey• 8 years ago

As a Java engineer, I really missed DI in golang. Then I found fx.

Once I wrapped my brain around fx, I really liked it and used it heavily in a project I wrote. It was especially good for initializing 'things' of the same 'type'. Like a bunch of controllers for a web app.

One nice thing about fx is that it does not rely on code generation (uses reflection instead). I'm very curious what other features wire has over fx.

gant5• 8 years ago

Oh no, not another code generator.

That seems to be a common thing in Kubernetes too, if you don't want to spend the time writing actual code and Go is too "boilerplate heavy" for you, just write a code generator and bring the complexity of other languages right back!

liveoneggs2• 8 years ago

ironically terraform is already written in golang.

https://github.com/hashicorp/terraform Go 96.1%

pageald• 8 years ago

Terrafirm isn't provider neutral. It is just a declarative wrapper for cloud provider APIs. Code using one provider is not easily translated to another.

I do think it's an excellent configuration management and provisioning tool, but Go Cloud solves a different problem.

cflewis4• 8 years ago

Yes, we are solving a different problem. Terraform is about provisioning resources, Go Cloud is about interacting with them.

We actually use Terraform for provisioning in our samples, and we see the two working hand in hand.

erik_seaberg3• 8 years ago

A frustrating choice. I spend a few minutes per week running Terraform on a recent laptop. If it were written in Python, it might use more memory and require a few dependencies (which I don't care about at all) but then I could write plugins without having to deal with IPC and weird language-specific serialization (which Java should have convinced us is a bad idea).

liveoneggs2• 8 years ago

heh I thought you were making a funny sarcastic joke and then realized you were serious :)

smnplk2• 8 years ago

err, are you not nil ? err, are you not nil ? err, are you not nil ? err, are you not nil ?

err, speak to me...

smnplk2• 8 years ago

I need one more downvote :)

bhamta• 8 years ago

I like how Google tries to masquerade their corporate goals behind the 'portability' banner. Google Cloud is in a distant third place behind AWS and Azure, so ofcourse they don't want developers to have tight coupling to their competitors.

This portable cloud strategy is a mirage. I haven't seen anyone migrate providers easily (we tried in my last company). In reality, you have to break the cloud agnostic abstraction to leverage provider specific functionality. And then you are stuck with unmaintainable mess of 'cloud agnostic' APIs mixed cloud specific APIs.

mfer2• 8 years ago

Note, Azure support is filed under "unplanned"... https://github.com/google/go-cloud/issues/76

Interesting that the #2 cloud provider is listed as unplanned in a library coming out of the Go team.

spraak3• 8 years ago

I mean, AWS is #1 so that makes sense, and then their own platform as their second choice to implement, that makes sense too. I don't think it's like a conspiracy that they haven't released the Azure support yet.

vsekhar2• 8 years ago

See: https://news.ycombinator.com/item?id=17604358

skybrian4• 8 years ago

Having some common types available seems useful for people writing portable libraries. Maybe it's not going to change the world for most developers, but it could help defragment the library ecosystem a bit.

pietroglyph• 8 years ago

I like how Google tries to masquerade their corporate goals behind the 'portability' banner.

This may further Google's corporate goals, but that doesn't mean it wont actually increase portability, or that it's actually a bad thing for people who want to avoid lock-in.

In reality, you have to break the cloud agnostic abstraction to leverage provider specific functionality.

I agree this is a difficulty, but I think that it's also avoidable if you make a disciplined commitment to interoperability. Just as you have to put in the effort to use this new "portable" API, you also have to choose not to defeat the former investment. Saying that this API is useless because you end up using provider specific functionality anyway is like complaining loudly that something is broken when you're using it wrong in the first place.

weberc2• 8 years ago

Reminds me of the situation with SQL vendor abstractions. SQL even has a standard and it’s still difficult to migrate any nontrivial application.

sethammons• 8 years ago

ORMs immediately popped into my mind.

tybit• 8 years ago

Yes this is a strategic move on googles part, but that doesn’t make it any less valuable to consumers. Getting less lock in and more choice is good for everyone (except AWS)

kissgyorgy• 8 years ago

My understanding about a cloud provider lock in that it is a well thought-out, conscious decision (at least it should be) and after you made that step, you can use the specific services of the given cloud provider and you never move out.

Are those hybrid/mixed cloud solutions or move between cloud provider viable or that common?

Is there anybody went through a huge cloud migration which went well?

If you need to move, there are huge amount of data you have to move for example between S3 and Google Cloud Storage, is that even possible to do beyond an early stage? I can't imagine.

ljnelson• 8 years ago

Is this a Go-flavored version of something like https://jclouds.apache.org/?

jhabdas2• 8 years ago

It took us a while to get here. But it appears as if we've finally recreated the guestbook. And now it runs on someone else's machine.

dschulz• 8 years ago

I bet y'all would enjoy reading the article after installing this Chrome Extension [Cloud To Butt Plus](https://chrome.google.com/webstore/detail/cloud-to-butt-plus...).

solipsism3• 8 years ago

Am I the only person not excited by this? It seems to be based on the idea that you can just paper over the chasms between vastly different implementations. Putting aside the fact that the differences are important and you can't ignore them, doing this well is simply not scalable. How could it ever be all of: comprehensive, correct, up to date, elegant?

pjmlp4• 8 years ago

You're not alone.

This is like those DB abstraction layers because "we need to change DB", but in the end almost no one doesn't the products lifetime.

Also given that most of our clients don't use these fashionable cloud APIs, rather their own infrastructure, makes also not that much relevant to me.

freespirit77• 8 years ago

Will this compete directly with something like Terraform?

sandGorgon• 8 years ago

Is this the equivalent of Terraform +Boto3 in the python world ?

dev_dull• 8 years ago

Simple NCSA-style logging webserver. So important! I can't believe this wasn't part of the standard lib.

h8liu• 8 years ago

Package list with dependency graph @smallrepo.com

https://smallrepo.com/repos/github.com/google/go-cloud

kjgkjhfkjf• 8 years ago

Disgustingly ingenuous! Any language can be used to implement the factory design pattern.

hiacynt• 8 years ago

Oh no, it comes with its own dependency injection framework. This is just bad, non-idiomatic Go.

bborud• 8 years ago

This is tempting in the same way ORMs were once tempting. But just like with ORMs, this is solving the problem at the wrong abstraction level.

devj• 8 years ago

How is it different from Pulumi(https://www.pulumi.com/)?

Portable Cloud Programming with Go Cloud

Related Stories