From zero to Go: launching on the Google homepage in 24 hours

http://blog.golang.org/2011/12/from-zero-to-go-launching-on-google.html

by enneff • 15 years ago

274 73 15 years ago

blog.golang.org

cma2• 15 years ago

adamio• 15 years ago

Transistor density doesn't guarantee performance increase. Is AppEngine's ability to scale and maintain <60ms consistent with Moore?

dextorious• 15 years ago

The AppEngine is not about transistor density, it's about share-nothing and scaling by adding cpus.

BarkMore3• 15 years ago

There's insufficient data to draw conclusions about the performance of the infrastructure. The application probably ran on a shared machine. We don't know the impact of the other processes running on the machine. We don't know how many requests the application handled in parallel, so we cannot divide by latency to get throughput. We don't know if the application ran on a modern machine.

danielrhodes• 15 years ago

This was interesting because I've never seen Go used in production yet. However, the solution is not optimal for performance (as his benchmarking suggests is one of the goals).

If there is a finite number of possible images and each image takes CPU time to generate, then why not lazy cache the images in a CDN after generating them once?

cma2• 15 years ago

The more complicated something is, the more chances for failure there are. Remember, this was only for compositing the image that would be shared to google plus -- the one in the browser is done client-side.

enneff9• 15 years ago

This works exactly as you describe - the app sets a cache header on the response and Google's geographically distributed front end servers will serve the cached response. You get all this for free by using App Engine.

luriel• 15 years ago

There are quite a few people using Go in production already, see: http://go-lang.cat-v.org/organizations-using-go

curiouskat• 15 years ago

How is Go's websocket support?

I have been thinking Go might be a good choice for a websocket proxy, like node.js is used in Juggernaut (http://flask.pocoo.org/snippets/80/).

enneff9• 15 years ago

It's great http://golang.org/pkg/websocket/

nl• 15 years ago

Do websockets work on AppEngine(Go)?

biot• 15 years ago

I'm curious to know how many instances were required and what the cost would have been had someone been billed for it.

nickik• 15 years ago

Maybe its just me but all blogpost about go read like they where checked by a marketing guru after the have been written. The always enforce the same basic points, it always sound the same. If I somebody says "go" my mind alwasys jumps to "feels like a interpreted language".

enneff9• 15 years ago

Well I'm the guy who runs that particular blog, and I can tell you I'm no marketing guru. While Reinaldo and I edited the post together, I'm pretty sure that the line was in his original text. (Just checked: it was.)

Maybe people keep saying Go feels like an interpreted language because it feels like an interpreted language? :-)

jbarham• 15 years ago

I've done a fair bit of Go programming (https://github.com/jbarham), but have programmed professionally mostly in Python for the past 10 years, and for me Go does feel an interpreted language because it requires so few type declarations compared to mainstream statically typed languages like C++ or Java.

Checking the app source code for the article at http://code.google.com/p/go-thanksgiving/source/browse/app/a..., the only type declaration I can see in the body of a function is in a type switch where it obviously makes sense. Otherwise the code uses the combined declaration and assignment := operator (http://golang.org/doc/go_spec.html#Short_variable_declaratio...) which is type-safe because it infers the types from the value(s) on the right hand side. IMO this is even better than working w/ a dynamically typed language like Python because the required parameter type declarations in the function declaration tell you the types you're dealing with, but unlike C++ or Java you don't have to repeat that information back to the compiler.

technomancy2• 15 years ago

I see that claim everywhere as well, and I don't understand it. Does it just mean "it has a type system that you won't hate"? There are still plenty of type declarations in that code, whereas if you ported it to OCaml they'd all be inferred.

callahad• 15 years ago

> somebody says "go" my mind alwasys jumps to "feels like a interpreted language".

Because it does! I'm a pretty hardcore Pythonista who has never really gotten very far with compiled languages. I wrote a simple shell in C, once, but beyond that they never felt right. Somehow, Go manages to get things sufficiently "right" for me.

But more importantly, I think Go has a very strong chance of being the Clojure moment for Algol-derived languages. It's trying to address many of the same problems, and taking a similarly refreshing and pragmatic approach. Not to mention that the system-level language ecosystem is quite overdue for some reinvigoration. The glowing tone of so many Go posts may be, in part, due to a fundamental yearning for something that gets so much right -- something like Go -- to finally take off.

shadowmint• 15 years ago

Well, when you're looking at posts by someone who loves a language about that language, its not really that surprising to see positive feedback in it; and that feedback is often about the expressive syntax being nice (edit <-- and that its fast).

...but that said, I wouldn't mind some impartial coverage.

Its stuck me several times how defensive and no-I'm-right the go crowd gets when anyone dislikes it. Try jumping on the google group sometime.

Anyone remember that academic paper about relative speeds of c, go and something or another? As I recall Andrew went out of his way to reimplement the whole thing in go, and make a few snarky comments about "I wonder where they got those bench marks from, because that's not what I saw".

enneff9• 15 years ago

There have been many, many blog posts critical of Go in the past couple of years. They're not hard to find. If you're looking for impartiality, the blog of the people who created Go is not the place to look. I am going to keep pointing out how great Go is, because I use it every day and I haven't ever had so much fun programming. That's the truth.

You mischaracterise the blog post about the C++/Java/Scala/Go paper. It was written by Russ Cox, and he used it as an opportunity to demonstrate our profiling tools and how they can help you write fast Go code. (that it refutes the claims of the paper is just gravy ;-)

It begins: "At Scala Days 2011 a few weeks ago, Robert Hundt presented a paper titled “Loop Recognition in C++/Java/Go/Scala.” The paper implemented a specific loop finding algorithm, such as you might use in a flow analysis pass of a compiler, in C++, Go, Java, Scala, and then used those programs to draw conclusions about typical performance concerns in these languages. The Go program presented in that paper runs quite slowly, making it an excellent opportunity to demonstrate how to use Go's profiling tools to take a slow program and make it faster."

http://blog.golang.org/2011/06/profiling-go-programs.html

comex2• 15 years ago

Is Go's character type signed? If so, it's possible to make p negative, in which case this wouldn't apply:

        if p >= len(em) {
            panic(fmt.Sprintf("element index out of range %s: "+
                "%d >= %d", t, p, len(em)))
        }

although Go would still do its own runtime check for the dereference.

BarkMore3• 15 years ago

The variable "p" is a rune. Although the rune type can have negative values, p will only have values >= 0 in this code.

comex2• 15 years ago

How so? If the character is any value less than '0' (say, '!'), p will be set to a negative value.

yesbabyyes4• 15 years ago

I really liked the turkey, and I thought the solution looked elegant, so I had a go at it in CoffeeScript and node-canvas: https://gist.github.com/1475034

All requests in less than 66 ms, mean less than 19 ms.

jphackworth• 15 years ago

To be fair, you loadtested with ab, whereas the author launched his feature on the Google homepage.

yesbabyyes4• 15 years ago

Sadly I don't have the opportunity. :-)

ariwilson3• 15 years ago

AppEngine instances, unless they're running in a special backend instance, exist in a special containerized machine. These machines are very slow and don't have much RAM - 600 MHz and 128 MB. Based on the source that was posted on the blog, this app was running on a normal instance, as the source did not contain a backends.yaml file.

So it makes sense that a Core 2 Duo running at 2.13 GHz and slow CoffeeScript would be ~2x as fast. My personal experience of running Go code locally (on a Core i5 2500K) versus on a normal AppEngine instance showed a slowdown of around 6x or more.

yesbabyyes4• 15 years ago

I can't argue with that, though I'm running this on just one core (single process).

If I was better with the v8 profiler I would check to see how much time it spends in JS land vs C, since node-canvas is all C, as well as the http parser.

kamens• 15 years ago

This isn't true any more. They recently announced faster frontend instances with more memory (http://googleappengine.blogspot.com/2011/12/app-engine-161-r...), and I'm willing to bet this has been available internally since long before Thanksgiving.

I'd bet money that this app used the beefiest class of frontend instances.

alexchamberlain2• 15 years ago

What was the median response time?

billmcneale• 15 years ago

I can't help but cringe whenever I read Go code because the following:

        paths, err := filepath.Glob(dir + "/*.png")
        if err != nil {
            panic(err)
        }

is repeated over and over again.

Didn't we learn in the 90's that exceptions are far, far superior to return codes?

ariwilson3• 15 years ago

There are some real trade-offs with using exceptions, as per http://google-styleguide.googlecode.com/svn/trunk/cppguide.x... . However, as Go is garbage collected, some of these trade-offs don't apply.

timtadh• 15 years ago

Short answer no.

Long answer, the Go authors (Rob Pike et all) have a pedantic issue with exceptions as implemented in Python, Java, etc. They feel they conflate errors and exceptions. Errors, unlike exceptions, are an expected part of the programming process. Exceptions, on the other hand should be for exceptional circumstances.

I did not say that as well as they do so see what Pike has to say about the issue here: http://groups.google.com/group/golang-nuts/browse_thread/thr...

here is an a typical example of the Go community's attitude:

""" I _especially_ don't want exceptions to become an oft-used alternative to multiple levels of error return, as at that point they deteriorate to action at a distance and make understanding large code bases much harder. Been there, done that, got the scars to prove it. (Mostly from C++ and not from Java, but I've seen enough Java to have a healthy fear of runtime exceptions.) """

       -- Giles Lean

zmj• 15 years ago

Here's a good Rob Pike quote from later on in the thread:

"This is exactly the kind of thing the proposal tries to avoid. Panic and recover are not an exception mechanism as usually defined because the usual approach, which ties exceptions to a control structure, encourages fine-grained exception handling that makes code unreadable in practice. There really is a difference between an error and what we call a panic, and we want that difference to matter. Consider Java, in which opening a file can throw an exception. In my experience few things are less exceptional than failing to open a file, and requiring me to write inside-out code to handle such a quotidian operation feels like a Procrustean imposition.

Our proposal instead ties the handling to a function - a dying function - and thereby, deliberately, makes it harder to use. We want you think of panics as, well, panics! They are rare events that very few functions should ever need to think about. If you want to protect your code, one or two recover calls should do it for the whole program. If you're already worrying about discriminating different kinds of panics, you've lost sight of the ball."

enneff9• 15 years ago

Not really, no. I've always found them more trouble than they're worth. Explicit, in-line error handling works much better for me.

I'm not sure what you mean by "over and over again," in this context, though. There are only three error checks of that kind in this program. It's an unusual program anyway, in that _any_ error condition jumps to the same path: displaying the default image. In most programs you want finer control over error handling than what's shown here.

fauigerzigerk• 15 years ago

I doubt that's really the case in proportion to lines of code. Three error conditions handled by one and the same handler doesn't seem unusual at all. Even 10 to 1 or more wouldn't be unusual I believe.

In my view, all attempts at making a distinction between recoverable and non recoverable errors inevitably fail simply because it is a non local distinction that cannot be made for any particular piece of code in isolation.

jemfinch• 15 years ago

The two decades of experience we've had since the 90s, building things with exceptions, have fairly adequately demonstrated that they're not superior to return codes, despite their (apparent) theoretical advantages.

hasmoo• 15 years ago

Bruce Eckel on exceptions and Go: http://www.artima.com/weblogs/viewpost.jsp?thread=331407

arethuza• 15 years ago

A lot of those criticisms seem to be based largely on Java - which, with its checked exceptions, is pretty much the worst case in the use of exceptions in a language design.

tibbe2• 15 years ago

Reading the same code I also cringed, but for a different reason: the validity of `paths` is tied to the validity of `err`; you should never look at at `paths` without first checking `err`. We can solve this much better using sum types (also known as discriminate unions):

    // Pseudo Go:
    maybe_paths := filepath.Glob(dir + "/*.png")
    case maybe_paths of {
      Some paths: {
        // Do things with paths
      }
      None: {
        // Handle error case
      }
    }

This makes it impossible to use `paths` if `Glob` returns an error, as `paths` won't be in scope!

Furthermore, not having sum types forces us to include a distinguish value `null` for reference (pointer) types, as a way to communicate "no value". This is bad as now we cannot distinguish references that should never be `null` from those than can be `null`.

enneff9• 15 years ago

This kind of fussy type system is at odds with Go's design goals. It would make the language more abstracted from the underlying machine for little gain. I can't think of an instance of the issue you describe causing problems in real Go code.

merijnv• 15 years ago

You can distinguish between nulls which are errors and nulls that are just the actual return value. That is what err is for.

The entire reason Go has multiple return values for functions (such as paths and err in this case) is to avoid having to encode errors in the "real" function result.

Now I agree, that this is less nice as say pattern matching and real sum types, but if you know how to appreciate those just switch to a functional language!

danssig2• 15 years ago

A lot of people in this sub-thread don't buy that return codes are inferior, but they really are. The issue is that the low level code can detect the error and knows some strategies that could be done about it but it's only the high level code that knows what's best (here's the best possible approach [1]). If your library is used by a batch job running on some network-detached headless server it will probably have to handle errors differently than a fat client would.

[1] http://www.gigamonkeys.com/book/beyond-exception-handling-co...

4ad4• 15 years ago

> A lot of people in this sub-thread don't buy that return codes are inferior, but they really are.

What a great argument, I know better!

ars• 15 years ago

This is quite offtopic, but if even programmers are using jpeg when they should use png, what hope is there for anyone else?

The images this generates have mostly flat areas of all one color - it should have been a png.

bostonvaulter2• 15 years ago

Would the size of the png's be smaller?

enneff9• 15 years ago

We chose JPEG because the files were smaller and the encoding faster. The images looked fine in either case.

JabavuAdams• 15 years ago

I'm working on a game with highly-stylized art that includes a lot of blobs of single colour. It surprised me at first, but the artists have had real gains using JPG instead of PNG. Also, strangely the artifacts give the art a kind of "texturized" finish that's actually kind of pleasant on tablet screens.

canop_fr• 15 years ago

Maybe it's for performance reasons. I have a web server in go that generate (big) png on the fly and most of the time is spent in the png encoding (mostly the compression). PNG encoding is a heavy task.

Here's (from my blog) a pprof profile of my server : http://canop.org/pub/pprof4353.0.svg so that you can see the burden.

(I didn't make comparisons with other languages so I don't know if Go's png encoder is fast or not, I suspect it isn't)

liruqi• 15 years ago

the link is broken. something wrong with golang blog?

obtu• 15 years ago

Great Firewall?

wallunit• 15 years ago

The pirate easter egg [1] seems not to work [2].

[1] http://news.ycombinator.com/item?id=3351407

[2] http://google-turkey.appspot.com/thumb/295552a0

From zero to Go: launching on the Google homepage in 24 hours

Related Stories