Rewriting a large production system in Go

http://matt-welsh.blogspot.com/2013/08/rewriting-large-production-system-in-go.html

by qdie • 13 years ago

238 187 13 years ago

matt-welsh.blogspot.com

zzzeek7• 13 years ago

If I could get out of the 1970s and use an editor other than vi, maybe I would get some help from an IDE in this regard, but I staunchly refuse to edit code with any tool that requires using a mouse.

So you've shown that Go is appealing to rigid curmudgeons. Personally I'm still hung up on the "every function (edit: that does anything which might itself return an error code, which in large scale code is quite a lot) must return an error code" thing. Just like not understanding the purpose and usefulness of graphical editors, the creators of Go seemingly (despite their legendary status...) seemingly not understanding the purpose and usefulness of exceptions (waving them off as "they result in convoluted code" with no explanation) is keeping me pretty skeptical.

It just requires I take in some well written, idiomatic Go code so that I can finally "get it", how I'd live without exceptions, but I'm too busy being productive in Python (and enjoying Pypy's ever increasing speedups) to get that interested.

BarkMore• 13 years ago

It is not the case that every function most return an error.

The Go creators do understand the purpose and usefulness of exceptions. They chose not to use exceptions with this knowledge. See http://golang.org/doc/faq#exceptions for their reasons.

jlgreco2• 13 years ago

Go basically does have exceptions, though they call it something else, it is slightly less powerful than you would normally expect, and the idiomatic usage is very different.

Honestly I that Go's idiomatic usage of panic/recover is a good idea even in other languages. Exceptions passing package boundaries is typically a pain in the ass in practice, it makes for ugly code.

(My "day job" is typically Java these days, some Scala. I am not speaking from inexperience like many Go detractors like to assume. There seems to be a perverse notion that anyone who dislikes exceptions must not understand them. Silly.)

sgift• 13 years ago

I don't know. The quality of this FAQ entry is in my opinion narrowed by their inclusion of FileNotFoundException, which it seems they did not understand. The use case for this exception is not to allow lazy programmers to use it instead of checking that a file exists, but to signal to a programmer that his world view of the state of his program may be wrong despite his best efforts, e.g.

  if(file.exists())
  {
    //do something
  }

There's a race condition between the if and do something which invalidates the programmers world view (someone can delete the file between these two statements). And this is an exceptional situation the program has to deal with. Error codes may tell the programmer this, but it is quite likely that the programmer will just ignore it, because "I've already checked that it exists - what could possibly go wrong?". Exceptions, especially checked exceptions (in my opinion the only good exceptions for "normal" program code)(1), force the programmer to deal with this problem. Or to say - deliberately - "Hey, program, I don't care for the stability of my software. Just explode if this happens!".

(1) Languages which have only unchecked exceptions do, in my opinion, cave in to the laziness of programmers: "but, but, it is so much WORK to deal with all of this. Can't it just go away? Please?" - the result is code which can explode everywhere. This is even worse than no exceptions. With return codes you know at least that you have to check the code yourself very, very carefully all the time.

lbarrow5• 13 years ago

There's no requirement at all that functions return error codes. If a function can't fail, or if you don't care about if it fails, don't return an error.

codezero2• 13 years ago

http://golang.org/doc/faq#exceptions

A function isn't required to return an error code, but if you plan to deal with errors, it's the easiest way to do that.

I think the Go creators understand the purpose and usefulness of exceptions, they just chose to not implement them in favor of other approaches which work fine, of course, it's a matter of taste, as with almost any "language difference" argument. If you get real hung up on these differences, one might argue that you are the curmudgeon.

I didn't find learning/implementing a service in Go to be terribly challenging, even though I translated an existing Python service to Go. There were frustrating caveats that I had to work around because of some of the great things Python does, but over all the benefits of doing it in Go paid off, and it's about getting something for your time/effort.

Your last argument is totally legitimate though, but it's hard to say what would help you "get it" by throwing random bits of code at you, it's really something that you have to care about, spend the time to dig into, and make the realization for yourself. It may never happen, and as long as Python does everything you need, of course you'd have no incentive to use Go.

There seems to be a common theme (generally speaking) on HN that after the front page reaches some saturation on a particular subject, people start being very critical of it, not on its merit, but because it is taking up space where they expect to see a diverse set of content. I can sympathize with this, but I think it's best to try to dig for the value that others seem to be getting out of things, rather than sighing at the constant cheers of others, just my opinion though.

rshm• 13 years ago

I was hung up on error as return instead of exceptions but came in terms eventually. Now i think, compared to exceptions the error handeling is better off with go's approach. They are local, and require more concious effort. Specially while coding for the critical systems this helps as you are not passing the buck to some other section of the code.

MetaCosm3• 13 years ago

Don't confuse not understanding something with understanding it and thinking it is a poor idea. The Go authors understand exceptions perfectly -- they just thought that it was a bad idea: http://golang.org/doc/faq#exceptions

zzzeek7• 13 years ago

yes I've read that and their reason is "it results in convoluted code" - which is not at all my experience, programming in Java and Python for many years, it's worse in Java for sure due to the heavy emphasis on checked exceptions, but in Python they are a dream. "It also tends to encourage programmers to label too many ordinary errors, such as failing to open a file, as exceptional." also not true in my experience. I really disagree with the notion of hobbling a language just to prevent beginners from writing bad code. Beginners will always write bad code no matter what. I'm not a beginner, and I really don't need to be denied useful tools just because beginners will misuse them - I mean we're talking about improving on C/C++ for chrissakes, in the hands of a beginner those languages are like nuclear weapons. This particular FAQ entry makes it seem very much like the authors have just not seen exceptions used effectively, which I find kind of astonishing.

I watched Bruce Eckel's talk at Pycon, "Rethinking Errors - Learning from Scala and Go" (http://us.pycon.org/2013/schedule/presentation/52/) and I was so ready to be converted. But his arguments were pretty unconvincing.

Can't someone just make this case convincingly?

dragonwriter5• 13 years ago

Go doesn't require all functions to return error codes, and Go has exceptions (called panics).

In Go, conventionally, the publicly exposed functions in a module shouldn't usually panic but should use error returns to report unusual conditions. But that's a convention to keep the behavior of functions clear from the interfaces, not a language limitation.

pekk2• 13 years ago

The language has panics. But you aren't supposed to use them. So...

mdwelsh12• 13 years ago

The joke goes, "If every other line of code you write starts with 'if err != nil', you might be a Go programmer."

JulianMorrison4• 13 years ago

No, we can just see the "if err != nil". It's still there in Java or Ruby or Python, lurking in the notional space between the lines and waiting to branch to an exception handler you installed in the caller's caller's caller.

pjmlp14• 13 years ago

That is exactly because of those type of developers in the industry that the latest Bret Victor's talk is so interesting

http://vimeo.com/71278954

Basically he goes on to show how many people still code like the 70's instead of adopting languages and paradigms that were already possible in late 60's systems.

zzzeek7• 13 years ago

i hate watching videos but yes, this is really interesting - von neumann: "I don't see why anyone would need anything other than machine code". I deal with the resistance thing a lot in my work with ORMs, I should integrate some of this into my writing.

gatestone3• 13 years ago

Exceptions are just glorified gotos. The longest single time in my life chasing a bug was because an exception was firing somehere down udner and nobody noticed, because it was kind of part of the logic but a fringe case.

zzzeek7• 13 years ago

this is a thoroughly debunked argument that Joel tried to make. Unlike goto, exceptions have stack traces, so when used correctly, their source and propagation are immediately obvious and traceable. An unreported exception in your buggy program is certainly no worse than an ignored error code.

pvdm3• 13 years ago

Unless it is open sourced, I can't validate the claims.

codezero2• 13 years ago

If you have the time, rewrite one of your own services in Go, open source it, and share the post here on Hacker News.

packetslave4• 13 years ago

I'm not sure anyone is asking you to "validate the claims". If you'd like some reassurance that OP knows what he's talking about, you might consider that Harvard tends not to hire stupid professors and then give them tenure.

pvdm3• 13 years ago

I am well aware of Matt Welsh's reputation and what institution he was from. However, even Einstein or Feynman's scientific papers are subjected to peer review.

mdwelsh12• 13 years ago

I would love to open source this system, but even if we did I'm not sure we'd be able to convince you that using Go was better than some other language in terms of developer productivity. How would having access to the source help?

pvdm3• 13 years ago

You mentioned in the article that you read a section of code by one of the lead engineers and immediately you became convinced that using Go is the "way-to-go". Well, I would like to experience that eureka moment as well.

derefr8• 13 years ago

You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go, I wonder why Google never adopted Erlang. They could have been getting all these same advantages (and then some) a decade ago :)

pekk2• 13 years ago

Are Go and Erlang that fungible with each other? Honest question.

jlgreco2• 13 years ago

Not particularly. They are basically just both languages that were inspired by CSP; the similarities don't go much deeper.

derefr8• 13 years ago

Not in theory, but yes in practice.

For just one example, Go produces static native binaries, while Erlang produces bytecode for a virtual machine. But the Erlang virtual machine is tiny and it's standard practice (with tool support) to ship it with your application as a "release", so either way you get the effects of having one self-sufficient blob of code in a folder that you can "just run" without having to think about runtimes or libraries.

What I would say is that, for every IO-bound highly-concurrent C++ project Google is rewriting into Go, the same project could have been rewritten into Erlang, and they'd see most of the same advantages relative to C++: better, more transparent concurrency; "batteries included" for solving various client-server and distribution problems; being able to just drop a code package on a server and run it; etc.

lbarrow5• 13 years ago

One of the big projects they're working on in the next version of Go is to make the internal scheduler more Erlang-esque =)

MetaCosm3• 13 years ago

Yep, full preemptive scheduler, I am very excited about it!

continuations2• 13 years ago

The author did say CPU efficiency is key, and Erlang isn't exactly known for its raw speed.

In general search is very CPU intensive, maybe that's why Google never adopted Erlang.

banachtarski5• 13 years ago

Yes but intentionally so. There is an intrinsic tension between latency and throughput. Erlang chooses willfully to optimize for the former rather than the latter. This works when the majority of the tasks occurring concurrently are typically small and lightweight (aka, a web server).

seiji3• 13 years ago

I wonder why Google never adopted Erlang.

Collectively, google believes they are right and the world is wrong. Anything pre-existing is dirty and unworthy of their genius if they didn't invent it themselves.

So, even though we have 20 years of Erlang and production concurrency experience out there in one solid language, it's just ignored (except for parts they want to get "inspired by").

All of that is fine in isolation, but then the fadsters jump in. You know who they are. They're the people who live to consume fads and only do the latest thing without any consideration to what came before. Soon you'll have thousands of blog posts about how Go is changing the future of programming because they invented VM-scheduled lightweight green threads load balanced over your CPU topology.

mseepgood3• 13 years ago

except for parts they want to get "inspired by"

Go's concurrency was not "inspired by" Erlang. It was inspired by Hoare's CSP. Pike and Luca Cardelli created a CSP based language called Squeak in 1985: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.26.8...

cageface4• 13 years ago

Google is heavily dependent on Java, Linux, Python, C++ etc. About two seconds of thought is all it takes to realize what an absurd claim this is.

Google is almost alone in building internet services at its scale. The people calling for adoption of exotic tech like Erlang without understanding their unique requirements are the fadsters.

packetslave4• 13 years ago

Collectively, google believes they are right and the world is wrong. Anything pre-existing is dirty and unworthy of their genius if they didn't invent it themselves.

Someone a bit more objective might say "Google believes that it's quite hard to integrate third-party software into their huge existing infrastructure and make it work at their enormous scale."

They might also say "Google believes the infrastructure they use is the right choice for their business needs, and Googlers like to tell the world about some of it, in case it's the right choice for them, too."

But I'm a Googler and you clearly have an axe to grind, so it's unlikely we're going to agree.

rdtsc3• 13 years ago

One silly yet simple reason could be the Not Invented Here Syndrome. For a company that size relying on Ericsson or Erlang Solutions for support could have been seen as something they didn't want to deal with. So they just wrote their own.

It is too bad though. Erlang, I think, has some features I like better such as hot code reloading, better supervision strategies, separate heap per lightweight process (hence no stop-the-world GC), and is battle tested for longer. Also, at least to me, actors with pattern matched receive, as central building blocks, makes more sense than channels. So I'll keep Erlang as my concurrent server side go-to language for now.

zaphar4• 13 years ago

(full disclosure: I work at google and also like erlang)

Erlang has fantastic facilities for robustness and concurrency. What it does not have is type safety and it's terrible at handling text in a performant fashion. So if you don't care about either of those things and only care about robustness and concurrency then Erlang is great. There were internal discussions about Erlang here but the upshot was. We had already basically duplicated Erlangs supervision model in our infrastructure, only we did it for all languages and Erlang didn't offer any benefits in performance for us. It's only benefit would have been the concurrency model. That's much less benefit than Go gives.

Go gives you Erlangs concurency model, a similar philosophy of robustness, type safety, all batteries included, and performance. Equating the two languages works in 1 or 2 dimensions but not on all the dimensions google cares about.

dbaupp• 13 years ago

Go doesn't have strong type safety either; I remember a recent story about a Go stdlib function "accidentally" calling an interface it shouldn't.

derefr8• 13 years ago

Interesting, thanks for that; it's pretty much what I guessed (especially the bit about the supervision tree and hot-code-upgrade advantages being mooted by your infrastructure.)

On a tangent, though:

What it does not have is type safety

I've tried to work this out before (I'm designing a new language for Erlang's VM), but as far as I can tell, type safety is in practice incompatible with VM-supported hot code upgrade.

If you have two services, A and B, and you need to upgrade them both, but you can't "stop the world" to do an atomic upgrade of both A and B together (because you're running a distributed soft real-time system, after all), then you need to switch out A, and then switch out B.

So, at some point, on some nodes, A will be running a version with an ABI incompatible with B. In a strongly-typed system, the VM wouldn't allow A's new code to load, since it refers to functions in B with type signatures that don't exist.

On the other hand, in a system with pattern-matching and a "let it crash" philosophy, you just let A's new code start up and repeatedly try-and-fail to communicate with B for a while, until B's code gets upgraded as well--and now the types are compatible again.

It's an interesting problem.

DanWaterworth2• 13 years ago

Go gives you Erlangs concurency model

There are a number of significant differences between Erlang's and Go's concurrency models: Asynchronous vs synchronous communication, per-thread vs per-process heaps, send to process vs send to channel.

banachtarski5• 13 years ago

Benefits in "performance" is somewhat vague. What performance criteria was measured? Throughput? Memory consumption? Latency? Long tails? Standard deviation? Mean response time?

It's highly unlikely that any language can possibly provide all the above and Erlang makes certain tradeoffs (as does Go). Low level languages like C++ let the user choose the tradeoff at any given point with enough effort.

What was Google's criteria in making such a decision?

AYBABTME• 13 years ago

When people ask «Why not Erlang instead of Go, Erlang is X and Y and Z...» they seem to be oblivious to the fact that Go is C-like and Erlang has a pretty weird Prolog-like syntax.

I've had some experience with Prolog before touching any Erlang. It's not a syntax or a programming style that I liked, not at all. When I came to do some Erlang [1], I found the same style and it was not a pleasant surprise.

I learned C-like languages first, so maybe that's why my view is flawed. But most people also learn C-like languages as their first language, so it might be that Erlang looks like an ugly beast when they come across it. And thus as a concurrent language, Go seems to be first of it's kind.

Meanwhile, Go has a very simple style that pretty much everybody can read out of the box.

[1]: CS Games in Canada, one challenge was a 'debugging' competition where programs in ~10 languages had bugs we had to find and fix in 90 mins. One of them was in Go, another one in Erlang. Out of about 20 teams participating, me and another one (2/20) managed the Erlang one while the vast majority of the other teams managed to do the Go one. FWIW, this can speak to people's ability to understand Go vs. Erlang.

derefr8• 13 years ago

I guess I learned Erlang as my 15th or 16th programming language, so "syntax" wasn't really a concern; I really was oblivious to it. What's that Matrix quote?--all I see is the AST :)

Still, when I say "Erlang", I don't mean the syntax, I mean mostly the VM and stdlib (the platform semantics, in other words.) You can get those with Elixir or LFE or any number of growing projects, just like you can get Java's platform semantics with Scala or Clojure. Everyone agrees Erlang's syntax is hideous, after all--even its developers.

rdtsc3• 13 years ago

they seem to be oblivious to the fact that Go is C-like and Erlang has a pretty weird Prolog-like syntax.

Yeah it has Prolog-like syntax. However, when designing and working on large distributed system looking at just the syntax is a kind of shortsighted. The problem is not syntax (which is different, and is actually pretty simple, and a lot less ambiguous than say Javascript or has less "features" than C++), the problem is _semantics_. And by that I mean structuring your programs as a set of multiple concurrent actors. That is the hard part.

Another way to put it. Erlang is probably getting looked at because someone wants to either scale, build a distributed system or a highly fault tolerant system. At that point, if dots vs semicolons is a major stumbling block what are they going to do when they hit a netsplit.

Now, Erlang like any tool has trade-offs. But those are about isolation and private heaps vs raw sequential performance. Hot code upgrades and compiled static code and pointers referenced everywhere in the code is not going to work well. Stuff like that. Single assignment is also another common one.

(And if syntax is a major stumbling block, there is Elixir or a lisp like languages LFE and Joxa that all take advantage of the actor model and run on the BEAM VM).

Dn_Ab2• 13 years ago

More than Erlang I think what Google really wanted was Ada. Since speed (ada can be very fast with low memory usage) and programming at scale were as much concerns as concurrency (they both take inspiration from CSP). Ada trades verbosity for clarity and rarely matched safety (design by contract, modules and extensive runtime checks). While I've never written a line of it before, proponents of Ada always have interesting things to wistfully say about how ahead of its time and slept on it was/is.

pjmlp14• 13 years ago

While touted a complex and big language when it appeared in the early 80's, it is actually smaller than C++.

The main problems related to its adoption had to do with the price of the compiler systems back in the day and its verbosity for the curly-bracket fans.

Nowadays there is GNAT, but the language ecosystem is very different.

easytiger• 13 years ago

Erlang is slow

pjmlp14• 13 years ago

You know, every time I see some Googler shocked at the effectiveness and various advantages of coding in Go

Me too, but for other reasons.

Many of the Go nice features were already available in other languages back in the 80's and got lost with C and later C++ becoming mainstream.

That normal developers don't know them is understandable, but Googlers, given the requirements to be part of the Chocolate Factory, it surprises me every time.

lbarrow5• 13 years ago

It's interesting to see that more and more internal projects at Google are getting redone in Go. However, this sentence stands out to me in the article:

"Before doing the rewrite, we realized we needed only a small subset of the functionality of the original system -- perhaps 20% (or less) of what the other projects were doing with it."

I'm guessing that a lot of the benefit of the rewrite came simply from the simplification of the core logic and dropping extraneous functionality. That said, having written a little bit of both C++ and Go, I can completely see why the author found that Go was both far more readable and more maintainable than C++.

dodyg3• 13 years ago

This is why a "go" rewrite story is not that appealing to me. You can rewrite a subset of a system and gain many improvements simply by benefits of hindsight and clearer requirements regardless of languages.

gizzlon• 13 years ago

Obviously, but the author is smart enough to acknowledge this and take it into consideration. So.. what's your point?

mdwelsh12• 13 years ago

OP here. It is absolutely true that we did not rewrite the entire original system in Go; I tried to be very explicit about that in the blog post. But, I feel confident that it could be done, in much less code, with greater clarity and modularization. The Go language by itself does not force good software design. A rewrite in any language would have been better than the original system, but our decision to use Go turned out to be fortuitous in that we managed to do so in record time and with much greater programmer productivity.

lbarrow5• 13 years ago

I'm glad to hear it.

Was there anything about the language that you found particularly forced clarity and modularity? I'm giving a talk comparing Go and Ruby next month and I'm curious as to what people with experience with larger Go programs find to be most helpful.

onan_barbarian2• 13 years ago

Hmmn. I think there's a fundamental advantage in any rewrite that's allowed to drop functionality - even a C++ to C++ rewrite.

It's not just the language. It's the fact that (a) you already have an instantiation of the idea to look at as a reference and (b) you're happily cutting bits off the original. The latter suggests that you're not only not supporting all of the original use cases, you're probably in a nice state of organization where you're free from the temptation to "astronaut" up a more general system than you really need.

It's always nice to have this power, but it shouldn't be confused with the wonderfulness of the language.

f2f4• 13 years ago

ironically, that's not the common experience; the term 'second system syndrome' was coined for a reason.

"allowed to drop functionality", as used by you, is a strawman -- the team _added_ functionality to the part they were rewriting without incurring the wrath of the second system. sure, the line counts are off, but your argument does not stand in the general.

Lewisham2• 13 years ago

It's interesting to see that more and more internal projects at Google are getting redone in Go.

I think one of the key drivers of this which hasn't come up yet in the comments is this one:

"The #1 benefit we get from Go is the lightweight concurrency provided by goroutines. Instead of a messy chain of dozens of asynchronous callbacks spread over tens of source files, the core logic of the system fits in a couple hundred lines of code, all in the same file."

I think it would be quite rare to see a project of any significance at Google which didn't have an async aspect to it. My guess is that this is particularly the case when it was decided to be written in C++ over Java or Python: speed for this project matters, and the way to get the fastest execution is a distributed C++ program.

When you combine that with Rob Pike's assertion that most of the programmers moving to Go are from Python rather than C++, and you can see why rewrites are starting to gain momentum. I think there are a handful of people who truly do love C/C++, but I think most programmers (Google or no) see it as a necessary evil to meet the desired speed requirements. Now Go is stable and proven to work at Google, I can see lots of teams getting internal pressure to start trading out components from C++ to Go. I can see managers greenlighting it (perhaps as a 20% project) if only for the readability argument.

kailuowang• 13 years ago

It disheartens me a bit when people compare Go with C++. It gave me the impression that Go was created to be better C++ (or Java) mostly to make life easier for the thousands of developers at Google who are stuck with those two languages.

lbarrow5• 13 years ago

My understanding is that that is, in a single sentence, exactly why the project initially got started at Google.

http://www.infoq.com/presentations/Go-Google

enneff3• 13 years ago

That is at least partially the case. The other part of the story is that no existing languages were specifically designed for programming at a large scale.

Details: http://talks.golang.org/2012/splash.article

mdwelsh12• 13 years ago

OP here. What do you think we should be using apart from C++ or Java?

azth• 13 years ago

Scala. Rust also looks extremely promising, though not mature yet. It is getting there quickly though.

ternaryoperator• 13 years ago

Ken Thompson (a Go lead) has held hostility towards C++ that is well-known and of long standing. (To wit, at the ACM Turing Centennial last year: 'I'm not sure what OO is supposed to look like, but I do know for sure it's not anything like C++.' (not verbatim, but accurate in the sentiment he expressed.)) As note below, Go was indeed designed to address perceived shortcomings of C++, among other goals.

bane2• 13 years ago

So programming in Go is making me soft.

Something similar happened to me years ago when I started picking up Perl. All the BS C++ and Java made me go through to get anything done seemed like such a huge waste of time I ended just writing lots of stuff in Perl until I had pretty much forgotten how to C++ and how to Java.

Go might be a new Perl in that sense.

I'd also add that as Google moves more and more production systems onto Go, it will literally become responsible for handling billions of dollars of business. That and because Google is using it (and everybody wants to work at Google so knowing some Go ahead of time will be useful) will lead universities to start covering it and before long this virtuous cycle will result in Go everywhere.

I've also noticed a number of posts here over the last few months extolling the virtues of moving off of slow frameworks built around slow scripting languages in terms of huge reductions in infrastructure costs. For each system that Google brings on-line with Go, even if it took a team of 10 6 months, they've likely saved themselves millions of dollars in new capex infrastructure acquisition. Imagine serving 40x the requests off of the same old hardware. Alternatively, you might be able to use slower, but more energy efficient hardware to serve the same number of users and save on power and heat management. All this combined with Moore's law means this is a fantastic idea at this kind of scale.

npalli2• 13 years ago

"For each system that Google brings on-line with Go, even if it took a team of 10 6 months, they've likely saved themselves millions of dollars in new capex infrastructure acquisition"

I thought the old systems were in c++. So, how did the infrastructure savings accrue? In fact, I don't recall anyone doing analysis on that front.

dgacmu• 13 years ago

Matt linked to my course, but also: I did a tiny analysis of this with my pi searcher (http://da-data.blogspot.com/2013/05/improving-pi-searchers-s... ) -- it's a toy compared to the kind of system that Matt described, but my experience was extremely positive. Rewriting it in Go made it easier to architect the system right to take advantage of persistence. It's hugely faster.

We also have a paper accepted to SOSP this year (the academic weenies will understand that) where the system - a Paxos variant - is implemented in Go. We also implemented about four other Paxos variants in Go to compare against them, a task that would have been absolutely grueling in C++. In Go it wasn't too bad. Our performance isn't super-blazing, but it turns out to be one of the faster publicly available Paxos implementations anyway. I'll have to write up a bit about our experience with it -- I'm with Matt 100% on this one.

bane2• 13 years ago

You're right. I think I got ahead of myself in what I was thinking, conflating the rewrite of the C++ system into Go with the some of the other speedups I've been reading about coming from other languages/frameworks.

I'd be interested to know more from Matt about speedups or slowdowns in specific subsystems that are feature-for-feature complete (as opposed to comparisons of Go to C++ where the Go rewrite is simply doing less stuff because lots of cruft was tossed out).

mdwelsh12• 13 years ago

David Andersen at CMU teaches the undergraduate distributed systems course in Go: http://www.cs.cmu.edu/~dga/15-440/F12/

jussij• 13 years ago

Go's type inference makes for lean code, but requires you to dig a little to figure out what the type of a given variable is if it's not explicit. So given code like:

   foo, bar := someFunc(baz)

> You'd really like to know what foo and bar actually are, in case you want to add some new code to operate on them.

The gocode utility (which works with several editors) does exactly this.

grannyg00se• 13 years ago

"So given code like:

    foo, bar := someFunc(baz)

You'd really like to know what foo and bar actually are"

Wouldn't your editor, mouseless though it may be, provide for that? Does ctags not work with vi?

And why use vi over vim?

mdwelsh12• 13 years ago

OP here. I do actually use vim but I have yet to adopt most of the fancy plugins that vim provides -- put me in a time machine back to 1977 and I would be very capable of programming on any UNIX system that you'd drop me in front of, provided it had vi installed. I'm not defending this choice of lifestyle; it's just how I learned to program :-)

lifebeyondfife• 13 years ago

Even in the 2010s I've found proponents of the UNIX console style of programming because, as you say, they can be dropped in front of any similar system and off they go.

However, tools improve and you're denying yourself productivity benefits. One of the advantages of a statically typed language is that the computer knows the type of the arguments. I love environments that allow me to 'mouse over' a function or variable etc. and tell me exactly what it is (and allow me to navigate to its definition). You don't have to use a mouse but what's the disadvantage in having alternatives to keyboard shortcuts where appropriate?

rgarcia• 13 years ago

Re: figuring out the signature of functions or the definition of structs, I've found godef useful: http://godoc.org/code.google.com/p/rog-go/exp/cmd/godef

The Go emacs mode has godef-jump (C-c C-j) and godef-describe (C-c C-d): http://golang.org/misc/emacs/go-mode.el

sytelus• 13 years ago

The number of strongly typed garbage collected compiled languages is surely shrinking. I can understand someone's disgust towards Java considering dwindling velocity in adding next-gen features and now sleeping in same bed as lawyer run Oracle. That pretty much had left C# in the arena until Go came on the scene. I was honestly hoping Go would give us head-to-head battle with C# but from initial looks Go pretty much feels like C# 1.0. No generics, no partial classes, no linq... C# 5.0 even has async and dynamic types. I like minimalism but these features are something we take for granted in languages that were created post-2000 era. So it all boils down to people choosing Go for singular reason: It's not from Microsoft! If licensing for C# wasn't a barrier and let's stretch our imagination by thinking C# was created by some other group in Google, would Go have a chance?

pjmlp14• 13 years ago

Yep, Go would be hardly mentioned here if it wasn't being developed at Google.

Having said this, Go can be a nice replacement for many use cases one would use C for.

scott_s• 13 years ago

Go seems to be gaining traction. Does it matter if that's because of where it came from? Maybe in a cultural, I-want-to-think-about-why-some-languages-get-adopted sense, sure, then it can matter. And I think it's worth having those discussions. But I get the impression that people think there's some absolute injustice in the idea that people are adopting a language, and it's partly because it's from Google. Consider:

"Yes, C would be hardly mentioned here if it wasn't being developed at Bell Labs."

axaxs• 13 years ago

This isn't a fair statement. A better statement would be "Go would be hardly mentioned here if it weren't for all of the effort and documentation put into it." There is no doubt though that Go would not be moving along so quickly if it weren't for Google, but I think this is a case of 'correlation is not causation.' Dart comes from Google also, and while I see it from time to time here, nowhere like I do Go. Myself, I tend to trust and prefer Mozilla as a company more. That said, I find Rust not only unusable, but really too big of a language. I find Go the perfect mix of features I desire, and I cannot be the only one. So, I like Go as a community and as language and certainly not because Google is behind it.

bad_user• 13 years ago

The JVM is the best mainstream VM. We use it with Scala. Try it out.

Tichy2• 13 years ago

Sometimes less is more. Not implementing every fancy feature under the sun might be a bonus and keeps the language clean.

I know I never managed to like generics in Java. Don't know C#, though.

pjmlp14• 13 years ago

Java generics are an example on how not to do it, but it does not mean all generic implementations are like Java's.

gnuvince• 13 years ago

Languages like SML and OCaml are pretty clean, pretty simple and have an awesome parametric polymorphism system.

threeseed2• 13 years ago

Java as a platform is absolutely not dwindling in velocity. All that is happening is people are looking into alternative JVM languages like Scala, Clojure and Groovy. This trend has been happening for a while and is as much to do with Servlets/Spring as it is with Java the language. It has absolutely nothing to do with Oracle which has been a fantastic steward for the platform.

I don't think you actually have much understanding of enterprise environments otherwise you would know how ridiculous the idea that Go or C# is going to take over from Java anytime soon.

jesstaa• 13 years ago

The C# and the Go cultures are very different. Culture is a big part of what makes programming languages different. Python and Ruby are basically the same language, but the culture of the users is very different and so the libraries and tool are quite different.

Go is from the unixy C culture, C# is from the Java/win32 culture and the libraries and tools reflect that.

Nitramp• 13 years ago

I'd take a second look at Go. It has a number of innovations over C# - and of course also a number of shortcomings. It's quite a different language, much more so than Java compared to C# (1.0 even).

A couple of things of the top of my head:

    * goroutines and efficient multithreading, both in terms of syntactical constructs and language implementation. Go doesn't have await/async because it fundamentally doesn't need them.
    * a type system that works without inheritance or subtypes but rather static duck typing, quite different from any other language
    * very memory efficient layout of data structures, internal pointers
    * small set of builtins with maps, slices, channels (that do have 'generic' types for their members) to reuse, rather than re-implement
    * comes with a well designed build and packaging system
    * comes with a very convincing standard library

It's worth taking a look. C# surely is a nice language, but it inherited many of the design flaws of Java, in particular with respect to the runtime and memory layout. Google does use Java internally a lot, Go gets uptake in spite of that competition.

Regardless of language merits, licensing C# can indeed be a problem, as can be the operational complexity of managing Windows running on hundreds, thousands, ten thousands etc of machines.

dubbledidu4• 13 years ago

Did you ever actually use Go for a few minutes? Most of these bullet points are plain wrong.

orenbarzilai• 13 years ago

Can you share your new system benchmarks comparing to the old c++ system?

mdwelsh12• 13 years ago

Unfortunately, probably not. Hopefully one day.

lucian1900• 13 years ago

Interesting that Go's lack of HM type inference is that problematic. In, Rust or Haskell, one could simply use that value further on with some type and have it inferred correctly.

dodyg3• 13 years ago

Go is a nice language with awesome features except that it doesn't fucking allow unused imports/variables. It makes exploratory programming with Go annoying as hell.

im3w1l2• 13 years ago

    don't {
        print x for {x : unused variables}
        x() for {x : function from unused imports}
    }

Rewriting a large production system in Go

Related Stories