A little Golang way

https://www.aerofs.com/blog/a-little-golang-way/

by vonsnowman • 11 years ago

201 186 11 years ago

ploxiln• 11 years ago

I'm all for replacing java bloat with a little bit of go, but it seems like the cause of the bloat becoming a real problem was hopping on the docker bandwagon, for a collection of services that always run together in an appliance and really benefited from being hosted in a shared JVM?

Isn't the JVM + servlet container thing supposed to be able to isolate the different services? They get their own bunch of threads, and their faults probably shouldn't crash other servlets?

markrages• 11 years ago

Most of our problems are self-inflicted.

eternalban2• 11 years ago

You should read Sun's J2EE specs. There is little semantic distinction between a stateless session bean and a container hosted micro-service. The specs were never fully grasped (imo) by the Java community and those who got it (per gossip I heard) -- appserver vendors e.g. IBM, JBoss, etc. -- effectively crippled the spec by resisting the completion of the APIs that would commoditize their containers.

Sun was really ahead of the game in various fronts.

timtadh• 11 years ago

So true. I had to read the entirety of the JSP spec (and a few others) when I was an intern to help the company do some static analysis things. I also read a good chunk of Tomcat. Sun had this grand vision of App servers which you could just push code to and it would run completely isolated. The app servers where supposed to be spec compliant meaning your app could run on any implementation. But, and this is what bit the company I was working for, all of the implementations add lots of non-standard bits and broke standard bits. This meant that if you had written your app to go on WebSphere there was essentially no way it was going to run on TomCat without modification.

If Sun had more tightly controlled the marketplace for these things I think the whole ecosystem would have been more robust.

Deinumite• 11 years ago

Traditionally you'd deploy a bunch of apps to one container (Tomcat, or whatever) and they'd all share the same JVM. I'm not sure if jars are shared in this way or not. I think OSGI was supposed to allow deploy apps to reuse the same jars and resources but I don't know how commonly that is deployed.

Now it is more common to run Tomcat or Jetty embedded for your app so that they are isolated.

colordrops• 11 years ago

Wow that pun in the title is painful.

sumobob• 11 years ago

Theres a pun in the title?

axyjo• 11 years ago

It's from a saying... a little goes a long way.

callum85• 11 years ago

Enjoyably so.

vonsnowman7• 11 years ago

Hehe, yes, I'm afraid I have a proclivity for terrible puns. I do think they're somewhat less cringe-worthy that the link-bait titles that are all the rage these days (or at least endearingly cringeworthy).

sz4kerto2• 11 years ago

"Code size was reduced by almost half, from 175 lines down to 96."

Hm, I how can we take a 175 LOC project as something relevant in any way?

mkozlows2• 11 years ago

The relevance is that even a 175 LOC project in Java takes up enormous memory and disk footprint -- that the JVM is this super-heavyweight thing that's really just inherently inappropriate for a lot of applications.

xienze3• 11 years ago

The relevance is that even a 175 LOC project in Java takes up enormous memory and disk footprint

_This particular_ 175 LOC Java project takes up an enormous amount of memory. For all you know it was just coded poorly and the Go one is a bit more reasonable.

the JVM is this super-heavyweight thing that's really just inherently inappropriate for a lot of applications.

The JVM introduces overhead but not so much that you can make these sorts of extrapolations.

microtonal• 11 years ago

Until you have enough microservices and each microservice carries its own Go runtime. Then having one JVM suddenly isn't so bad. The problem (as someone else said) is that using Docker may not be the right solution here.

There are enough solutions (e.g. servlet containers) where you can run multiple services in one VM, with isolation, and security hardening (using security manager).

With regards to memory footprint - the difference here is that Java uses a minimum and maximum heap size that can be tuned with parameters. This has downsides (typically more memory use) and upsides (upper bound on the maximum memory use of a process).

sz4kerto2• 11 years ago

Yes, it's larger than a C or Go process. Does it usually matter? No.

The whole Netflix is based on the JVM and Java. They are enormously big, they have high CPU and I/O requirements in many cases, and yet, they manage well on Java.

I'm not a big fan of Java as language (to say the least), but the JVM as a runtime is very-very sophisticated.

benashford• 11 years ago

The disk footprint was measuring the size of the docker container, who knows what was in it. Something would have to be wrong for 175 lines to require 650MB of dependencies... it could if it were loading all of Spring/Hibernate/etc., but then I'd challenge it wasn't a "microservice" at all.

nine_k2• 11 years ago

The idea is that a server this small should probably not consume oodles of memory. They reduced RAM footprint by 100x. Apparently, their Java implementation of this toy service was wildly out of hand.

I wish they provided the JVM startup info and stats to compare. Maybe JVM's resource usage could be shrunk, too.

xienze3• 11 years ago

The idea is that a server this small should probably not consume oodles of memory.

The number of LOC has no bearing on how much memory something will consume. For all you know the service could just be poorly implemented.

GauntletWizard• 11 years ago

As much as I am a fan of go, this sounds like it's a bad build infrastructure more than any fault of Java. JAR shading exists, their imports were probably out of hand (and significantly unused), and while they were microservices, they probably could have had groups of them share a JVM (and benefited from the shared permgen space).

Still, that's time and effort, and in Go you get all of those things for free. There's far fewer janky edges (I spent hours figuring out the problems with signed jars, shaded jars, and symlinks when trying to deploy)

vonsnowman7• 11 years ago

As mentioned in the blog post, this was a hackathon project originally, hence the motivation to start with a very small server.

The other servers that went through a rewrite also ended up being significantly smaller in go but that's a story for another day.

KirinDave5• 11 years ago

I'm not sure what you'd expect a microservice with a straightforward bit of logic and a single HTTP route to actually cost in terms of lines?

I suspect the same (or greater) linecount gains could have been gotten by using a language even more pithy. How much you wanna bet the equivalent Clojure or Scala versio would be half again as many lines?

jfb• 11 years ago

How much you wanna bet the equivalent Clojure or Scala version would be half again as many lines?

263 lines?

pbreit• 11 years ago

It's merely a positive (to most) data point. Do with it what you wish. Dismissing it outright seems impruduent.

e12e2• 11 years ago

Of course, neither program was actually 175 or 96 LOC -- they included thousands of lines of code from the standard library. Which is why it's possible to "do useful things" in ~200 LOC either in java or go (or python or...).

But just because you wrote just 200 LOC, that doesn't free you from bugs and regressions in the remaning thousands of lines of code you rely on.

As an anecdote, at a previous work place we had a program that parsed CVS-files for import into an SQL database. This particular program was written by a researcher (meaning: someone with great domain knowledge, but perhaps not a very strong software engineer) in Visual Basic.

The program worked great, but we had to upgrade the server that ran it to newer versions of windows, and at some point needed to recompile the code (I don't thing this was releated to a bugfix, IIRC it was simply a run-time/linking issue).

As it turned out, the code compiled, but the program didn't work -- MS had changed the API for CVS handling (probably fixing a bug). I don't recall exactly what it was, might have been how floating numbers were parsed/handled, possibly an edge case with Norwegian/English localization or something.

Net result, we had a pretty though debugging job on our hands, for a very small program...

(I feel a bit bad for singling out MS for this -- as I understand it, they are generally very good about maintaining backwards compatibility -- even to the extent that that becomes a problem. I guess we just hit on a corner case with our use-case and this particular old VB code).

jsnk• 11 years ago

More and more I hear about Go, I feel more convinced that it would be worth giving it a shot.

Can you share what the microservices are doing? What are they for?

vonsnowman7• 11 years ago

At this point we have 6 microservices written in go in the appliance:

  - team-server probe: already mentioned in the blog post.
    Determines if any installed Team Server is down.

  - ca: as mentioned in the blog post, as simple Certificate Authority

  - charlie: a checkin service. Desktop clients periodically post to it
    to signify they are up. This data is used in each user's device list
    to show if the device is up and which ip it was last seen from.

  - auditor: takes audit event in an HTTP endpoint and forwards
    them to a raw TCP connection as expected by splunk and co

  - valkyrie: a relay server used for data transfers when desktop
    clients cannot establish direct TCP connection (more about that
    in a future blog post)

  - lipwig: a messaging/pubsub server used for peer discovery
    and notifications (more about that in a future blog post)

aikah4• 11 years ago

I'll bite.

While it is absolutely true Go servers use less memory that classic servlet deployments, The question is what were you using at first place? where you using a big framework? with this or that big IoC container ? with a bloated ORM ? ... or where you using barebone jdbc and writing servlets without any framework? because essentially that's what you're doing with Go, Go has 0 big framework(and no Beego or Revel are not complex), 0 big ORM , 0 IoC container ... And while the Java culture is about heavy decoupling and reusability , the Go code culture is basically about writing correct code without thinking about re-usability at all, because often you just can't. So I'm curious. My point is didn't you reduce memory usage because Go forced you to write code a certain way ?

jgalt212• 11 years ago

the Go code culture is basically about writing correct code without thinking about re-usability at all, because often you just can't. So I'm curious.

That's a very strong statement. Do you have evidence to support this? I don't code in Go right now, but have been keeping an eye on it as it continues to gather momentum.

dcsommer2• 11 years ago

Lack of generics can make it harder to write cleanly reusable modules in Go. I've noticed gophers are often less allergic to copy pasting code with tweaks (as long as it isn't too many lines) and don't mind rewriting similar code if it is obviously correct. It was a bit of a culture shock for me at least.

Zikes• 11 years ago

The parent comment is incorrect. Orthogonality is a really big deal in Go, and is highly stressed in the culture. Libraries are strongly encouraged to present methods that work with interfaces that are defined by the standard library to ensure the libraries are composable.

If your library works with streams of bytes, its methods had better use io.Reader and io.Writer. If you're writing an ORM, it had better be built on database/sql.

zer00eyz• 11 years ago

I'll jump on this, but I'm going to change it up a bit.

the Go code culture is basically about writing correct code without thinking about USABILITY at all

The reality is go is YOUNG and it shows. Lets take two good examples that are part of what is going on in golang today.

Vendoring: The core teams approach, and what the community are doing have diverged pretty rapidly. Recently there have been some efforts bring vendoring into alignment, but what is being used isn't the best approach/solution, its good but on the whole could be better.

http: The core library is great if you want something small, and its fairly easy to put together a tool chain that will remain "idiomatic". However if your going to go out and build something "large" then you have an interesting issue, because OOTB core http lacks any concept of context. There are some interesting tools and frameworks out there to make up for this (gorilla/context, codegangsta/negroni) but if you adopt them your no longer "idiomatic"... your libraries are now less reusable because they are tightly coupled. It looks like a new method signature is going to be required in the http package, one that uses golang.org/x/net/context and returns errors so we can have a sane http stack OOTB in go....

Log vs syslog: Logging in go is fairly messy right now... there are a bunch of packages that try to make up for this, but really better logging has to be built into the core. Syslog has levels/features, log is just dry and basic (maybe too much so). The core not only needs a unified solution, but one that is going to be context aware.

Will these get fixed? Probably! The core team isn't deaf. However if they don't start moving on a path to 2.0 soon and address some of the real issues, I fear that go will end up in the same boat that python did in 2.x vs 3.x.

P.S. in spite of all this I have been writing a LOT of go lately and enjoying it, but it really does need to grow and soon!

KirinDave5• 11 years ago

The problem got exacerbated when the container-ized deployment stopped letting the OS more efficiently serve shared memory. There are lots of tricks and wisdom built into platforms like Tomcat to reduce memory use, too. They're just confused when they're alone in what appears to be their own universe.

This is why there's such a strong pressure to move to more svelte language implementations and "microservices" as you start to see containerization take off.

zik• 11 years ago

the Go code culture is basically about writing correct code without thinking about re-usability at all, because often you just can't.

What a ridiculous statement. Go encourages elegant and simple interfaces to make code reuse easier. The tooling provides an incredibly easy way of sharing code. The "culture" is all about sharing code. Go's much better at facilitating code re-use than (say) Java, both at a language level and at a cultural level.

threeseed5• 11 years ago

This makes absolutely no sense what so ever.

The JVM has the largest library collection of any platform (currently 1,026,516 libraries just in Maven Central repository). Sure a lot of is to do with the age of the platform but this idea that Java is somehow impeding code reuse due to language/cultural reasons is ridiculous.

One of the biggest complaints for people coming to Java is actually just how many transitive dependencies libraries are brought in.

melling2• 11 years ago

I've got a little Swift Language "search engine" that I built with Go on App Engine. I haven't crawled any sites (yet). I simply created a Go data structure to do in memory searches because I didn't feel like using Google's data store.

Here's the site: http://www.h4labs.com/dev/ios/swift.html

Here's the data: https://github.com/melling/SwiftResources/blob/master/swift_...

I'm approaching 1400 URL's and it's still snappy. I was hoping Go's claimed "efficiency" would keep freely hosted a bit longer than Python.

akhilcacharya• 11 years ago

I'm approaching 1400 URL's and it's still snappy. I was hoping Go's claimed "efficiency" would keep freely hosted a bit longer than Python.

That is exactly why I picked my Go for my latest side project.

I just hope the framework I'm using isn't the bottleneck, because I really hate writing pure Go servers.

stesch2• 11 years ago

How tied is Go into Google? What if Google drops Go?

shocks• 11 years ago

Go is a programming language born out of Google for sure, but that does not mean it depends on Google. It's a community project after all.

aikah4• 11 years ago

It's a community project after all.

It is absolutely not a community project. Go governance is a 100% @ Google . There is no go committee or go working group outside Google. It's backed by one company that has full control over it. Sure it's open-source, but good luck with a fork. How can anybody be so misleading about that fact ? What did make you think Go is a "community project" ?

but that does not mean it depends on Google.

There is a top down , vertical relationship between the Go team at Google and the rest of Go users, Go main goal is to fulfill the Go team needs, period. If you find it useful then it's a bonus. That's exactly how the Go team speaks and act, in fact the Go team makes it really hard to contribute to the core.

Please, stop saying what you say, it is completely untrue and a total mis-characterization of how the Go project work.

I still want to know what made you think Go is a "community project".

patio11• 11 years ago

You're probably being downvoted for off-topicness, FYI, since this comment is equally germane to any mention of Go. That said, I feel it is a comment in good faith and deserves an answer.

Question #1: How invested should you model Google as being in Go's future? Answer: Extraordinarily invested. They have a whole lot of code written in it, most of which the world will never hear about, and which powers services that they will continue running until the sun goes nova. Google will likely continue maintaining and extending Go programs (as well as C++, Java, and Python) for the foreseeable future.

Question #2: Can Go survive without Google's corporate backing? Answer: Go is an OSS project. Like many OSS projects, it receives a substantial amount of support from corporate interests. Go is not uniquely a Google priority -- many organizations like having a systems programming language which has its feature set. In the absence of meaningful support from Google, Go code existing as of August 2015 would continue to function. Further versions of the language/toolchain/etc would have substantial question marks about them, but the community feels large enough that this would be a Significant Event rather than a death knell.

Does that help?

stesch2• 11 years ago

I'm not sure if this will help me. I guess I'm in a programmer midlife crisis. Every new thing is potentially a waste of time because you can't predict if it is still around in 2 years. And with Google you know that they invest millions of dollars in projects and then abandon them.

Would I be better off maintaining decade old COBOL projects? Sometimes I think so. Other times I'm glad I can chose the technology I want to solve the problems at work.

August 2015. 31 years since my first computer.

KirinDave5• 11 years ago

If you're concerned about Go, you might also find better results with Rust or Nim. Both are up-and-coming in similar spaces to Go.

pcwalton13• 11 years ago

Go isn't going away anytime soon. Neither is Rust. You shouldn't choose Rust over Go because of fear of the language being abandoned.

(I'm not too familiar with Nim's community so I can't speak to it.)

travjones• 11 years ago

There have been several blog posts and conference presentations with a similar theme -- "Go is efficient, especially compared to X."

I know it seems like hype, but I encourage others to try Golang out, maybe even slap a web app together with it. What you'll find is that your Go app will be extremely efficient and performant. It's kind of unfair to compare Go to many of the other languages of the web because it is compiled, which is a huge reason why it is so performant. So rather than compare Go to other languages, I encourage you all to give it a try. If you already program, working on a small Go side project will get you up to speed quickly and you'll learn about some of the awesome packages individuals from the Go community have put together for us.

Disclaimer: I do not work for Google. I write Go code and enjoy it. I think others will too.

eropple7• 11 years ago

What you'll find is that your Go app will be extremely efficient and performant.

This is true, for sure. But my experience--and I've written plenty of Go, though I find it unpleasant to write and think about for all the usual reasons that Go partisans roll their eyes and so would rather deal with it as artifacts rather than actually writing it--has led to watching lots of developers make mudball codebases in the process (while blogging very heavily about how great it is three weeks in, and less a year-plus in). A pursuit of faux-simplicity has led to what I see as reinventing a lot of Java-1.4-era (because the language itself is essentially that) design patterns--and we should be reminded that design patterns exist to address defects in tooling--that more expressive languages have managed to avoid; the general desire for smaller applications helps to some extent, but software has this tendency to grow that I don't think Go gives you the tools to effectively manage and maintain. Reasonable minds can differ, of course.

Past that, while I think it's an alright choice for company-internal deployable products--web servers, worker nodes, etc.--I have straight-up problems with it being the new devops tool of choice. Statically linking your SSL library makes you an asshole when it inevitably fails and now an application has to be regression-tested so that new features and new bugs don't hose you just because that's the only way to upgrade Heartbleed 2.0. (Go also discourages program extensibility through components and rather as recompilation; the stuff Packer and Terraform do to provide "plugins" that the same company's Vagrant just did with a `require` is gross and, to my mind, completely foolish.)

So I wouldn't say it's hype, but I would say it's not all true, either.

Conlectus• 11 years ago

Go also discourages program extensibility through components

Can you elaborate on this? Because in my experience with Go, I found that I had to make _far_ more components (assuming this means libraries?) than other languages.

Statically linking your SSL library makes you an asshole when it inevitably fails and now an application has to be regression-tested so that new features and new bugs don't hose you just because that's the only way to upgrade Heartbleed 2.0

Not sure I understand that last bit. I get that statically linking could be bad(ish) because you would need to know to recompile with a fixed library, but what was that about new features?

chimeracoder3• 11 years ago

design patterns--and we should be reminded that design patterns exist to address defects in tooling

This is a really important point, and I think it's why a number of best-practices in Go are actually not best practices in other languages, and vice versa.

One of the explicit, top-level design goals of Go was to focus on creating top-notch tooling as part of the language. While it is not the only language that has tried to do this from the get-go, it's one of a very small number[0].

Because the tooling was a first-class design goal, a number of the problems that traditional design patterns were created to address are less problematic in Go code[1].

[0] Case-in-point: gofmt, which other languages are now adopting due to its success.

[1] Again, gofmt: there are a number of design patterns and style bikesheds around code format, but really, the most important thing is that there exist a uniform standard. gofmt provides that reliably and a way to enforce that as a pre-commit hook, which has done wonders for eliminating minor style variations that IMHO cause more problems than they solve.

staunch• 11 years ago

Statically linking your SSL library makes you an asshole...Heartbleed 2.0.

Go has its own SSL library, crypto/tls, which is not linked to any C libraries and wasn't affected by Heartbleed 1.0. You haven't written much Go if you don't know that.

The argument is specious anyway, there's nothing difficult about building a binary from an old version of your code or just upgrading in most cases. Deploying a new Go binary is always trivial as compared with upgrading, testing, and deploying Python, Ruby, or Java applications with their associated libraries and interpreters.

grey-area3• 11 years ago

reinventing a lot of Java-1.4-era (because the language itself is essentially that) design patterns

Really? Java 1.4 had an extensive and consistent standard library, implicit interfaces, composition instead of inheritance, static builds, and built-in concurrency?

Go is not early Java, it's more like C 2.0, and I don't really see anything faux about the simplicity, it really is pretty simple, perhaps too simple for some tastes, but it's not pretending to be simple, nor is it simplistic. Which specific design patterns did you have in mind?

mratzloff• 11 years ago

But my experience--and I've written plenty of Go, though I find it unpleasant to write and think about for all the usual reasons that Go partisans roll their eyes and so would rather deal with it as artifacts rather than actually writing it--has led to watching lots of developers make mudball codebases in the process (while blogging very heavily about how great it is three weeks in, and less a year-plus in).

I get what you're saying, and I agree with some of the criticism around the tooling, although I do think much of that owes to the age of the language. There are some specific things in the (sometimes rather cranky) replies to you that I think are incorrect but I'm not going to debate the merits of the language. Use it if you like, don't use it if you don't.

I will say that we've been using Go for nearly all back end services for nearly 3 years now, and we all still think it's pretty great. Our code base has also remained pretty clean, and in fact we're investing more heavily in Go going forward.

nickpsecurity5• 11 years ago

It's actually more like a combination of ALGOL, Oberon, and practical solutions to problems. It's supposed to be simple, effective, easy to compile, and efficient at run-time. Like Wirth's languages. But with extras to help it go mass market, esp standard library. So, although I often poke that it's not-novel, I at least give credit that designers chose wisely in what to copy for their simple-to-use language.

Pretty much the opposite of the mess that was and is Java.

omginternets4• 11 years ago

I think Go invites comparison to dynamically typed languages because it's compiler is so damn fast.

Yes, Go is technically compiled, but the development cycle is closer to that of dynamic languages. In a similar vein, I have a lot of trouble taking criticism of Go's type system seriously. Is it as robust as Rust's? Probably not (I've never used Rust), but that's way beside the point. Go's type system gives you a great deal of benefits while keeping the language very dynamic-y.

tl;dr: haters gonna hate. I like Go very much for certain things.

swah2• 11 years ago

Me too. Some folks prefer to think instead of messing with the code and retesting. I hate it.

nickpsecurity5• 11 years ago

That's definitely an advantage. People used to rag on me because I used a BASIC dialect for most of my development. One of main reasons: tens to hundreds of thousands of lines of code compiled and linked effortlessly. I later moved to LISP with its incremental, function-by-function compilation to get pause times down to tenth of a second.

Either way, the elimination of long compile cycles and other interruptions maintains the mental flow of the programmer. This results in a significant productivity boost. Also, the job feels better as interruptions and backtracking can be stressful.

eternalban2• 11 years ago

I prototyped a cache in Go and the ref. impl. in C. I'm a long time Go coder but did not expect the Go prototype to run so damn close to the C (and for certain ops actually faster than C).

The runtime is shaping up quite nicely. Given dynamic linking and this language can pretty much pull its weight in the backend for almost anything.

vardump2• 11 years ago

I wrote a small/medium size, non-trivial tcp-server in golang two years ago, in just about 2-3 weeks. It pretty much worked first try! So far 2 minor bugs.

Originally it was meant to be just a quick kick-golang-tires prototype, to be replaced with C++ later, but it turned out to work really well.

My impression of golang is that it's very useful getting things done. The code has also been readable to other people pretty much immediately, and they've been able to contribute features into it. None of us used golang previously.

Had I written same in C++, I bet there'd been a lot more bugs to fix.

vorg2• 11 years ago

slap a web app together with it

Web apps are on the decline, being gradually replaced by Android/iOS apps. When people can use Go to slap up a performant Android app, it may find more use.

StavrosK• 11 years ago

What are these mobile apps that don't have Web backends?

mkawia• 11 years ago

So slap an http api ?

Building native apps is hard unless you are specialized , but puking up some html and json is easy ,hence building a blog is usually good for introduction.

coldtea• 11 years ago

Nobody cares about Android apps. And they ain't gonna make you rich. Neither will iOS apps for that matter. That's a market raced to the bottom, that only makes sense for a few top players.

The web is not going away anytime soon.

duaneb2• 11 years ago

Go is only a massive improvement over interpreted languages. Anyone who was already writing performance software will have a much harder time justifying the switch: many low and high level features are simply not in the language. (E.g. parametric types, macros, a runtime using a c-like stack, avoiding the GC).

lpsz• 11 years ago

Anecdotes like this makes me wonder how much funding money and electricity could be saved if people migrated en masse from Ruby/Python/Something else to Go.

(Edit: not meant to be a political statement, more of a practical observation. I only recently started using Go.)

tracker12• 11 years ago

Also, the more complex a system is, the less the runtime overhead is by comparison. Hello world examples in .Net, node.js etc are 35-20% the memory overhead compared to Java.

It's not like golang doesn't have some overhead of its' own... There's also Rust, and D to consider.

jacquesm• 11 years ago

That may be one of the bigger chances for go: mobile. After all being more efficient on mobile translates into longer battery life and go potentially has a huge advantage over the current java (ok, not java) based environment on Android. For google this would be a triple win, get out of the Oracle mess entirely, give their mobile developers a super fast toolchain and give the end users better battery life.

swah2• 11 years ago

I love Golang, but its ain't very expressive, so it cannot be a good match for writing UIs. And Swift is much closer to Rust than to Go, IIUC...

melling2• 11 years ago

These guys went from 30 servers to 2 by switching to Go.

http://www.iron.io/blog/2013/03/how-we-went-from-30-servers-...

dougbarrett• 11 years ago

the best part was, they had 2 only for redundancy, they could have gotten away with only having one.

jaytaylor• 11 years ago

Can anyone shed some light on how/why running the same Java apps in a docker container significantly increased the memory footprint?

Is JVM overhead shared when multiple Java apps are being run on the same machine?

lobster_johnson• 11 years ago

Assuming this is referrring to the fact that separate containers will link in separate copies of all the binaries (executable + shared libraries), instead of sharing the pages across all instances of he JVM. There's no way for the kernel to know that they're all the same files. So a lot of code is duplicated.

haldean• 11 years ago

A lot of it is, yes. The JVM loads a large number of largeish class files when it starts; these contain implementations of the standard library and whatever else is on your classpath that you've imported. These have the nice property that they're read-only, though, so multiple JVM processes can safely share them. When you move to sandboxing each JVM off by itself, you lose the ability for them to share memory (which is, in a sense, a _feature_ of sandboxing), so now each of them has to take the 50-100MB hit of those formerly-shared memory regions.

(Note that the huge size of the classes is also the big reason why JVM startup time is so crap; another reason that multitenant JVM systems are great is that every process after the first starts much faster)

_JamesA_• 11 years ago

Java's main feature is compile once - run (almost) anywhere. From embedded to mainframe as long as a JVM is available.

How does Go stack up for cross-platform development? Does every application and library have to be (re)compiled for the target platform?

What about support for alternative architectures (ARM, PowerPC, etc)?

aikah4• 11 years ago

AFAIK Go support cross-compilation for multiple targets... as long as one isn't linking C libraries.

xienze3• 11 years ago

Go does require recompilation for every platform, but I'll give them this, it's very painless. Apparently in 1.5 you can just set an environment variable for your architecture and one for your OS and you're good to go.

findjashua• 11 years ago

the only thing that's preventing me from jumping on the Go bandwagon is lack of a nice collections library (a la lodash).

politician2• 11 years ago

That's due to a lack of generics. There are some workarounds like gonerics [1] -- a clever abuse of import declarations.

[1] http://bouk.co/blog/idiomatic-generics-in-go/

davexunit3• 11 years ago

And the cycle continues, from one crappy enterprise language to the next. I don't who said it, but whoever said "Go is a bold step backwards" is spot on.

hwh• 11 years ago

There are times when a step back is the mandatory step to get a new perspective on things. Just sayin'.

davexunit3• 11 years ago

But this is a step back to the cold, dead, static systems of UNIX. How about stepping back and taking a look at the dynamic, hackable environments of Lisp?

umhan35• 11 years ago

I hope that there is a compiled version of Scala.

kentonv2• 11 years ago

Resident memory usage dropped from 87MB down to a mere 3MB, a 29x reduction!

This isn't so much Java vs. Go as it is JIT/interpreted vs. AOT-compiled. The numbers are entirely typical across a wide range of such comparisons.

With an interpreter or JIT, you need to load all your code at startup and process it. Generally, _all_ of your dependencies need to be loaded and parsed upfront, either converted to some internal in-memory representation or JIT'd directly to machine code. This will allocate a bunch of heap structures during the processing, and the end result is a bunch of data that needs to stay resident and can't easily be shared with other processes.

With AOT, you mmap() in a file. The OS only pages in the code you execute, and can page it back out as needed. Pages are shared between all processes running the same executable.

At sandstorm.io our rule of thumb is that an app written in Node, Ruby, Python, PHP, etc. will take 100MB of RAM while an app written in C++, Rust, or Go will take 2MB. Since Sandstorm runs per-user (and even per-document) app instances, this is a pretty big deal.

The good news is that https://github.com/google/snappy-start should fix this problem: by checkpointing the process after it finishes its parsing/JITing but before it starts handling requests, we can get an mmap-able starting state that is very much like an AOT-compiled binary. At least, in theory -- there's still a bunch of work to do for this to actually work in practice.

The resulting docker image shrunk from 668MB to 4.3MB

While I would expect the Go image to be smaller (since Go builds static binaries, so literally all you need in the image is the binary), I suspect that the 668MB Java image was at least 90% unnecessary garbage that was not actually needed at runtime. Unfortunately the package managers we all use are not optimized for containers; instead they evolved targeting systems with dedicated disks that can easily absorb gigabytes of bloat.

In Debian, for example, every package implicitly depends on coreutils. That's perfectly reasonable when installing an OS on a machine: you almost certainly need a working shell to boot and administer your machine. But a container can get by just fine without coreutils, and a typical web server probably (hopefully) doesn't need to call out to a shell. Even if a shell is needed, busybox/toybox is probably sufficient and will take a lot less space.

Packages also often contain things like documentation, unit tests, etc. which obviously aren't needed in a container.

For Sandstorm.io we deal with this problem by running the app in a mode where we trace all the files it actually uses, and then we build a package containing only those. It mostly works and manages to keep packages reasonably-sized, but it does lead to bugs of the form: "I forgot to test this feature while tracing, so the assets it requires didn't make it into the package." We're looking for better options.

vezzy-fnord2• 11 years ago

Why snappy-start as opposed to any other, far more sophisticated checkpointing mechanism like CRIU or DMTCP? The idea is ancient.

kentonv2• 11 years ago

CRIU actually doesn't solve the right problem.

We need to run the app up until the point when it diverges -- i.e. when it first observes input that will be different across different runs of the app. For that, we need to be watching the syscalls and evaluating each one for potential divergence. As long as we are doing that, we might as well at the same record a log of those syscalls which we can replay later. Then once a divergent syscall happens, we dump the state of memory. Later, we can restore the memory and replay the syscalls to reproduce an identical starting process.

CRIU has no concept of divergence. CRIU takes an already-running process with arbitrary state and snapshots it whole.

CRIU's problem is actually orders of magnitude more complicated than snappy-start's: it needs to understand every possible file descriptor type that the process could have open, every aspect of process state, etc. snappy-start only needs to understand the specific syscalls that we care to implement; it can simply consider any call it doesn't recognize as divergent, and stop there. Adding support for more syscalls is then merely an optimization.

CRIU also requires special kernel features to support, which means more attack surface. Sandstorm wants to block everything except the most common kernel APIs for security reasons. snappy-start requires no new kernel features; it uses the well-understood APIs debuggers use, and we know we can still prohibit apps themselves from using those APIs.

Meanwhile, CRIU is much harder to customize. How would we decide when to do the snapshot? We'd have to re-implement much of snappy-start just for that purpose. And how do we teach CRIU about the specific assumptions that are safe and useful to make given our particular environment?

None of this is to say that CRIU is bad -- it's actually pretty amazing. But it's not the best fit for this specific problem.

vonsnowman7• 11 years ago

This isn't so much Java vs. Go as it is JIT/interpreted vs. AOT-compiled. The numbers are entirely typical across a wide range of such comparisons. [...] while an app written in C++, Rust, or Go will take 2MB. Agreed. As mentioned in the blog post, I considered Rust but decided against it because of I found it much less mature than Go. I did not consider C++ because, as mentioned in the blog post, part of the point was to experiment with new language/tools and even though I consider myself proficient with it, I learned the hard way that the lack of memory safety is rarely worth it.
I suspect that the 668MB Java image was at least 90% unnecessary garbage that was not actually needed at runtime. Unfortunately the package managers we all use are not optimized for containers Exactly. That was part of the point of this blog post, which I may not have been successful at getting across. Switching to go was, if not the path of least resistance to solve this issue, at least one of a few relatively easy routes. It also happened to be a great deal of fun.

mwcampbell• 11 years ago

I assume you didn't use the normal Dockerfile-based build system to build your super-small Docker images for the Go-based services. So how did you do it?

s73v3r2• 11 years ago

You know, as fun as it sounds like this was, wouldn't the natural solution to the problem be to roll back the Docker rollout?

skarap2• 11 years ago

I guess nobody will be saying that Java is lightweight, but the actual results of porting from Java to Go will vary wildly from usecase to usecase.

Both examples mentioned in the article (the 175LoC program and the CA) sound like very simple programs. E.g. I once wrote a C program which watched some directories with inotify and compressed new files using zlib. The memory footprint was 350KB. Obviously an empty JVM alone would use 100x more RAM. This static ~30MB overhead might be important in some cases and not relevant at all in others. The incremental (per-object) overhead is probably more relevant to almost all real-world usecases which are a bit more complex then the ones mentioned above.

Also - care should be taken not to compare apples (no TM) to oranges: e.g. if you use a huge ORM in Java (which among other things also caches results of each query) and then do a simple SQL query in the new implementation, would be strange to expect those two to perform similar. This happens quite often with rewrites - they almost always aim for simplicity, do short cuts, get rid of "unused stuff". Basically a rewrite from Java to Java will also usually improve performance.

Must-read about rewrites: http://www.joelonsoftware.com/articles/fog0000000069.html . Though, I guess, everybody has already read it.

carloscarnero• 11 years ago

At my organization we have several Java-based services (some of them can even qualify as a form of microservices, if you squint your eyes). We have found that when you have very good developers and you write almost to the letter of the spec, Java can easily provide a stable base (which is probably true of any language/runtime/library).

However, we've been eager to try Go in several places. Believe it or not, what has held us back is the lack of a solid LDAP library. We could/should scratch our own itch and be done with it, we lack the time... still so many things to do! In the mean time, for us, Java support for LDAP is nothing short of stellar; and has been for years.

bliti• 11 years ago

I've been writing Go for some time. Just finished writing a small CDN for the company I currently work for. It's It's a fast language that performs well in systems programming (what it was made for). But boy is it ugly. Not verbose like Java, but ugly to write and read. It does force a good coding convention because otherwise you end up with a pile of ugly code. People seem to be bent on writing it like 80s C. Full of single character variable names and odd function naming. Dunno if it's just my experience. I do like the fact that it resembles python in how it feels. Overall I'd say it's a nice language that is not for everybody. I am playing with Elixir these days. Go is good enougb, but I'm not happy with it being that.

mavelikara2• 11 years ago

I have a question on Go's performance that I'd like someone here, who has experience writing programs in it, to help shed some light upon.

1) Go does not have a runtime. This means that there is no JIT to do any optimizations based on runtime profiling.

2) Go is also designed to compile fast. Since it compiles fast, the compiler's time budget to do compile-time optimizations is small and it probably can't do the best job possible.

Are these two points accurate? If so, how does Go perform as well as it is claimed to do? Where is the catch?

flippant• 11 years ago

1) Go has a runtime that's included in the binary that you get when you run `go build`. It takes care of garbage collection, goroutines etc. It does not do any JIT compilation afaik.

2) Go compiles fast because it gives up a lot of modern features (namely generics).

anoother2• 11 years ago

Something troubles me about this article. I hope I'm misunderstanding it.

It's stems from the following quotes; knowing this information, why do AeroFS still think Docker is a good fit for their use-case?

However, after our move to Docker, we noticed a sharp increase of the appliance's memory footprint.
...
We identified several major factors behind these symptoms:
1. an increase in the number of running JVMs, as each tomcat servlet was placed into a separate container
2. reduced opportunity for the many JVMs to share read-only memory: the JVM itself, all the shared libraries it depends on, and of course the many JARs used by multiple services
3. memory isolation could in some cases confuse some sizing heuristics, which lead to larger caches being allocated by some services

ambrice• 11 years ago

Presumably Docker offers some benefit. It's weird that you and several others have jumped to the conclusion that it's obvious they should get rid of Docker. I would say: knowing this information, why do you still think java is a good fit for their use-case?

anoother2• 11 years ago

Because:

a) Their service was already written in it b) Their service was already performing adequately before they brought docker into play

But those aren't answers to your question. Java may have been a terrible fit for them, but their decision to move away from it was not motivated by that -- it was motivated by their already-written, already-performing Java code no longer performing as well, because of Docker.

The only question is, does Docker bring them enough benefits to justify reimplementing large amounts code, in a new and unfamiliar language? Maybe it does, but somehow I doubt it.

djhworld• 11 years ago

I find this article confusing, in the first part the writer talks about Tomcat and servlets, in the second part he talks about reducing the LOC from 175 lines to 96 by using Go.

To me it seems the Java solution was wildly over engineered and probably could have been refactored to not even need Tomcat.

themartorana• 11 years ago

I am not sure I understand where the issue is Java pure and simple, and where the issue is that Java services were impaired by being wrapped by Docker?

"Everything was fine until we switched to Docker" makes me think the trouble may not be all Java. Anyone have educated thoughts on this?

nickpsecurity5• 11 years ago

I appreciate you publishing this experiment as it will show Java's problems for what they are. However, a real test of Go's safety and efficiency would be a comparison to a similar language such as an optimized Wirth language, a subset of Free Pascal (Delphi-style), typed LISP (eg Racket), Julia, or Ada. People keep forgetting about the last one in safe, efficient, systems programming despite it doing for a long time what Go and Rust hope to do eventually. Its long-time use in embedded systems indicates it can be quite efficient.

Regardless, Go certainly improves on the reliability of applications vs C++ while being much more efficient than .NET or Java.

tapirl• 11 years ago

I always think Golang is a Java killer, but I think Java itself is more a Java killer than Golang.

A little Golang way

Related Stories