On Distributing Command Line Applications: Why I Switched From Ruby to Go

http://codegangsta.io/blog/2013/07/21/creating-cli-applications-in-go/

by reactor • 13 years ago

96 72 13 years ago

delinka• 13 years ago

He leaves out discussion of a crucial part of app distribution: dynamic libraries.

While you can't create and link with dynamic libraries in Go, you can call into C or C++ libraries. Go binaries are self-contained as long as all your code is Go. But if you're depending on a C or C++ library, you still have to worry about this aspect of distribution.

rwcarlsen• 13 years ago

It is now possible to statically compile+link C libraries into your Go binary. The sqlite3 package (http://godoc.org/code.google.com/p/go-sqlite/go1/sqlite3) is a good example of this.

archgoon• 13 years ago

Another aspect is that some opensource licensing schemes may require you to link dynamically. The LGPL, for example, requires you to do this unless you release the rest of your source code.

octo_t• 13 years ago

I'd have thought you could just statically link with the C and C++ libraries, creating a static binary?

edit: beaten by rwcarlsen

avtar• 13 years ago

After attempting the distribution of a command line application via Rubygems I came to the following conclusion: Any application not intended exculsively for Rubyists should not be distributed via Rubygems.

Another option could also be to convert gems (and dependency gems) to deb or rpm packages https://github.com/jordansissel/fpm/wiki/ConvertingGems

As far as I know, the Foreman project utilizes this approach http://deb.theforeman.org/pool/precise/stable/f/foreman/

codegangsta3• 13 years ago

That is definitely an option, and it is essentially what we did for our linux support when we were building out a cross platform toolchain for mobile games. The tricky part was supporting Windows, Linux, and Mac all at once. Again, not suprised that distribution was hard, there were just a couple things that made distribution in Ruby a more painful experience

izietto2• 13 years ago

Why don't compile Ruby using f.e. LLVM?

TylerE2• 13 years ago

Because it's kinda pointless. The thing that makes Ruby (very) slow isn't the runtime (there might be at most a 1.5x - 2x speedup there), it's the language design. All that dynamism and runtime reflection is really a performance killer.

gngeal3• 13 years ago

it's the language design. All that dynamism and runtime reflection is really a performance killer.

So why are Lisp and Smalltalk, which are equally dynamic and reflective, even faster?

ef4• 13 years ago

The article isn't about performance difference, it's about ease of distribution.

james_madison2• 13 years ago

Ruby without dynamism runs a bit faster: http://rubyluwak.com/pages/benchmarks

izietto2• 13 years ago

As the Topaz guy said, the slowness is due to the abuse of hash table lookups, allocation and copying (HTLAC) https://speakerdeck.com/alex/why-python-ruby-and-javascript-...

And the abuse of HTLAC patterns is due to the lack of non-HTLAC APIs; the syntax is trascurable

vidarh5• 13 years ago

The dynamism and runtime reflection can be a performance killer. It only is a performance killer in most cases because the Ruby runtimes are still extremely simple as dynamic language implementations go.

Method dispatch in Ruby, for example, can for the typical case (where nobody redefines random methods on all your classes on a regular basis once initial startup has been done) be little more expensive than a C++ virtual member function dispatch, but this requires work to either cache (and invalidate) method lookups, or pre-building vtables (and waste a bit of memory as well as being prepared to propagate updates to them if methods are redefined) for known methods.

Inlining of code is also possible, for example, but requires more work to add "guards" to protect against less common patterns that are still valid.

99% of Ruby is reasonably simple to optimise well. The problem is that the 1% can change the whole world for you and force you to either be able to selectively detect what changes are made or throw out all your optimisations), and so to get fast Ruby implementations will either need to ask for guarantees from the developer (pragma's to promise you'll never alter Fixnum, even in eval() blocks, or flags to by default freeze all standard classes, for example), or they'll need to start employing all the dirty tricks in the book.

Thankfully, the Smalltalk, Lisp, Self and Javascript worlds, amongst others, have pioneered plenty of dirty tricks that'll do just fine (e.g. consider Polymorphic Inline Caches, pioneered in Self, I believe, that are now quite widely applied for dynamic languages)

I think we'll eventually head towards a bit of both: Optional ways of making the job easier, such as promising not to much with things we normally don't muck with anyway, combined with new meta programming libraries to remove the reasons for a lot of the eval() usage, that will allow implementations to make much better judgements about what can actually change (instead of assuming all bets are off after an eval()), and a large number of dirty tricks (in the form of caches + heuristics + guards in various variations, to be able to take fast paths most of the time and "just" fall back to slow code for pathological cases).

_pmf_• 13 years ago

How many architectures are officially supported by Go?

codegangsta3• 13 years ago

Supported architectures listed here: http://golang.org/doc/install#requirements

dmm• 13 years ago

There are two implementations of go: gc and gccgo. The link above describes gc. gccgo is built on gcc and supports many more environments, at least in theory. gccgo will run on addtional processors like powerpc, mips, and sparc and other operating systems like solaris. These aren't necessarily well tested though.

MetaCosm• 13 years ago

Straight forward, easy builtin cross compiling is supported from<=>to: darwin/386 darwin/amd64 freebsd/386 freebsd/amd64 linux/386 linux/amd64 linux/arm windows/386 windows/amd64

But, once you start going down the path of other implementations of go (gccgo, gollvm, gollum, goios, etc) the number of platforms supported goes up, but not sure how well supposed / stable the massive list of things go has been ported to is...

kinofcain3• 13 years ago

This also makes go applications trivial to deploy. What used to be a massive amount of plumbing and bootstrapping for a ruby environment with multiple app server instances behind a reverse proxy with all the associated configuration and dependencies can now be little more than pushing a binary to a (nearly) stock machine instance. It really simplifies ops.

FooBarWidget5• 13 years ago

I really don't see what having multiple app server instances behind reverse proxy has anything to do with Ruby vs Go. App servers behind reverse proxy is an architectural decision. You can write a Go web app that must be put behind a reverse proxy, or you can write a Go web app that is supposed to act as its own web server. Likewise, your Ruby app does not have to run multiple app server instances behind a reverse proxy. You can run it together with your main web server (Apache or Nginx) and you can run a single instance instead. See https://www.phusionpassenger.com/

Of course, installing dependencies still matters. For Ruby, you can just use the Brightbox Debian/Ubuntu packages.

kinofcain3• 13 years ago

All the mainstream ruby web frameworks require something like nginx in front of multiple instances of something like thin. You can use something like phusion, unicorn or rainbows that do forking for you instead of launching multiple processes by hand, but you still have to configure them and they're still running multiple app instances.

Go's built-in http library, and all the associated frameworks, run multi-threaded out of the box, and are fast enough that you don't need to put nginx or haproxy in front of them.

I love ruby, I've been building and deploying rails apps since 1.0, the ease of deploying go apps is precisely the type of thing the Ruby/Rails community cares a lot about: making developers' lives easier.

codegangsta3• 13 years ago

Blog author here. Long story short I got frustrated with distributing Ruby cli apps and decided to switch to Go for speed and distribution.

I also created a nice cli library built on top of the Go flag package. I've been having a lot of fun in Go so far and encourage anyone who hasn't given it a shot to try it out.

https://github.com/codegangsta/cli

johnbellone• 13 years ago

Made the same decision as you recently.

The only pep-peeve of mine is that I'm not a huge fan of using JSON as a configuration mechanism. I much preferred having a Ruby file for configuration. My middle ground here is using a system ruby via /usr/bin/env ruby and generating the necessary JSON output for people that want the nice syntax.

rlpb5• 13 years ago

As a non-Rubyist, I share your pain of having to use Rubygems (and pypi, and CPAN, and all the others). However:

"Since Go is a compiled language, binaries of your app can be precompiled for each platform that you wish to distribute for."

For me, this is worse. There's no audit trail. No way for me to verify that the binary of your app is the same as the source you provide. So there's no way I'm running your binary on my system. I simply cannot trust it with my user account.

Not only is this bad from a security perspective, it's also bad for maintenance. Over time, many end up in a situation where others cannot rebuild binaries from source at all, since binary distribution becomes the norm and the source->binary mechanism isn't maintained except on the developer's own system. Look at the Java community for an example of this. This is a problem because it means that others cannot easy be a member of the community working on the source.

If you want to promote a community around the source (like Github does), you must also provide an easy way to rebuild binaries from the source.

Unfortunately, there's no good answer here.

Distributions like Debian have solved this problem - there are source packages, and there's a standard way of building them to get the binary packages. But there's quite a bit of arcane crud that's built up over the years, making it difficult to learn and understand Debian packaging. And the solution isn't cross platform.

Each language community has produced their own solution (Rubygems, Python eggs, CPAN, etc) but none of them work with the packaging provided by any other language. To me, this may be cross platform, but not being cross language is just as big a problem.

j_s• 13 years ago

Hacker News hosted the following discussion regarding problems that have already occurred maintaining Go software due to reliance on binary distribution: "Golang dependency hell in Haunts The Manse game" https://news.ycombinator.com/item?id=5796597

In this specific instance it was more of an incorrect understanding of Go dependencies and misuse/lack of source control, but it seemed to at least come close to an echo of the issues you are raising. The specific quote from the comments: "There is no version control for dependencies".

tptacek2• 13 years ago

That wasn't a binary distribution problem or anything like one, was it? That was as I remember it a development team that simply hadn't created a known-good repository of their third-party dependencies. While I grant you that's a common pitfall in the rubygem era, it's hard to feel much sympathy as someone who came up building commercial C code, where not having a one-command third-party build-world system was a cardinal sin.

pkulak• 13 years ago

Nothing is stopping you from building the app from source instead, just like every other open source project. But at least with Go you have the _option_ of just downloading and running a binary if you trust the author.

rlpb5• 13 years ago

This is true. I do have that option.

But I very rarely trust the author. Most of the time I've never heard of the author. This is a good thing, and exactly why Github has been so successful. By nature, if there's something obscure that I need doing, and I find some software or software module that lets me do it, then chances are I won't know the author.

This means that downloading and running the binary is suddenly a very obscure use case. It should be for you, too.

coldtea• 13 years ago

For me, this is worse. There's no audit trail. No way for me to verify that the binary of your app is the same as the source you provide. So there's no way I'm running your binary on my system. I simply cannot trust it with my user account.

It's not for you then.

For one, billions of people run tons of binary third party apps all day long.

Second, when you deploy to people, they have to trust you. Whether they trust your code is irrelevant. Most of them are non-programmers anyway and wouldn't even check if you gave them a source code printout.

_sabe_2• 13 years ago

I think todays programmers is doing everything back ways. First they think about what language they want to use, then what libraries do they need, and then finally on what platform will it run and how will it be deployed.

In the old days everything was much easier. First you chose a platform, depending on budget, support and so on. When you decided what platform to use, that then dictated what libraries you had at hands, and what package manager you needed for deployment.

That maybe sounds very limiting, but on the other hand you don't always have to come up with new solutions how to bundle and deploy stuff all the time...

FooBarWidget5• 13 years ago

Programmers "today"? Maybe I'm not old enough but I cannot remember a time when programmers would choose a platform based on their target userbase, unless they are forced to do so (e.g. iOS). Even back in 2000, given the choice, programmers would always choose to use their favorite language if they can. I remember that back in the days I was very much concerned about distribution. If I write an app in Java, users needed to download a 20 MB runtime (mind you, circa 2000 dialup was common). Visual Basic, Visual C++, etc all required some form of runtime. .NET was in its infancy and not widely distributed. I chose to write as much as possible in Delphi so that users only had 1 executable to worry about, even when Delphi wasn't the most productive platform. I asked questions on various developer forums, and I was being ridiculed for thinking so much about user distribution; the consensus from the community was: pick a language they like, and the user will just have to put up with whatever additional multi-megabyte runtime it requires.

_sabe_2• 13 years ago

Back in 2000? Good tools like Make where built in the 70s that's utilized by Autotools and other great build systems to bridge the gap between different platforms, so that you don't have to distribute as the author of the article we're discussing, a whole runtime with all dependencies.

riobard• 13 years ago

Not just command line apps. Any time you need to deploy on multiple machines, Go saves you a lot of trouble.

At work we use primarily `buildout` [1] to deploy Python apps with complex dependencies. It is rather slow and fragile. Additionally, PyPi and various external repositories are not very reliable, and we ended up setting up local mirrors. Nevertheless, the process is not very pleasant.

We are experimenting with Go in some smaller projects, and so far the experience is great! We can build on a development machine (OS X) with all the dependencies version controlled (using git-subtree) along with our code, and cross-compile a static binary for Linux for deployment. Deploying is simply copying the binary to the production machines (usual configuration management is done via Puppet).

Besides, Go can efficiently make use of multiple cores (remember to set GOMAXPROCS correctly), and we can just deploy one instance of an app per server, unlike one instance per core as in Python's case (I know there's multiprocessing, but it's bad for operation and management). This greatly reduces memory overhead if the app is big (e.g. consumes a couple hundred of MB upfront and you have 32 cores per machine = a few GB's memory is wasted).

Only problem is that Go's 3rd-party libs are not as comprehensive as Python's yet, and many packages we rely on do not have mature/reliable counterparts in Go. So we are still using Python for our core stuff, but smaller projects that do not require many external packages can be done in Go elegantly.

[1] http://www.buildout.org/en/latest/

earlz• 13 years ago

I've written about this before, but this guy definitely wrote it better. I experience a very close relation between this and Ruby vs C#(Mono). I love Ruby as a language, but the pain of using Rubygems and random things requiring Ruby <1.9 and <1.8 (And setting up rvm to manage that) is an absolute pain. Python is slightly better, but I still have a python2 and python3 installed.

This is why I love mono. Things may not be backwards compatible, but I've never had a problem where something wasn't forwards compatible. And I think this is a result of it being compiled. The compilation step decouples the language and syntax from the actual execution of the code. This means that compiled applications can have drastic and breaking syntax improvements, but still be forward compatible on future runtimes. The same can not be said of interpreted languages unless you somehow work a version number into the execution, and even then it makes implementation of the languages much harder because future versions must support every version of the syntax

pnathan• 13 years ago

In a prior job, I did a decent amount of Python distributions. I had it simple: 1-2 python libraries plugging into a single app which was distributed on a .deb server. Even that was not particularly fun. I can definitely understand the appeal of a build n run system.

As a user, I find rubygem based installs rather horrific (same for pypi & cpan).

n.b.: I'm sure there are tools out there that will appropriately massage your app and put together a app.zip package for you. But IME those sorts of tools are a lot of work to get going.

mwcampbell• 13 years ago

I agree that the self-contained nature of compiled Go programs is one of Go's most appealing characteristics. In fact, I think you were too diplomatic toward Ruby, and most other popular languages. In my opinion, to insist that distributing a program should not be easy is to expect too little of our tools. I can identify two main deficiencies that make it hard to distribute programs:

1. Lack of a module system that scales well to large programs while still using static linking. According to the talk "Go at Google: Language design in the service of software engineering" (http://talks.golang.org/2012/splash.article), this is one of the main problems that the Go developers aimed to solve.

2. Too much dynamism in the language, such that a packaging tool can't tell with certainty what a program needs, and just as important, what it doesn't need. Go is static enough that this simply doesn't apply. Dart is more static than JavaScript, but still allows some dynamism. The Google Closure tools solve the problem by subsetting JavaScript. Is there any such subset for Python or Ruby? (I don't count RPython, since AFAIK that's only intended for PyPy itself.)

So when choosing a language for a program that one expects to distribute to many users, I think it's a good idea to take these things into account. This may mean choosing a language that lacks some dynamic features that many of us find convenient. (Compile-time metaprogramming can compensate; too bad neither Go nor Dart has it.) Go looks like a pretty good language for servers and command-line utilities.

shurcooL2• 13 years ago

Related to cli and Go, if you ever want to quickly execute Go funcs from a terminal [1]:

    $ goe go/parser 'ParseExpr("5 + 7")'

[1] https://github.com/shurcooL/goe#examples

astrodust2• 13 years ago

What it needs is a `-e` flag:

    ruby -e 'puts 5 + 7'

shurcooL2• 13 years ago

Can you please elaborate on how that would help?

Is it just for familiarity/similarity? I kinda see the 'e' from ruby -e going after the go to form goe. goe -e would be just redundant, wouldn't it?

conroy2• 13 years ago

One feature I miss from distributing on RubyGems or PyPI is easy update functionality. Thankfully, go-update[1] looks to solve the problem by making it easy to create self-updating Go binaries.

[1] https://github.com/inconshreveable/go-update

stephth• 13 years ago

What is the DSL used in the Ruby example?

drcube• 13 years ago

What's wrong with distributing source + makefile?

On Distributing Command Line Applications: Why I Switched From Ruby to Go

Related Stories