Using Go to improve your Ruby application's performance

https://antoine.finkelstein.fr/go-in-ruby/

by vinnyglennon • 12 years ago

65 51 12 years ago

antoine.finkelstein.fr

obeattie• 12 years ago

    JSON.parse(`go run json.go`)

No.

cotillion• 12 years ago

   `exec #{Rails.root}/tmp/go_#{m} #{arguments}`

The escaping leaves a bit to be desired.

steventhedev• 12 years ago

More than just the escaping. You should also clean your environment, providing the child process with a bare minimum of information it needs to do its job.

Otherwise you leave yourself open to future security issues, especially if you keep API keys or DB passwords in ENV.

VeejayRampay2• 12 years ago

While I can sympathize with your critique, I wish it were a bit more elaborate so that the author has an idea about what should have been done better. What is bad about that? What are the alternative solutions, etc.

Svip• 12 years ago

I think s/he objects to the idea of asking Go to compile and then run the program each time, rather than running the resulting binary:

    JSON.parse(`./json`)

alayne• 12 years ago

Further down you can see he only does that in a development environment.

chuhnk• 12 years ago

Why would you do this?! Please tell me this is some form of elaborate trolling?

im_dario• 12 years ago

I guess that is a simple way to delegate heavy processing tasks to Go. Communication through stdout in JSON is an easy way to communicate between Ruby and anything else faster than Ruby (C, Rust, Nim, Go, etc.) in specific tasks.

oinksoft• 12 years ago

  > Ruby uses green threads (meaning only one CPU will get
  > used) and they are no easy equivalent to Go channels.

Restriction to a single core is a consequence of the GIL. Also Ruby uses native threads, not green threads, since 1.9.

The author's `execute` method is dangerous because it doesn't escape its arguments.

thinkingkong• 12 years ago

It might make sense to put some compute intensive tasks in Go, or any other compiled language thats more efficient, but exec'ing and returning JSON isnt really a great idea.

If you were going to consider this seriously, then you should have the target app start once, listen for comands on some channel like a socket, and have it return information. JSON is probably a little too verbose for this kind of use-case as well, but it depends how efficient Ruby would be at unpacking alternative serialization formats.

3pt14159• 12 years ago

Why couldn't the coder use Extconf and just include the Go program into its scope like C?

jamescun• 12 years ago

With regards to Go, it is currently not possible as the Go compiler only supports building static binary executables. For it to work with extconf, Go would need the ability to build dynamic libraries.

There has been talk of incorporating this ability and slim executables go (I think it might even be in the roadmap now).

vinceguidry• 12 years ago

It might make sense to put some compute intensive tasks in Go, or any other compiled language thats more efficient, but exec'ing and returning JSON isnt really a great idea.

It's not a terrible one, though. You don't choose Ruby for its performance. If you find yourself creating and returning lots of JSON with this, that should be a clue that you should be pushing more functionality down to that lower level, and just returning the bulk results back up.

pselbert• 12 years ago

Using a library like Oj[1] it is vastly faster to unpack JSON in Ruby than it is even to use the built in marshaling. It's extremely portable, performs well and is easy to debug. There really aren't many practical drawbacks to using it for transport on the same system.

[1] https://github.com/ohler55/oj

joshcrowder• 12 years ago

I agree there are some pretty big issues with this tutorial. However it's still a pretty good article the "ideas" here are useful, I started thinking about how I could use Go in my ruby stack.

microtonal• 12 years ago

How so? The idea is not novel, in fact he links to a proper implementation (having a Go process listen on a UNIX domain socket) and the only contribution is to introduce a huge security vulnerability

JonnieCache• 12 years ago

If you want to extract parts of your ruby app to go in a simple way, may I suggest a look at http://www.goworker.org instead of following this articles advice.

(It's basically a golang port of Resque::Worker)

_mikz• 12 years ago

For the ones that do not know, the escaping issues could be handled either by Shellwords.shellescape or passing them as individual arguments to IO.popen & friends (Kernel.system, Kernel.spawn ...)

mitkok• 12 years ago

And the point of this is what exactly ? The author advertises Goss "Easy and cheap concurrency", "Low memory overhead" and "Easy deployment", but the example does not show how Go will help in this particular case.

issaria2• 12 years ago

You could replace `Go` with `erlang`, `haskell`, `rust`, `nodejs`, `c`, `whatever language faster than ruby` and remain true here, amazing!

issaria2• 12 years ago

People, please don't upvote any post that has fancy word 'Go' in the title!

jerf• 12 years ago

This is true, but it's a point worth making every so often too. I created a system with Erlang and Perl that punched well above its weight (in terms of resources allocated to it) because I was able to cleanly separate the "concurrency" into the Erlang and the "reusing tons of existing code and working in a language the devs are familiar with" into the Perl. It's a powerful pattern, and if you are, like I was, stuck in a language with poor speed and in the case of perl no significant concurrency story [1], it's important to hear that there are options other than "throw it all out and start over again in a Cooler Language (TM)".

[1]: Perl has plenty of event loop choices, but in terms of trying to maintain thousands of live SSL connections at scale it's a terrible, terrible choice for that connection manager. But it was just fine as a service provider.

thescrewdriver• 12 years ago

Almost anything with "Go" in the title gets upvotes on HN.

briandear• 12 years ago

Similar to the way Bitcoin was HN upvote bait for a long time.

steveklabnik2• 12 years ago

While this is true, the strategies for each are different. For example, the ways you include Rust or C into a Ruby program can be much more varied than the others in your list. This article, for example, chooses a 'shell out' option. While you can still do that with C or Rust, you can also compile them as an extension directly to your Ruby. There's upsides and downsides to both approaches.

halostatue• 12 years ago

Has anyone written a Ruby extension in Rust yet? That would be very cool to see.

Zikes2• 12 years ago

If the title used any of those other languages, you likely would not have made that comment.

But it says Go, because the author used Go, but apparently some people are tired of hearing about Go.

dkarapetyan• 12 years ago

So instead of using a unix domain socket and amortizing the cost of startup he forks a process every time. This is ok for prototyping but definitely not ok for any kind of production use.

reledi• 12 years ago

I wish this article talked about some of the reasons you may not want to do this instead of just selling the strategy.

For example, this doesn't look appealing to me because it adds complexity and makes your codebase harder to work with.

It's a neat hack, but I think in reality there would be very few cases where this would be a good idea.

scosman2• 12 years ago

Anyone wanting to do this at scale should look at posix_spawn to avoid forking the process (ruby gem here: https://github.com/rtomayko/posix-spawn). Also probably ruby's ShellWords for escaping.

VeejayRampay2• 12 years ago

Can this be used as a replacement for backticks or system calls? Would it result in a lower memory cost?

scosman2• 12 years ago

A small savings for memory, but much faster. The link has benchmarks.

pjmlp2• 12 years ago

Others would use RubyMotion or similar.

evilduck2• 12 years ago

The article is about speeding up a Rails app. Unless you're planning on hosting that web server on OS X, iOS or an Android device, RubyMotion is not going to be of much use.

pjmlp2• 12 years ago

I said "or similar". Is RubyMotion the only Ruby compiler?

Personally I always favor solutions that don't require changing language.

As for RubyMotion, I thought it already supported any UNIX system.

andyl• 12 years ago

Another alternative using Rust...

http://blog.skylight.io/bending-the-curve-writing-safe-fast-...

kibwen• 12 years ago

It says something sad about the state of HN when a well-written blog post that is relevant to the topic gets downvoted merely for not being written in Go.

MoOmer• 12 years ago

Using an RPC is a much more performant way to do this. Unless having ruby run the Go program is a negligible performance cost.

glasnoster• 12 years ago

In our startup, we've taken this a few steps further. Here's our CTO's RubyConf talk about our approach:

http://www.confreaks.com/videos/5076-RubyConf2014-harnessing...

And my PyConZA talk slides:

https://speakerdeck.com/pyconza/pyconza-2014-inspired-by-lis...

nkozyra2• 12 years ago

Grabbing the data from the exec response is a super clunky way of communicating between two applications, particularly since Go is really good at keeping long-lived TCP connections.

Even without connecting directly (which is just as trivial as reading a response from exec), you should be looking at a formal message passing system. This also allows you to easily distribute these dissonant applications, which is a good idea when you're talking about "utilizing all the CPUs available" and potentially high network usage.

Zikes2• 12 years ago

I think you're talking about a microservice, and I would agree. Go is well suited for that sort of thing.

nkozyra2• 12 years ago

Well it's a microservice as described in the OP, I'm just talking about formalizing a communication method. Reading CLI responses isn't a very smart way to do it. REST APIs aren't really ideal for slow processes, either, which is why my first thought would be something like RabbitMQ or just a simple TCP connection.

xentronium• 12 years ago

Yes, yes and yes! It's not just clunky. Process spawning is super slow for sub-second operations.

Jweb_Guru• 12 years ago

Process spawning is barely slower than spawning a thread on Linux.

Exuma• 12 years ago

Very cool, I've actually been doing this lately in a slightly different way and the performance differences are just monumental.

andrewvc• 12 years ago

I don't really know why this is an article. So, uh, the author has discovered that you can fork/exec a process? I'm not sure what is novel or interesting that is presented here.

This article could have been written in 1998, titled "Using C to improve your Perl application's performance".

I'll say this about this strategy. For most web apps (and ruby is mostly used in webapps), it's usually more robust to just write a go background worker listening over RabbitMQ or the like and just shove messages around as a service.

al2o3cr• 12 years ago

Neat article. BUT:

Please, for the love of all that is holy, read up on `Shellwords.escape`. NOW.

Or at least do the rest of us a solid and post where you're setting up these gigantic security holes-waiting-to-happen in production, so we know where NOT to put personal data...

Using Go to improve your Ruby application's performance

Related Stories