Shopify's Kafka producer pipeline for Ruby on Rails

http://www.shopify.com/technology/14909841-kafka-producer-pipeline-for-ruby-on-rails

by wvanbergen • 12 years ago

21 3 12 years ago

shopify.com

lobster_johnson• 12 years ago

I'm in Mexico right now, and the article redirects me to the front page of es.shopify.com, so I can't read it.

lkrubner• 12 years ago

I see this attitude often:

"At Shopify, we strive to keep our containers as minimal as possible—we think of them as a strong isolation layer for a single process, not as a complete virtual machine. "

And yet there is this:

https://devopsu.com/blog/docker-misconceptions/

"Misconception: You should have only one process per Docker container! It's important to understand that it is far simpler to manage Docker if you view it as role-based virtual machine rather than as deployable single-purpose processes. For example, you'd build an 'app' container that is very similar to an 'app' VM you'd create along with the init, cron, ssh, etc processes within it. Don't try to capture every process in its own container with a separate container for ssh, cron, app, web server, etc. There are great theoretical arguments for having a process per container, but in practice, it's a bit of a nightmare to actually manage. Perhaps at extremely large scales that approach makes more sense, but for most systems, you'll want role-based containers (app, db, redis, etc)."

Seems like there is no agreement on best practice yet?

Sirupsen• 12 years ago

What we actually mean is single role, not a single process. E.g. you might have a server that has several worker processes (e.g. Unicorn) and we'd run that in a single container. In other words, our container's init process will never have more than one direct child.

But no, there's definitely not agreement yet on what's the best approach. We hope to blog more about this in the future.

Shopify's Kafka producer pipeline for Ruby on Rails

Related Stories