RubyGem is from Mars, AptGet is from Venus

http://stakeventures.com/articles/2008/12/04/rubygem-is-from-mars-aptget-is-from-venus

by pelle • 18 years ago

16 42 18 years ago

gleb• 18 years ago

I don't believe Debian ever managed to deal with 3rd party package management tools well. E.g. XEmacs has its own package management and apt-get equivalent, and it never worked right in Debian. It's fundamentally hard problem to support different 3rd party release cycles while fulfilling traditional Debian goals of stability and correctness.

Having said that the situation is particularly poor in Debian ruby gem land. There are 2 kinds of people -- the ones that understand Unix dynamic library versioning scheme and the ones that do not; the one that understand while non-distribution software needs to go into /usr/local and the ones that don't; the one that understand why Debian needs all the complexity in dependency management and the ones that don't; the ones that understand Unix rule of silence and the ones that don't, etc. Debian and Ruby communities are on the opposite side of this divide.

old-gregg2• 18 years ago

One of my Rails projects runs only on aptitude-provided gems and it has been a breeze. Yes, I don't get the latest and "hottest" versions of popular gems, but everything works and very reliable.

With other project I installed gems and Ruby from the source and have been handling dependencies with RubyGems. Not nearly as smooth: maintaining this mess without google would have been close to impossible. Internet is full of blog posts and article about "installing X on Leopard or upgrading Y on Ubuntu without apt-get". And guess what: those posts make absolutely no sense in an aptitude-controlled environment, where everything just works: the entire system can be re-created with a single apt-get command and a copy of your /etc files.

And the author's position of fuck you all, or as he puts it: "I'm all about application, systems management is not my job" is precisely why Windows is such a piece of shit: system management is nobody's job there, everyone is all about "application". This is why after you install a few creations of these "application guys" the whole thing becomes an unusable mess and you start paying attention to "I'm a Mac" commercials.

pelle3• 18 years ago

Hmm, I don't remember writing "fuck you all" in my post. I'm just talking about there being different domains in play here. Different domains have different needs.

System management thanks in many ways to the great work of Debian but also virtualization has become a commodity more or less. Over the last 30+ years the best tools for the job have become more or less standardized. Which is great.

The application development space is a lot more dynamic though. You can not expect all unix sysadmins to know all about all the different programming languages packages. There is just too much for there to be a single gate keeper now.

The world is changing and for the better. Deal with it.

old-gregg2• 18 years ago

No, it's not about Linux sysadmins but rather about writing a robust software. Aptitude is an application, just like your Rails app, written by very very smart programmers. And those developers are moving at exactly same "tremendous speed" as you are, and their "core DNA" is not a bit slower than yours, and results of their work is as much of a "commodity" as yours.

I suggest you re-read articles you linked in your post, because they explain why RubyGems has issues and needs fixing, something you haven't even addressed.

Your post basically says "YOUR work is commodity, YOU are moving slower than us, while MY developemnt is fast and bleeding age, YOU should get over your priorities and ME should continue as I please". You even went as far as calling someone on "spreading FUD" while in reality Gunnar is absolutely right: RubyGems is dangerous on production systems and I dare you run sudo gem update --system on a server that your paycheck depends on. While sudo aptitude safe-upgrade runs pretty much automatically on thousands of mission critical Debian-based deployments. That's not FUD, pelle, that's called high-quality engineering.

davidw2• 18 years ago

He doesn't address this point at all, calling security concerns 'FUD':

With apt-get, you have one thing to keep up to date in order to keep your system secure. It's so easy that you can automate it if needs be. It's also easy to keep track of security updates. With 'gem', now you have two things to keep track of.

I currently use apt-get along side the gem stuff to maintain my servers, but it's something I do worry about. I have no hesitation about trusting my servers to Debian or Ubuntu, but gem isn't quite so much a known quantity, and furthermore, I'm not sure the update system distinguishes between security updates and "hey, there's a new version of Rails (or whatever) that might break a bunch of stuff".

It's not an easy problem, that goes without saying, but I wouldn't be so quick to dismiss either side.

FooBarWidget2• 18 years ago

Maybe that's because the "security concerns" don't really explain what the actual concerns are, but only claim that they exist? If "RubyGems is a security hazard, I won't tell you why but it is so and if you disagree then you're an idiot" isn't FUD then what is?

davidw2• 18 years ago

It's a security hazard because it's an entirely different update channel for security updates, as well as a second system that runs with root privileges and installs things from other people. It is not an immediate threat, but it is worrisome.

justindz• 18 years ago

I'm currently using gems because I haven't gotten to the level of complexity where it's made any difference. Also, I tend to operate on Windows and Linux interchangeably and using the same approach in each platform is easy. I suppose I might feel differently if I were not doing personal projects and in single user mode.

This article did bring up a question I had with Firefox on apt-based systems. Why would I install a Firefox plugin through apt rather than just through Firefox plugins? Is the argument basically the same--that there are fewer/older plugins in apt but they have been proven stable by someone you can decide to trust so you get to choose what you care about most?

graywh• 18 years ago

For something like Firefox I'd stick with its built-in add-on manager. Mostly b/c there's not likely to even be many of them available through apt.

etal• 18 years ago

I imagine the Firefox thing is to make the initial setup of a machine faster (doing Firefox addons and other packages in one shot with apt-get), and possibly because the maintainer's didn't trust the security of Firefox's update mechanism. If you can rely on apt-get always being available, you tend to use it as a first resort.

ruby_roo• 18 years ago

How does Debian handle CPAN? Must you rely on apt-get for your Perl modules as well?

jeremiah• 18 years ago

The debian <--> CPAN interface is easily managed, and there are more that a thousand CPAN modules in debian. (See the debian-perl web site: http://pkg-perl.alioth.debian.org/)

The great thing about debian packages is that they not only manage all your perl dependencies for you, but they also manage operating system dependencies too. Gems can't do that.

The Perl Foundation actually holds up debian's packaging work of CPAN modules as the model for other distributions. (See http://www.perlfoundation.org/perl5/index.cgi?hints_for_dist...)

rbanffy• 18 years ago

With Python, which has the easy_install utility for installing its own packages, I use the virtualenv mechanism to create a "shadow" environment connected to the machine's main Python but with its own library/package folders.

With that, I am free to install anything I want in the secondary Python, I still have access to distro updates for libraries that I chose not override in my environment and I am completely cool I am not hosing my main Python install.

In the unlikely event there is nothing similar to it for Ruby, it would be trivial to develop it.

gcv• 18 years ago

I deployed one project by creating a completely separate deployment area, /opt/projectname, which contains a separate tree, /opt/projectname/sw, where I manually compiled all the dependencies: ./configure --prefix=/opt/projectname/sw. I installed ruby and gems in there, as well as the specific version of Apache I wanted to use. On the downside, it means that managing upgrades comes problematic, as I have only isolated the bin/lib/etc/share mess into a separate tree, not eliminated it.

To make life easier, though, I'd like to use an alternative source-based package manager. I'm not sure how to do this, but I'd like to run the './configure --prefix=/usr && make && make install' invocation in an environment which tracks the files copied by the install process, and knows which files in the system my packages affected. At this point, it would (ideally) update the .deb database on the system. It could (alternatively) keep track of these files in its own database. Maybe a chroot trick could help implement this.

inklesspen• 18 years ago

http://www.gnu.org/software/stow/ is meant to provide some of what you want there. Unfortunately, it seems to be abandoned, but it looks like http://xstow.sourceforge.net/ might be picking up the torch.

e1ven7• 18 years ago

Coincidentally, I'm sitting here staring at a terminal for the few hours because of RubyGems.

As a systems administrator, I'm absolutely not a fan of Ruby gems for exactly the reason the article mentions-

They're designed to be convenient for the developer, but they make an end-run around all of the methods that we admins set up to make things easy to deploy, and safe to update.

For instance- At Darkenedsky, we have a fleet of Ubuntu 6.06 webservers that are configured to run Apache, Postgresql, etc.

One of our coders has recently made a change which means that I need to push a library out to all 50+ machines..

Normally, this is a pretty trivial affair.

    for i in `cat hostlist`; do ssh $i apt-get install  library; done

Unfortunately, this particularly library is a Ruby gem, so things aren't nearly as simple ;(

The first step is that I need to ssh into each machine, and manually install Rubygems. Rubygems aren't packaged for Ubuntu 6.06, so I need to do a source install and configure.

Once it's installed, I need to update gems to have the most recent package definitions, and then I can start to install the package I actually want.

So far, it's annoying in that there are several extra steps, but they're scriptable, so it's liveable.

Unfortunately, installing the packages isn't very scriptable at all. When I install, I'm presented with a menu-

    Select which gem to install for your platform (i486-linux)
    1. fastthread 1.0.1 (i386-mswin32)
    2. fastthread 1.0.1 (ruby)
    3. fastthread 1.0.1 (mswin32)
    4. fastthread 1.0 (ruby)
    5. fastthread 1.0 (mswin32)
    6. Skip this gem
    7. Cancel installation

If this were a non-gems library, and/or packaged as a normal Ubuntu release, I wouldn't need to manually choose one of these. I could set my choice in debconf (write it to a file), and then push it out everywhere.

If we're dealing with only one machine, like most developers do, this isn't a big deal. It downloads and installs the gem, and you're good to go. But when you're dealing with a fleet of machines, anything that requires manual interaction is a sin against man and god.

Debconf allows me to choose whatever level of interaction I want-

    "Ask me all details, even the trivial ones"
    "Only ask me the important stuff"
    "Use the choices provided in this file"

Why should I need to choose between 1.01 and 1.0 after I start the install?

On Ubuntu, if the packages were really so important that both needed to be available, I could choose to install something between

    apt-get install ruby-fastthread-1.0

    apt-get install ruby-fastthread-1.1

Again, we're down to one command. Nice and simple, no interaction needed.

Further compounding the pain is that this isn't a choice I should even have to make- Why would I want the win32 version of a library on by Linux box?

If I'm installing a packaged version, It's already been tested in this configuration, and it's designed to work with everything else on the system.

Too many servers give me an error on install-

    Building native extensions.  This could take a while...
    ERROR:  While executing gem ...      
    (Gem::Installer::ExtensionBuildError)
    ERROR: Failed to build gem native extension.

While that's certainly fixable, it means that I need to babysit the install far more than I do for something that's properly packaged.

After I choose what I want, I get to wait while the package downloads across the public internet.

When dealing with Ubuntu, all our servers are pointed at an in-house repository for packages- We keep a copy of everything we need on-site, and I can update it with one command.

This means that when installing software, it only has to travel to the server a few racks over, not out the internet at large. With gems, this system is broken- Each one reaches out to the public internet.

Not only is this is much slower, but I can't trust that a package is 100% identical.

Having one machine be different, even by a point release, can be a guge problem in trying to debug intermittent errors.

Finally, upgrading the packages are far more difficult that they have to be.

If a bug is discovered in a library, such as the zlib bug that came out a few years ago, I can update it trivially and easily across all the machines.

Ubuntu has built-in commands for updating security bugs, and it separates that from updating features.. That way, I don't need to move to the next version of a package, possibly breaking the code, to resolve a security bug.

Now, that's not to say that issues with Ruby gems couldn't be fixed.. I'm sure that one by one, they could be addressed- Scripting support could be beefed up, local repositories could be added. Upgrading could be improved, and configuration could be rewritten..

But even if they were to fix EVERYTHING on the list, it would still be layering one packaging system atop another, which means that we need to have two ways to do everything.

Two sets of config files. Two repositories of packages. Two places for bugs.

As the author suggests, the problem comes in that OS packages and Ruby gems are are focused on different audiences-

Developers love ruby gems because for the one machine they use to write the code, it's easy, fast, and convenient.

Admins hate gems because when you're trying to deal with a fleet of machines, it's none of the above.

litewulf2• 18 years ago

Doesn't gem install path_to_gem.gem work?

(I could have sworn there was something like that.) Though, most of your complaints are very vaild. Also add on that ruby gems LOOOVES to eat all the memory on your machine if at all possible...

e1ven7• 18 years ago

In this particular case it's complicated by the fact that ruby gems isn't pre-packaged for Ubuntu 6.06, but in the general case, yes, that would make the installation component of it faster.

My frustration comes mainly in that Ubuntu makes nice, easy systems to keep everything consistent; Ruby gems make an end-run around these systems. No matter WHAT it might do on it's own, it bypasses the rules that the rest of the system lives by.

pelle3• 18 years ago

Look. This is a real practical criticism of RubyGem which is absolutely valid. But these kinds of things are getting better.

Rails 2x and Merb 1.x with their support of Gem dependencies are a lot easier to manage now than they were in the past.

It would be relatively easy nowadays to manage one server with lots of different applications with different Rails and Gem dependencies purely using gems now. You would not be able to do this if Rails was installed via aptget unless a special require_aptget library was written.

RubyGem is improving a lot with every release. Poolparty and other similar tools for remote management look like they would be able to deal with many of these kinds of issues in the future.

I believe EngineYard maintain their own local gem server for speeding this up.

e1ven7• 18 years ago

Great point, Pelle, Thanks.

I think that's exactly the problem, though.. Yes, I'm 100% that all of these problems can, and probably will, be fixed.

But even once they are, there will be two methods for doing everything, and that makes things more complex than they need to be.

It's one of the disadvantages of such a relatively unpredictable environment- Every tool has has to be able to work across a variety of platforms, which means that every tool has to assume the worst case. ;(

The Ruby devs can't assume packages are sane everywhere- Redhat packages, for instance, have been a world of hurt for years.. OS X and Windows don't even HAVE native packaging systems.

I entirely understand why it is the way that it is.. But I look forward to it getting better.

As another poster pointed out, in newer releases of Ubuntu, they DO package up the gems as individual .deb packages.

Essentially, the Ubuntu devs bypassed the gem manager, just as the gem manager itself bypassed Ubuntu.

Locke2• 18 years ago

Wow. I'm not going to claim that rubygems is the best package management system, or that it's better than debian's packaging system. But, some of your frustration comes from the fact that you know apt very well and you don't seem to know rubygems very well.

For example, you can setup your own local gem server. You can load it with only the gems you want to install on your fleet of machines and use 'gem install --source your-local-server fastthread', for example. You might even get around your "pick the right gem" problem by installing only the fastthread gem you need on your local server.

But, I think the real problem you have with rubygems is that you don't want to learn another package management system. I really don't think the rubygems devs can do anything to remedy that problem.

e1ven7• 18 years ago

Great point, Locke, Thanks for replying.

I'll certainly admit that I don't know the gems packaging system as well as I know dpkg and aptitude, but I think that that criticism misses the larger issue.

You're certainly right that the gems system could be made to do everything that the Debian/Ubuntu system does, particularly if given time to mature. But as I pointed out in my post, this wouldn't fix the problem.

Apache, for example, has hundreds of modules, each of which have complex dependencies.. They're released on their own schedule, and they're released across lots of platforms. Firefox has it's own set of extensions, and updates- You see the same thing with CPAN, and Python modules.

But the more disparate systems you add, the more sets of configurations you need, the more repositories you need, and the more ways to update everything you need.

So for our servers, I could set up a Gems repo, an Apache repo, a System Software Repo, a Cpan repo. I could have separate upgrade commands for every software package that comes along.

But the complexity is increased with each addition to this system-

* If I want to get a list of everything that's installed under this system, I need to run X commands.

* During installations, I can't chain everything together, I need to run X set of separate installations.

* I need to review X different release chains, to see if there are packages I need to test before deploying to multiple machine

* And yes, I need to learn X different systems for installing software, rather than one system for installing system wide.

There are a lot of good reasons to have the gems system, and I think it's a great tool for individual systems, but for anything that's going farm-wide, It's substantially easier to have everything go through one tool.

nailer4• 18 years ago

"But, I think the real problem you have with rubygems is that you don't want to learn another package management system. I really don't think the rubygems devs can do anything to remedy that problem."

You don't think the RubyGems devs can use the normal packaging system, and hence get rid of these odd requirements? Why not?

FooBarWidget2• 18 years ago

Thanks a lot for your post e1ven. Yours is actually constructive and makes sense, a far cry from the usual "RubyGems sucks and I won't tell you why, but if you use it then you're an idiot"-kind of FUD. I've replied to your points at http://stakeventures.com/articles/2008/12/04/rubygem-is-from... (search for "Actually, the Hackernews comments were surprisingly constructive"; the post might now show up yet because it's still in the moderation queue)

A lot of your criticisms have already been addressed in the latest RubyGems version.

lsb• 18 years ago

Why can't Central Command for RubyGems just upload any gem it gets as a Debian and RedHat package?

drnic2• 18 years ago

I'll have a crack at auto-converting all gems into debian packages if someone can point me to the "this is what debian packages need to look like" spec. Or the CPAN module->debian package converter that I recently heard about @ OSDC conference. Email drnicwilliams at gmail.com

e1ven7• 18 years ago

Sounds like a great project. You might want to ping the debian team, and see where things currently stand;

If they don't want to accept your auto-packaged debs, you could set up your own repository, and auto-push them there.. That way people could pull in the Ruby Gems from there on a regular basis, as soon as they are auto-updated, without waiting for them to be released officially.

Debian team policy: http://pkg-ruby-extras.alioth.debian.org/rubygems.html

How to create packages:

http://www.ibm.com/developerworks/linux/library/l-debpkg.htm...

http://tldp.org/HOWTO/html_single/Debian-Binary-Package-Buil...

http://quietsche-entchen.de/cgi-bin/wiki.cgi/CreatingDebianP...

They're basically just tarballs with a pre and postscript, questions via debconf, and the dependencies mapped out in a manifest.

stedwick• 18 years ago

Here's what I don't like about apt-get install on Debian. There is NO DOCUMENTATION. At least, not as far as I can tell.

For example, "apt-get install phpmyadmin"... and then what? It took me HOURS to figure out what to do next. There was no message saying, "Congrats, phpmyadmin has been installed in the following directory. Navigate to the following URL to start using it." Nothing. Or what about "apt-get install rails"? I have absolutely NO IDEA what that does. I would never build an application on that rails installation.

Rubygems, on the other hand, is unbelievably easy. "gem install whatever" and then in your application "require whatever". What could be simpler?

metajack• 18 years ago

README.Debian in /usr/share/doc/PACKAGE-NAME usually has this information. Of course, some package maintainers aren't as nice as others in this regard.

The problem is much simpler with Ruby packages since they have only one interface. Debian packages have many interfaces, and so it can never be as easy as that. I'll note that Debian/Ubuntu python packages work just as you expect: apt-get install python-twisted, then import twisted in your code.

TBBle• 18 years ago

And this is exactly what the FHS is for. Documentation for "blah" lives in /usr/share/doc/blah/. Simple and easy.

As mentioned, Debian extends this by recommending a README.Debian to explain what differences from upstream documentation or defaults exist, and the rationale.

For example, enjoy the simplicity of /usr/share/doc/phpmyadmin/README.Debian. It answers all your questions, and also makes mention of security concerns you may not have realised exist had you, for example, just extracted a copy of phpmyadmin into your web site.

And now you have enough information to know what to do next with anything that you install using apt-get, Or yum, or rpm, or portage, or (I'm speculating here) autopackage for that matter.

dwright• 17 years ago

I'm very late to this party. (not sure if this is still a hot issue)

Just thought I'd mention, that there seems to be some work in this area.

gem2deb - a gem to a *.deb (Debian/Ubuntu) package http://github.com/thwarted/gem2deb/tree/master

(and for rpm's) http://rubyforge.org/projects/gem2rpm/

a repo for rhel/deb/others http://rubyworks.rubyforge.org/

I'm sure there are others,...

RubyGem is from Mars, AptGet is from Venus

Related Stories