Giving up on Ruby packaging

http://www.lucas-nussbaum.net/blog/?p=617

by philf • 16 years ago

92 144 16 years ago

davidw15• 16 years ago

I started using RVM a little while ago, and... I am not that excited about it (although it is well done and mostly works as advertised). It seems like the direction Ruby is going is to have things like Rails applications that require their own version of Ruby and a whole slew of gems, which also means that if you have a number of applications, you start getting a large amount of duplication? I'm not entirely sure as I'm new to the system, but it adds an extra level of installed stuff that I'm not 100% comfortable with.

Edit: as an aside, Zed Shaw's wild attacks and untruthful accusations against Debian were some ugliness that the world could have done without. I hope Lucas finds other places to more productively (and happily) employ his talents in the free software world.

count2• 16 years ago

Makes applying security patches a whole bunch of fun too!

gregwebs2• 16 years ago

how is applying patches more difficult? I would think it is normally easier because one can RVM install a new release that has the patches applied.

steveklabnik5• 16 years ago

It's still pretty easy.

  $ rvm install ruby-head-r29837 --patch refinements.diff

For example. Bam! You're done.

gregwebs2• 16 years ago

You don't have to have any duplication (with RVM and bundler) if you use the same ruby version on your projects.

Requiring an exact version of Ruby for a project is a practice that avoids rare issues of version inconsistency.

davidw15• 16 years ago

> Requiring an exact version of Ruby for a project is a practice that avoids rare issues of version inconsistency.

If they're rare, you ought to identify them, and, rather than require an exact version, say "this version or greater", and spell out exactly why in the comments, so that future users of your code will know precisely what code is affected and be able to reason about it themselves.

FooBarWidget10• 16 years ago

With Bundler you do have duplication most of the time. Bundler installs the gems to ./help by default so you get your own gem directory for every project.

mechanical_fish5• 16 years ago

What is wrong with duplication?

In general, there are a few possible answers:

A) Duplicate code sucks up storage space, and bandwidth to transfer it around. There was a time, in living memory, when this argument was strong. But it became moot several years ago when they started giving away free encyclopedias in my breakfast cereal.

B) Duplicate code sucks up memory in running processes. This argument is still potentially strong, but without getting into the important Ruby-centric details (Can separate processes usefully share Ruby code anyway? What are the mechanisms for that? Do people really often run more than, say, two Ruby interpreter codebases on any given machine?) let's just point out that this is premature optimization -- especially in 2011, when RAM is cheap, boxes are cheap, everything is virtualized, and you should probably be running every application in its own virtual server anyway.

In the general case, it is more important that code be simple to install, correct, and reliable than that it have a light memory footprint. If you are smart enough and motivated enough to understand the need to optimize your software you have the tools to do so.

C) Duplicate code creates management headaches. You think you've updated the gem, but you've really only done so in one place. You forgot the other six places.

This is a real concern, but RVM plus Bundler represents an improvement, not a regression. For one thing: Before RVM there were several incidents where I battled the infamous `gem install` vs `sudo gem install` dichotomy. That problem -- getting every app and library to load the right library path without using RVM -- is an order of magnitude more confusing than anything in RVM.

davidw15• 16 years ago

I agree completely with A for most things outside of embedded systems, and even those are generally getting more and more space, and Ruby isn't really ideal for that anyway.

I disagree with B somewhat. I currently have a machine running about a dozen different Rails applications with no problems at all. I am indeed unhappy with the fact that Ruby does not share code better, because I could either save money on monthly costs or add more applications (I do know about "Enterprise" Ruby but prefer to stick with the system version). And yes, memory is cheap, but costs do multiply when you have to pay for extra ram every month (ala Linode). I also don't want or need the overhead for separate virtual servers for each and every application I happen to write unless they get big.

However, yes, C) is really the main concern. And I've found that just using system (root) level gems is the best way of keeping everything sane. I don't install per-user gems. So for me RVM/Bundler represents a real extra step and more things to know about than just gems by itself.

In the end, I use Ruby a lot these days, so I'm not unhappy with RVM. It's well done, and generally a good thing. I guess I just don't like the direction it seems to represent: packaging each thing in its own world. I prefer a shared world with divergences only where they are required.

Vivtek3• 16 years ago

How many other free software projects are significantly hindered by the language used by the main developers? I found that the most fascinating aspect of this post. Software tends to be so very English-centric that the fact that Ruby is Japanese really stands out.

masklinn2• 16 years ago

> Software tends to be so very English-centric

I'm not sure software is "english-centric" so much as english being the linga franca of computing, much like diplomacy used to be mostly (if not solely) in french. You need a common language to share, and english was as good as any (better, in fact, since most early research [Church, Turing, Von Neumann, McCarthy] in CS was in english as were pretty much all the early implementations (Colossus, ENIAC, FORTRAN, ALGOL, BASIC, …)

gwern2• 16 years ago

> I'm not sure software is "english-centric" so much as english being the linga franca of computing, much like diplomacy used to be mostly (if not solely) in french.

What exactly would be the difference there?

Vivtek3• 16 years ago

That's pretty much my point: the "A" in "ASCII" stands for "American", and software as a culture has remained plagued and characterized by that ever since. What I found remarkable about this post is that Ruby's development culture is distinctly not run in English, and it's creating the predictable amount of friction.

The Japanese are no less insular than Americans when it comes to actual belief in the independent existence of anything off their shores - I find it fascinating to see that reflected in the Ruby project.

kscaldef3• 16 years ago

I believe that early on most of the OCaml community was French-speaking.

Personally, I found that aspect of the article borderline offensive. Oh, people shouldn't speak their native language because... speak English dammit! Oh, and don't release software on Dec 25th. (News flash: the majority of the world population does not celebrate Christmas.)

moe• 16 years ago

Oh, people shouldn't speak their native language because... speak English dammit!

Nothing offensive about that, as long as it's expressed in a polite manner. To me it sounds like a constructive proposal from someone who has witnessed the problems first-hand, he's not just "ranting from the sidelines".

This is not about imperialism. It's about improving the communications in a project that has long turned international. English is the lowest common denominator. Refusing to acknowledge that simple fact does not help anything.

zppx• 16 years ago

I think it's a problem when your project goes international. You have to choose a language to communicate with the larger community.

Some countries do not have this problem, the Anglo-Saxon world can communicate with a common language, they are inflated by the Indian community that generally uses English in their communication as well.

India, with its many official languages, is a living example of how this can be a problem in Science and Technical communities, restricting our problem to programming you'll hardly see an Indian developer starting a project and documenting it in Marathi, Hindi, Urdu, Tamil or Malayalam (or others with more than 10 million speakers).

This can be a problem, so you have to decide very early how you'll approach this problem, Linus Torvalds for example chose English in the beginning instead of his native Swedish. The Lua developers chose a similar route, since they native language is Portuguese, I do not know much about Python, but I expect that Guido also chose English to communicate with developers instead of Dutch. Ruby apparently took another way, some people sees this as a problem, others don't, personally I would not like to work in a project in which I cannot ask some guys directly because we do not know a common language to communicate.

For me English is a easy choice since the majority of people learn some English in school or college, or even playing games.

cmelbye• 16 years ago

A majority does not celebrate Christmas, but a large chunk of it does. Even if you don't celebrate Christmas, you may have time off of work anyways. It's just not a good time to do the yearly release in my opinion (and the author's too)

pgroves2• 16 years ago

This reflects a larger issue about project management: Do you want more people working a little on the project or a few top performers working at their absolute maximum productivity? If I was them I'd keep running the project in the way that makes the top 10 contributors as productive as possible, and it sounds like keeping the discussions in Japanese is the way to go.

Think about if we were discussing Apple. Would you really suggest it would be worth making the top performers (including Steve Jobs) do their day to day work in Japanese if it meant more people could participate in the discussions? Or do you want to do everything you can to have the people who really know what they're doing working at their optimum?

davidw15• 16 years ago

It could well be a 'local optimum' - switching to English would slow things down for a bit, until the much larger pool of people who speak some English started to get involved, and a few to stand out, and take things to a level the smaller group could never have reached.

crazydiamond• 16 years ago

RVM has been a boon. 1. Easy to test my library/app against multiple ruby versions.

2. Easy to run an app which requires an older version of ruby (I typically use 1.9.2).

Advanced features like gemsets allow finer grained control but only if needed.

Corrado2• 16 years ago

RVM is great for some things and a bane for others. Like you, I use it all day, every day to build & test multiple Ruby projects on my MacBook. It has really increased my output and reduced my hair-pulling.

On the other hand, automating and managing a server install using RVM is a nightmare. I don't want/need compilers on my servers, I just want to push a package out and have it work (i.e. APT).

herval• 16 years ago

out of curiosity, what other issues does RVM pose to "others", other than having to apt-get install compilers? also, wouldn't you need compilers anyway, for a whole lot of gems that use native extensions?

wccrawford• 16 years ago

Whoa... Even though the developers of Ruby are Japanese, he's demanding that they start speaking fluent English instead? And he thinks others are being assholes?

Granted, he only said 'English' in his rant, but if it's typical Japanese English then he won't understand it half the time anyhow, so it'd have to be fluent English.

Here's a novel idea: FORK IT. If you don't like how a project is being run, that's your option. Telling others how to do their work doesn't work with open source. They don't have to listen to you, and calling them assholes certainly doesn't help.

davidw15• 16 years ago

You are the one who used the word "asshole", not Lucas, who isn't a native English speaker either.

gwern2• 16 years ago

Telling someone to FORK IT is only useful if the maintainers are so incredibly incompetent or toxic that even accepting the massive penalties of a fork (especially massive for huge commercially popular projects) one is better off with the fork.

Nowhere else. I don't want to live in a world where this is considered an acceptable response to every small and big criticism or suggestion for improvement:

"Hey, maybe we should remove that dead code."

"IF YOU DON'T LIKE DEAD CODE, FORK IT!"

"Have we considered moving to Git from our crufty old CVS?"

"IF YOU DON'T LIKE CVS UP, FORK IT!"

"Docbook really isn't very easy to contribute doc fixes to; I was thinking about rewriting our manual in Markdown since it's really not that complex a document - "

"FORK IT!"

dudus• 16 years ago

It's very true. And if you decide to fork it than that's what you hear:

"OMG, YOU CAN'T FORK RUBY. ARE YOU CRAZY? THAT WOULD BE TERRIBLE."

Corrado2• 16 years ago

I really hate to see Lucas burn out and I think the Ruby & Debian communities will both be negatively affected. I'm a Debian Ruby user whose contributed a tiny bit (first Redmine package) and I have watched Lucas and the rest of the team repeatedly bang their heads into a wall when trying to get agreement on something from either side.

My personal view is that the Ruby/GEM packages should be laid out like the Perl/CPAN, which seems to be doing just fine.

If you want to help, go to the Debian Ruby Team page (http://wiki.debian.org/Teams/Ruby) and join us (http://wiki.debian.org/Teams/Ruby/RubyExtras/JoiningTheTeam).

telemachos7• 16 years ago

> My personal view is that the Ruby/GEM packages should be laid out like the Perl/CPAN, which seems to be doing just fine.

Could you say a bit more about what this would look like and how Debian's treatment of Perl differs from how it handles Ruby?

runjake• 16 years ago

I love Perl, and I'm a Perl god. But CPAN is a plague unto it. If any piece of software should be called a ghetto, CPAN is it.

telemachos7• 16 years ago

I like cpanm[1] quite a lot. Have you tried it as an alternative installer for modules? (I'm no Perl god, but I still maintain a healthy chunk of Perl code I wrote for sysadmin things a few years ago. I find that cpanm does exactly what I need it to, without the overhead of CPAN.)

[1] https://github.com/miyagawa/cpanminus

sigzero• 16 years ago

I love Perl too. I couldn't disagree more. I love the CPAN and I have had no trouble with it at all.

rb2k_• 16 years ago

I just know that whenever I wanted to use the official ubuntu repository, I ended up not being able to install a gem because rubygems was too old. Since gem update --system doesn't seem to work on debian-based systems, I just had to install it from source again

xiaomai• 16 years ago

This wasn't a problem for a long time, but then a major release of release of rubygems came out, and then the switch to gemcutter happened, etc. I've gotten around this by installing everything but rubygems from the regular Ruby packages from the Debian (or ubuntu) repos. I install the rubygems.org version of rubygems and everything works great.

netghost• 16 years ago

So what is it that RubyGems does differently than other languages that makes it so hard to support? Having a binary debian package for each ruby distro makes sense (as an alternative to rvm), but I'm not really sure why gem is a hindrance.

telemachos7• 16 years ago

The Rubygems package can be updated itself (normally) via `gem update --system`. On Debian (and Debian-derived distros), this feature is disabled.

I believe their reasoning is that they don't want a package installed and tracked via APT to be updateable outside of APT control and tracking.

kscaldef3• 16 years ago

CPAN can also self-update. So, why is Rubygems bad and CPAN presumably okay?

pkulak• 16 years ago

I've never had very good luck using the Ubuntu Ruby packages. The binaries are never linked, and I can't update Rubygems, which means I can't install a lot of gems that I need. Is it tough to use RVM on a production server? Is it easy to get Thin setup and all that?

thmzlt• 16 years ago

The same thing happened to Gentoo. I talked to the people behind Ruby packages in Gentoo and they wanted to manually create packages for each gem. It is one of the main reasons why I gave up on Gentoo and went back to Arch Linux.

I see gems as just snippets of code instead of "real" OS packages. They change a lot, they are application specific, and (most of the time) they are simple.

davidw15• 16 years ago

> I see gems as just snippets of code instead of "real" OS packages. They change a lot, they are application specific, and (most of the time) they are simple.

And, like me, if you're using Rails, they are code that is visible to the outside world. That is code that you want to make sure is secure, and is not just some minor detail.

grimen• 16 years ago

Evolution is key: Survival of the fittest, and RVM is the fittest in my experience.

tptacek13• 16 years ago

Can I be the jerk on this thread to raise my hand and say, "from what I can see, every single thing Debian has done with Ruby, from the way they split Ruby up into separate packages to the mindboggling damage they inflicted on RubyGems, has worked to the detriment of Ruby and provided no discernable benefit to end-users"?

For at least a year now, I've simply accepted that Debian's Ruby lives in a parallel bizarro-universe, rm'd any trace of it that intrudes on my own (real) universe, and installed everything from source.

I write this not to feel better about myself but to attract the passionate, angry disagreements that will help me better understand the problems this guy was trying to solve, so I can understand how he looks at the world and what his decision means instead of silently hoping that this post means nobody is going to try to integrate Debian and Ruby anymore.

halostatue7• 16 years ago

Thank you. As a Ruby package developer, I have to care about MANY platforms, not just Debian.

This is why RubyGems was developed, because neither Windows nor Mac OS X have package managers built in. Debian maintainers may despise RubyGems and what it represents, but the RubyGems developers have never been unwilling to take patches that make them better citizens. They have not, however, been interested in neutering fundamental features.

bryanlarsen7• 16 years ago

That argument would make sense if RubyGems was actually useful on OS X or Windows. But every single application targeted at "normal users" on Windows or OSX written in Ruby bundles all dependencies, usually even including Ruby itself.

RubyGems needs to REMOVE harmful features, like the ability to install multiple minor versions of the same gem. Developers should be using RVM to get this capability.

jamtur018• 16 years ago

Rubbish. Try to install a binary into a directory that isn't bin and then tell me about being better citizens.

another_ali2• 16 years ago

maybe Ruby should get something like perlbrew http://search.cpan.org/~gugod/App-perlbrew-0.15/lib/App/perl...

this is good magic

telemachos7• 16 years ago

Um, you're joking, right? Perlbrew itself is fine (I use and like it - I actually wrote a small Bash completion script for it[2]), but it implements only a fraction of the functionality of rvm[1]. (Rvm also precedes Perlbrew by some time, I think).

[1] http://rvm.beginrescueend.com/

[2] https://github.com/telemachus/perlbrew-bash-completion

bryanlarsen7• 16 years ago

You've done the right thing, but attributed blame in the wrong place, it's Ruby that's in the parallel bizarro-universe.

Ruby, RVM, ruby-gems etc were all designed and built for developers. APT was designed and built for system administrators.

In the world of system administrators, API differences between versions 2.3.5 and 2.3.8 would be considered bugs. In the world of Ruby/Rails, that's just standard practice. APT would not let you install both 2.3.5 and 2.3.8 simultaneously because that would be silly. It is silly, but it's useful for developers. The detriments are obvious to everybody but the Ruby community -- DLL hell is one of the major factors in the decline of the Windows platform.

So yes, keep Ruby off in it's own parallel universe, partitioned off using RVM or bundler. As a developer, this brings you many real benefits. But recognize the real detriments to normal users and system administrators, and place the blame squarely where it belongs -- on the Ruby community.

tptacek13• 16 years ago

Ruby is overwhelmingly used by developers. I understand that there are a few projects that are so popular and useful that end-users and admins might want to install them without actively developing for them. But the decision to optimize Ruby for that use case, instead of for its overwhelmingly more common use case as "developer tool", just - seems - wrong.

I don't "blame" anyone. I think Debian is wrong about Ruby.

pgroves2• 16 years ago

I agree wholeheartedly. I develop in ruby and ocaml on ubuntu, which both suffer this problem (I assume all languages do). As a user, I have no tolerance for dealing with dependency hell for things like a desktop pdf viewer or rss reader, and a decent gui frontend to APT literally provides a one-click solution to installing an application and all it's required dependencies. As a developer, I gave up on using APT as part of my development process long ago - I manage my own library of sources and binaries. It's just too painful to synchronize a development environment with the debian/ubuntu repositories that are being updated on their own schedule. From personal experience, I can say the worst case is having your development environment break b/c you updated your email client, which is easy to do if you use APT to manage your binaries, regardless of the languages involved.

davidw15• 16 years ago

I am not going to say that the current packaging scheme is ideal, but to say that Ruby shouldn't be packaged at all on systems like Debian and Redhat is to ignore a lot of the very real good that has come of quality package managers like dpkg/apt and rpm.

One use case would be if someone, say a sysadmin, wants a good bug tracking system for internal use at their company.

    apt-get install redmine

Fetches Rails, Ruby and so on, and sets it up in a default, reasonably secure way (as far as I know, I haven't the Debian version myself).

That sort of thing is pretty commonplace with PHP web applications under Debian and Ubuntu. Granted, if you're going to hack on and modify the application in question, yeah, you may want to get it and install it yourself, but when things work well, a good packaging system lets you install and work with code that perhaps you do not know intimately.

I think you're making the mistake of viewing the world exclusively from the point of view of a Ruby developer. What would happen if you had to deal with different systems to install, maintain and upgrade the Linux kernel, Apache, Postgres, Sqlite, Ruby, GCC, Perl, Python, Gnome, and various submodules used by them? There are times when it's necessary to install your own version of something, but it's awfully, awfully nice that you can "choose your battles", and really only do that with stuff you care about and follow closely.

Sounds like Linux in the days when you installed Slackware and then just upgraded stuff by randomly recompiling bits and pieces every now and then.

BTW, I would hope that we could skip any "angry disagreements" here. Passion will do more than suffice!

erikpukinskis• 16 years ago

I don't think that really makes sense. I think the sysadmin should instead have to do:

    apt-get install rubygems
    gem install redmine

I know it's two steps and they have to understand the dependency, but the truth is everyone in the Ruby world kind of assumes you're using Rubygems, and repackaging things from one packaging format to another just seems like it's asking for trouble.

Sysadmins can learn that if they want something from the ruby world they have to install rubygems.

wglb• 16 years ago

A similar situation exists in Lisp. Most everyone I know downloads the lisp of their choice (SBCL, CCL, Clojure) from the project site and goes forward from there.

This seems particularly true of efforts that are cross-platform. In particular, developing for Linux and Mac pretty much makes this the way to go for these two languages.

lusis4• 16 years ago

What would be nice is if RVM allowed a package option for an installed Ruby + gemset. RVM doesn't work for me on my servers because it break idemopotence but I'd love to install RVM on a testbed, type 'rvm package' and get a tarball that I can dump in /opt/ on my production systems.

tenderlove• 16 years ago

At RubyKaigi, we talked about closing the Japanese list. It's a bad idea because closing the list will not stop conversations from happening in Japanese, it will just move the conversations to IRC. At least, if we have the conversations recorded, there is an opportunity to translate them.

now• 16 years ago

Has anyone considered that crap like RubyGems, RVM, and Bundler is a symptom of us as rapid (read: careless) developers not being able to manage our code and our releases in an organized and well-ordered manner, not a symptom of poor package management by distributions?

jhealy• 16 years ago

I understand Lucas' decision, however it leaves some worrying questions about the state of ruby on Debian systems.

Despite the rumours, the ruby1.8, ruby1.9 and rubygems packages are are excellent shape thanks to the hard work by Lucas, Diago and Akira.

There were consistent fundamental workflow clashes between upstream and the Debian packagers that have made their job exceedingly difficult. These clashes aren't malicious or antagonistic, they're just the meeting of two very different communities with very different priorities.

To name a few of the clashes: API changes in minor releases, effective forking of rubygems by copying it into the 1.9 tree, opaque decision making (mostly due to the language barrier), allowing installed programs (ie rubygems) to update themselves and expectations on what is allowed in /usr/bin.

I'm not saying either community is necessarily correct on these issues, just that they add up to some pretty big differences for debian-ruby to navigate.

I use the Debian packages for ruby1.8, 1.9.2, jruby and rubygems in both my development and production environments. I've found them stable, rapidly updated in response to security issues and predictable. Lucas - thanks for your efforts, they will be missed.

amacater• 16 years ago

People - you forget that the Debian community is not just random packagers but also developers. Debian packagers try VERY hard indeed to produce a stable system that works out of the box for all of their users - and that includes developers of all sorts with all types of development environment for several common programming languages.

As part of that, it's useful to know what's stable and what isn't. Debian tends to track version numbers and look for stable versions for a reason - once released, Debian commits to supporting a release for the lifetime of that release plus one year - roughly three years now. Anything that isn't actually released as stable and thereby fixed will go into testing/unstable - there's nothing to stop you running the latest and greatest at any time, but possibly not on Debian stable.

In another life, I have to deal with developers who say to me: just mirror Ruby and Rails for us on the inside of the corporate firewall. There isn't an easy place to start - if you want to build a Ruby infrastructure inside a large company, with software management systems, management suspicion of FLOSS but committed developers - where do you begin to convince people that this is a usable infrastructure?

Red Hat support a subset of very common packages and will support these now for up to 10 years. That's your banks, stock exchanges, infrastructure companies: if you want Ruby developers and programmers to make inroads into that sort of environment, then you need sane packaging sense or at least a sense of what's usable.

Debian produces a far larger subset for use of the programs out there - and on 13 hardware architectures.

As regards the English/Japanese language flaming: Debian has 1000 or so developers and maintainers, spread worldwide - Lucas is a French speaker, originally, but Debian developers speak and use most common human languages - English is a convenience to a truly distributed, international community.

Giving up on Ruby packaging

Related Stories