Metaprogramming in Ruby: It's All About the Self (2009)

https://yehudakatz.com/2009/11/15/metaprogramming-in-ruby-its-all-about-the-self/

by Tomte • 2 years ago

92 46 2 years ago

I remember when this came out. I think I met Yehuda Katz briefly after that at a conference (along with Jim Weirch, RIP). Since then I've been using Ruby pretty much every day. I built some crazy weird stuff in Ruby, often using lessons learned from this post, and many more since. One of the rare memories I have of my 20's is figuring out Eigenclasses.

Now I've landed in using Elixir and I can safely say I kinda think it was all a little bit of a waste. I know why Ruby had these kinds of ideas, I remember the release of Refinements (2015 was a hell of a year) and I think it enabled a bunch of interesting software, but holy crap how many hours or days did I spend debugging some absolute method_missing mess? Or the people that were my peers? Was it ultimately worth it to have classes, instances, and ancestors? I don't use any of that stuff now and I write pretty much the same software I did back then.

Either way I think it's particularly revealing to compare the post's first sentence and last four paragraphs.

benreesman• 2 years ago

I worked with Yehuda for quite a while (great guy, super smart) and even having him explain it in person never got me past a mechanical understanding. You can learn the mechanics, even the implementation.

But there isn’t any blinding symmetry echoing the very thoughts of God at the end.

Which is how it’s sometimes presented.

vidarh13• 2 years ago

having him explain it in person never got me past a mechanical understanding

Which part? The meta classes bit?

The implementation is trivial: It's basic single inheritance. A meta class is just like any other class except for two basic things:

* It's automatically "injected" into the inheritance chain as the immediate class of an object when you define methods on it.

* It's "hidden" by some methods, like "#class" so that it looks like nothing has happened (I'm not sure that was a good choice - it creates part of the appearance that something weird is going on, though there are some context where you'd otherwise have to filter them out yourself)

Most of the time people freak out over the meta class bit because they assume it's more complex than it is. Put another way: If you were to implement meta-classes in C, you'd "just" do something like this:

    struct Object {
       struct Class * class;
       ...
    };

    struct Object * ob = SomeClass_new();

    /* Defining a meta class inheriting from the original class */
    struct Class * my_meta = Class_new(ob->class);
    Class_define_method(my_meta, ...);
    /* Whatever other setup you want for this new meta class, which is *just like any other class* */

    /* Make the Meta class the new class of the object */
    ob->class = my_meta;

That's basically it, other than a flag to let us hide it from some methods when traversing the inheritance chain.

noelwelsh• 2 years ago

Yeah, I think Ruby represents the end of the line for the one of the main ideas of Smalltalk: that everything should be the same thing. Classes are objects, objects are hashes, have fun. The problem is that programming is primarily about making distinctions. This is a user, that is a product, and they are not the same thing. When everything becomes an undifferentiated soup it's hard for the programmer to figure out what's going on, which leads to bugs, and it's similaryly hard for the runtime to figure out what's going on, which leads to slow code.

More modern approaches have clearer boundaries between different systems. E.g. compile-time metaprogramming has much more structure than Ruby's runtime metaprogramming.

vidarh13• 2 years ago

Objects are also very specifically their own thing and of a specific class, so they are by no means undifferentiated.

I agree a somewhat sharper split between something like compile-time and runtime would be helpful, because it would allow some distinctions that are otherwise hard, but those are surprisingly few. There are absolutely challenges in optimizing Ruby, but I'd take those over a much more constrained language any day (some minor constraints would help; e.g. a "everything after this is runtime" marker would impose nearly no limits but would enable better tools)

igouy• 2 years ago

… one of the main ideas of Smalltalk: that everything should be the same thing.

No, not that everything should be the same thing.

1981 "Design Principles Behind Smalltalk"

https://www.cs.virginia.edu/~evans/cs655/readings/smalltalk....

riddley• 2 years ago

What do you mean by "objects are hashes?"

vidarh13• 2 years ago

You'll note the article doesn't mention method_missing even once, and in general method_missing is rarely used much in modern Ruby meta programming, for good reason.

Overally I just don't get this difficulty people have debugging Ruby, and to me it feels like there's just some fundamental disconnect where people insist on working with Ruby as if it is statically typed, and then get confused when they find it hard, instead of embracing the dynamism and message passing nature of it.

In Ruby I usually work with a "dev mode" in my code that makes it trivial to drop to a repl (there's also ruby-debug which allows dropping to a repl "anywhere" but since you then run in a trap context it limits what you can do) and optionally reload code, which means I can poke and prod at every aspect of the runtime system whenever and wherever there's a problem, and can often write a fix, load it into the running system and retry the basic block that triggered the error to see if it fixed it.

My editor is written in Ruby, and if something breaks, I get thrown into a pry prompt and can fix it that way and continue. Or I can exit, because since every method call is a message, DrB lets me trivially forward it over a socket, and so the actual text buffers sits safely in a different process (and if that throws an exception, I can debug that too, via the same mechanism, from the client, while it is still running)

It might well be Elixir would be the better choice for how you think - I'm not trying to convince you Ruby works better for you. But I think a lot of people would be a lot happier with their Ruby experience if they actually explored what a more dynamic system makes possible. If you don't take advantage, there's really no point - in that case you're paying a performance cost for no gain and would be better off sticking to a static language.

(Regarding refinements, in the years since I've found exactly one use for it - enabling torturously invasive monkey patching that'd be entirely unacceptable system-wide, and keep that firmly confined to inside a DSL - but that use is very much worthwhile)

john-aj• 2 years ago

My editor is written in Ruby

Which editor is that?

weatherlight• 2 years ago

Sidenote, What editor is it that you use that's written in Ruby? I <3 Ruby and really do lean into its dynamic nature, (it's also one of the reasons I love Elixir as well.)

danmur2• 2 years ago

I prefer editors that don't break personally, don't need another thing to fix while I'm fixing things

revscat2• 2 years ago

Let me ask you something: is there a good source on how to debug in Ruby? I keep running into issues when I dive into it outside of Rails.

For example, yesterday I ran into an issue with bundler where a gem was failing to install and needed to step through the process. I set a binding.b in the gem’s extconf.rb, but when that breakpoint got hit the program just stopped without allowing input, I.e. there was no debug prompt presented.

Is bundler running inside a trap context similar to what you mention above?

krainboltgreene4• 2 years ago

You'll note the article doesn't mention method_missing even once,

That's because `method_missing` isn't the subject of the post. Are you also going to point out that the blog post doesn't mention Eigenclasses or Jim Weirch?

and in general method_missing is rarely used much in modern Ruby meta programming, for good reason.

You're just extremely wrong.

I'm not trying to convince you Ruby works better for you

I've been doing Ruby for 15 years, I don't think you could.

throwaway918274• 2 years ago

My first job interview out of college, the interviewer's last question was "what is the Eigenclass". I had no idea, so I just said "I don't know, what is it?" - and then we just chatted about what it was. I'm convinced I got that job because I was 1) honest that I didn't know, 2) was curious enough to ask about what it was and learn about it from a potential colleague. That one question taught me A LOT about what it means to be a professional developer as a fresh grad.

Many years and companies later I now use that same question when interviewing Ruby developers for $DAY_JOB. It's an excellent way of knowing if they're a ruby developer or just a developer that knows ruby.

To this day, no other interviewer has ever asked me what the Eigenclass was.

1123581321• 2 years ago

If someone tells you they don't know, do you hire them? :)

m12k• 2 years ago

I still make a living writing ruby every day, but the meta-programming is easily the part I like the least, and that I try to avoid at every opportunity (which thankfully, if you're not trying to write a framework, is all the time).

I'm sure I'm benefiting from using frameworks that use meta-programming to great effect, or that there's patterns, conveniences and ergonomics that are hard to achieve without them. But I still tend to lump it in the same category as C++ templates, raw assembly and manual thread synchronization - I'm glad it's there for the chosen few to do great things that we can all benefit from, but most of us should stay the hell away from it for our own sanity's sake, if there's any reasonable alternative at all.

derefr2• 2 years ago

I don't use any of that stuff now and I write pretty much the same software I did back then.

I mean, if you're writing Erlang/Elixir code, then you kind of do have "classes, instances, and ancestors" — it's just that you and every other Erlang/Elixir programmer are just relying on an extremely ossified and low-tree-depth set of them for ~everything, in the form of the OTP-framework behavior modules.

What do I mean by that? Bit of a tangent, apologies in advance:

Elixir might have taken syntax inspiration from Ruby, but if you're comparing the Ruby runtime to the Erlang runtime, then the analogous OOP structure to a Ruby object isn't an Erlang record or Elixir struct — it's an Erlang process.

Originating from Smalltalk, OOP as an abstract model is all about things (capital-O "Objects") that interact by passing messages to one-another's opaque "receive a message" interfaces; where there can be no static guarantees about what a given Object will do with a message sent to it, because on the receiving end, a message will be fed into to code the Object controls to "parse" and "route" and (potentially) "respond to" the message; and the code being used to do this "parsing" and "routing" of a message, can be partly or wholly determined by the Object's internal state. (In the extreme case: you can send a closure to an Object, and it can replace its receive logic with the closure, effectively becoming a different Object entirely — even while still being the same lowercase-o object in reference-passing terms.)

In the Ruby runtime, all objects fit the OOP definition of an Object, and basically every operation is a message-send.

In the Erlang runtime, regular values aren't Objects and regular operations aren't message-sends; but Erlang-runtime processes fit the OOP definition of an Object, and `send`s to those processes fit the definition of OOP message-sends.

As such, Erlang/Elixir fundamentally has the capability to do `method_missing` shenanigans — it's what you'd get if you defined a custom (non-proc_lib) process whose toplevel loop runs through an explicit `receive` with an `AnyTerm ->` match-clause.

(In other words, you could totally build something like e.g. Ruby's DRb on top of local Erlang processes listening for arbitrary messages and proxying them to a remote. Not that the Erlang runtime needs this, given that you can just send messages to remote PIDs in a distribution set...)

But nobody would ever consider it idiomatic to implement these sorts of custom processes in Erlang. Why?

Well, the Erlang runtime generates several kinds of "system messages" that any given process is expected to handle in a "well-defined" way. If it doesn't, then your process won't be able to, say, participate in a supervision hierarchy correctly, or accept insertion of runtime trace hooks.

But there is no real way, given only Erlang syntax, to achieve anything like "compile-time receive-selector inheritance" — to have one `receive` statement "pull in" boilerplate clauses defined elsewhere. If you want to write a custom `receive` statement, then every bit of what it's matching on has to come from your own code. Your choices, when writing a custom `receive` in Erlang, would be to either copy all the boilerplate system-message-handling clauses into the code of each and every `receive` statement you write... or to not handle those system messages at all.

Given two bad choices, Erlang's initial programmers took the third path — the "any problem can be solved by adding a layer of indirection" path — and developed the OTP framework (specifically, proc_lib and gen_server.) You could say that OTP emulates "receive-selector inheritance" through composition: the OTP behaviors each have a static set of receive clauses, but recognize generic structured message types for "user messages", that they respond to by unwrapping and delegating those messages to what is, essentially, a receive function supplied by the user (`handle_call` et al.)

With OTP behaviors in hand, Erlang programmers then eschewed writing processes with explicit toplevel `receive` loops in almost all cases, replacing almost all such code with usages of OTP behaviors.

Later, when Elixir was being designed, it followed suit in exposing the actor system mostly through the OTP framework's abstractions.

I think this was partially because there was no actual "Elixir-idiomatic" exposure of the actor system at all for the first few versions of Elixir — you were instead expected to just call the Erlang OTP modules directly; and partially out of a desire to reuse code and seem idiomatic/familiar/inviting to Erlang programmers considering adopting Elixir.

But, here's the trip: unlike Erlang, where this inversion-of-control pattern was the only "good" solution, Elixir had other options here! It didn't have to do things this way!

Elixir doesn't have the syntax constraint that Erlang does: Elixir has hygenic macros. At any point during Elixir's development, someone could have easily:

• defined a `defreceive` macro, that gives a name to a `receive`-statement-as-reusable-AST that can be invoked elsewhere in the module;

• defined a version of this macro, that takes other, already-defined `defreceive`s (named `receive` statements) as "ancestors", and merges them into this `receive` statement's AST at compile-time.

With such a macro in hand, you wouldn't need to rely on the delegation approach of proc_lib/gen_server at all. Every process could have its own "bespoke" receive function tailored to its needs. To handle all the system messages correctly, there'd just need to be a receive-selector "root class" for your own defreceives to inherit from, that has all the same match-clauses in it that `proc_lib` does.

And from there, every stupid Ruby trick would also be applicable to Elixir processes. (Especially since the Elixir compiler / bytecode generator is inherently+inseparably part of the `elixir` library.) Singleton methods? Runtime self-modifying classes? foo.become(Bar)? Sure, those would all work.

In short: I think it's only an accident of history (of Elxiir copying Erlang's delegate-module approach to receive-selector "inheritance", instead of going for a macro-based approach) that held Elixir back from blossoming into an ecosystem of "stupid OOP tricks" that are just as messy and ridiculous as Ruby's.

(Well, that, and Elixir currently doesn't have any concept of typing for the message-ABI of a process — let alone type reflection ala Ruby's Object#class, Object#responds_to?, etc. Among Erlang-runtime languages, Gleam gets closest to having something like this... but its OOP abstractions (Subjects, Selectors, and per-actor Message types) are all about enforcing compile-time invariants on an actor's ABI type — not on reacting to an actor's claimed ABI type as it potentially changes over time.)

krainboltgreene4• 2 years ago

I mean, if you're writing Erlang/Elixir code, then you kind of do have "classes, instances, and ancestors"

and

In short: I think it's only an accident of history (of Elxiir copying Erlang's delegate-module approach to receive-selector "inheritance", instead of going for a macro-based approach) that held Elixir back from blossoming into an ecosystem of "stupid OOP tricks" that are just as messy and ridiculous as Ruby's.

Directly contradict each other unless you ignore what I wrote.

Alifatisk• 2 years ago

I never understood why people prefer

class << self over class self.method, the former adds more cognitive load in my opinion.

typeofhuman• 2 years ago

I think this is to allow you to define class methods on modules that can be called directly. I think the Math module does this.

module Foo

  def self.bar

    "Baz"

  end

end

Foo.bar # won't work

module Foo

  class << self

    def bar

      "Baz"

    end

  end

end

Foo.bar # works

breckenedge• 2 years ago

Both of those work. There is also `module_function` for when you don’t want to write `self`.

vidarh13• 2 years ago

This is conceptually no different from metaclasses for other objects. The reason for this is that the class object for module Foo is Module. You have no way of specifying that the class object for a class or module should be anything other than Class or Module. But you can insert methods into the metaclass of the Foo Module object.

cout2• 2 years ago

I used to prefer class << self because it is less repetition. Then I discovered this led to crashes in the GC in early ruby 2.x, so I switched for a while to class self.method.

Now I write an inner module ClassMethods and use extend at class scope. This I think makes it clearer that we are dealing with two distinct classes, the class and its metaclass (which is a factory).

bdcravens• 2 years ago

Private class methods

entropie• 2 years ago

This should be the only reason, imho.

OTOH ruby programmers have usually many ways to achieve something. I think thats inerhited from perl. Its good and bad.

throwaway236188• 2 years ago

Is Gen-Z adopting Ruby? I'm worried it has become the next Perl without us realizing it.

tmpz22• 2 years ago

The Gen-Z range is 12-27. I think it would be fair to say their languages are whatever is being taught in schools and whatever the jobs in their region are.

I would think there's a lot of Javascript, Swift, and Java/Kotlin, and C#/C++ (gamedev). Ruby's position for Ruby on Rails app, server-rendered web apps, are probably dwindling with this group but certainly still valid. There's definitely a lot of jobs still pushing RoR adoption - just not as many (like PHP?).

realusername• 2 years ago

There's a lot of Rails bootcamps out there and it's still a popular option for beginners.

kazinator• 2 years ago

What's going on here is that we're adding the speak method to matz's metaclass, and the matz object inherits from its metaclass and then Object.

Ruby uses the word "metaclass" for something completely different from the familiar OOP concept that a class is itself an instance of something, and that something is a metaclass.

It looks like a Ruby metaclass is actually a hidden class that is instantiated for each object instance. An object inherits from its purported class, plus also its hidden metaclass. When you customize an object instance with a new method, it goes into that metaclass, and so other objects that are instances of the visible class are unaffected.

It seems like a pointless complication over (in the context of a single dispatch OOP) just representing methods as slots of an object. Normally, methods are class slots, but if an instance slot is used for a method, then that is specific to an object.

Hash tag #ICantBelieveTheyCalledThatMetaclass

vidarh13• 2 years ago

A lot of people prefer the term eigenclass, but yes, it's poorly named.

You're absolutely right, a Ruby metaclass is nothing more than a hidden class that acts as if it exists on each object in the inheritance chain before the class you specify that you want to inherit from.

In practice it is dynamically inserted first if you define a method on it, and so the vast majority of objects do not have a meta class.

In terms of being a pointless complication, consider that if you add a field to the object, then it complicates the method call logic, because now at every call site, in addition to considering the inheritance chain, where each element is identical, you now first need to rule out whether a given method name has been overridden by a given object. (this applies to MRI; there's no inherent reason why a Ruby implementation needs to follow a chain like this - e.g. my half-finished, buggy work-in progress Ruby compiler uses C++-like tables, and instead propagates updates downwards, and so will only ever do a single indirection function pointer call, at the cost of larger class objects)

The metaclass solves that by just inserting a pointer in the inheritance chain if you're defining a method on a meta class and the object does not currently inherit from one. Then all other logic is the same other than the `class` method, which skips over classes that are meta classes.

Metaprogramming in Ruby: It's All About the Self (2009)

Related Stories