Rubinius takes the fun out of Ruby

Ah, if only life were so simple.

Ruby, it turns out, effectively has functions. Unfortunately, they’re implemented in the most complicated, unintuitive way. I know, right? Ruby, unintuitive, the blasphemy.

Let’s look at a simple example:

Wait, what? Hold on, I could’ve sworn I’ve seen…

Ok, there we go. But it’s a private method. String is-a Object, and the method puts is defined on Object. It would be so natural to write this as ”howdy”.puts. But we don’t.

Why?

Let’s look closer. We know we can invoke private methods, so let’s try that:

Huh, it worked, sort of. It didn’t raise an exception, anyway. But it sure didn’t print the text that I was expecting.

If you consider that example for a moment, you’ll notice that ”hello”.puts and puts “hello” are very similar, yet fundamentally different.

In puts “hello”, the receiver is irrelevant and ignored. Even when I explicitly sent puts to the String ”howdy”, the “method” ignored it. Turns out, puts only operates on its arguments; the receiver is completely irrelevant.

Now wait, where have I seen something like this before? Oh yeah, functions! In y = f(a), there is no such thing as a receiver.

So, yes, Ruby has functions, and we use them naturally in code without batting an eye. In fact, Ruby has a peculiar, module_function, method to help define them!

We see that Ruby effectively has functions. We use them like functions in Ruby code. We even have module_function for designating them. But, underneath they are just like private methods that ignore their receiver. Kinda weird, but ok, it works.

It works, except it’s a mess, like sweeping some dirt under the rug. You don’t see it but it’s there, diminishing the elegance and making everything a little less sanitary. What if we…

Hold on. Before we wax philosophical about what might be, let’s push a little harder on this objects versus functions thing.

Why do we have objects? Why create programming languages where this concept is a central organizing principle. Why is “everything an object” in Ruby considered a strength, something that contributes to Ruby’s elegance?

If you listen to some arguments, objects are apparently total, ad hoc, confused bullshit and we all should be using just functions, Monads and types like grown-ups, thank you very much, show yourself out.

On the other hand, there are other quite accomplished people, like Alan Kay, Gilad Bracha, Jonathan Aldrich, and Gary Bernhardt who suggest (even if they equivocate at times) that there is something essential to this objects thing.

I want to focus on two works here (but feel free to read anything ever written by Alan Kay and professor Bracha in your free time).

1. The Power of Interoperability: Why Objects Are Inevitable
2. Functional Core, Imperative Shell

If reading isn’t your thing, ignore the first paper and watch Gary’s screencast. Better, read the paper and watch Gary’s screencast. Or just read the paper. Basically, do anything beyond just listening to people who say objects are bullshit and we need types or Ruby is ruined… sorry, almost started ranting there.

Ok, back to our question, why objects (and whence functions)?

I’ll offer a simple rule: Objects are for interactions; functions are for data.

What does this mean in software terms?

Fundamentally, interactions presume at least two entities. Two or more separate things “have an effect on each other”. What happens when the rates of change of those things differ?

Let’s illustrate the question with the example of a car’s internal-combustion engine and wheels. The engine and wheels move at different rates of speed (i.e. the rate of change of position). To allow these systems to interact, a car needs a transmission, a component that mediates the different rates of change of the engine and the wheels, even allowing one to move in the opposite direction of the other.

In this example, the engine and wheels have very limited interaction, via the transmission. Changing the oil in the engine doesn’t affect the tires, and changing the tires doesn’t impact the engine.

This principle of minimizing the surface area between two interacting components is the essence of the concept of objects in software. We build more robust systems by carefully limiting the scope of interaction to a small set of well-defined activities (via an API or protocol).

Now, what about functions? Consider functions in math: one key idea is whether the function is defined for a particular element of a set. For example, for the function f(x) = x^2, we know that f is defined for any real number x. Check out a nice plot of this.

But what about a function like f(x) = 1/x? Check out the plot. There’s that part of the graph at x = 0 that doesn’t show up. This shouldn’t be too surprising because at x = 0, the function value y = 1/0 would be undefined. When a function is not defined for every element of the input set, it’s called a partial function.

Partial functions are messy. Where the function is undefined, it’s like asking me a question and getting a blank stare in return. If you want to analyze some data by computing a function of that data, you don’t want gaps. You want a total function. You want an answer for every piece of data.

Here’s what we have so far: objects are more robust by limiting their interactions, and functions are more robust by being unlimited (i.e. defined everywhere for the input set). How do we resolve this apparent conflict?

We don’t. These forces are in conflict. They are both legitimate and competing, and one shouldn’t win. What we want is to balance them in a particular context. But to do that, wouldn’t we need to put objects and functions on equal footing? You betcha.

Let’s reiterate that last point, just to be sure we’re on the same page:

1. When my app is talking to a remote API, for example, I want to limit that interaction as much as possible. I want to get in, get my JSON, and get out while touching as little as possible of that other system. And the other system really doesn’t want to care about details in my app. It likely has thousands or tens of thousands of different apps to interact with. Knowing even a tiny bit about any of those would be extremely costly.
2. When I’m processing the bytes of that JSON blob I got back, I absolutely want a total function. I don’t want to misunderstand a single byte. And I never should. Bytes are well defined, unambiguous, and I should expect a total function to process them. I don’t need to “interoperate” with bytes.

One thing the astute reader may notice is that data, functions, and types go together extremely well. They are an exquisite combination. They are precise and rigorous and oh-so neat and tidy. Perfect.

And this is where the world goes to hell in a hand basket.

Someone comes along and puts types on objects. Not just any old types, but static types. It’s like oil and vinegar. They don’t mix, period. Undeterred, this brave soul just shakes it up real, real good, and us poor mortals end up with WidgetFactoryFactoryFactory things and lots of tears and suffering.

You think I’m kidding. I wish.

What to do? Ah yes, objects are for interaction, functions are for data.

We need objects to support modularity and interaction between components, especially when dealing with the messy, stateful world of missiles and services APIs. And we need functions to have well-defined semantics when operating on data, like money and bytes and lists of addresses. The two go together like peanut butter and jelly. Where have I heard this before? Oh yeah, Functional Core, Imperative Shell.

I’m not making this stuff up, folks. The world’s messy; I’m just trying to deal with it.

And that brings us back to where we started: the idea of functions in Ruby. What does it mean when I say Rubinius takes the fun out of Ruby?

Uh, what?!

Yep, that’s it, no more fun in Ruby; it’s a syntax error. All those happy programmers bursting into tears…

Ok, it’s not all bad. There’s another side of this:

You’ll have to excuse the syntax highlighting. It’s behind the times. But your eyes are not fooling you. That’s really a thing that looks exactly like a method but defined with fun instead.

Totally not pulling your leg.

The output is a bit odd. It’s the compiler saying it got a request to compile a function, but that part isn’t done yet, so it’s just a confirmation that, “yep, this here thing is a function”.

Pretty cool, right? Wait, there’s more.

You’re probably way ahead of me here, but yes, defm enables defining a new method with the same name but different arity that joins, rather than replaces, the existing method. And funm is similar, but for functions instead of methods. Which method or function is invoked depends on the code calling the method or function.

You may be wondering, why does Ruby need defm? Excellent question. Let’s explore that using Array#[] as an example. In a more perfect world, Ruby would have this:

The Ruby core library tends to keep method names concise by overloading the meaning of the method based on the arguments passed. This can support very clear code, where the nature of the data (arguments) is an implementation detail. I love this aspect of Ruby.

In contrast, Rails would probably define this with long, explicit names:

I’m not bagging on Rails, and maybe they wouldn’t define it this way, but I want to make the contrast clear. Using a simple name and overloading the meaning based on the arguments can make code more robust. The code says, “I want a part of this Array” (the meaning of the code), and the arguments specify the implementation detail based on what data is flowing through that code. Really nice.

In contrast, the explicit method name approach couples those two. The name of the method limits the meaning of the code to the implementation detail. Not so nice.

The problem with simple names and overloading the meaning based on the arguments is that the method bodies become a huge mess of conditional code and utility functions. Don’t believe me? Go read the Ruby C code.

The solution to this problem isn’t to hide the mess. Rather, give the programmer the ability to both support concise method names that capture meaning and implement the “method meaning” with clear code.

That’s what defm enables.

It’s not all roses, though. Earlier I said that data, functions, and types are an excellent combination. That’s true. But making it true in Ruby isn’t easy.

Ideally, we’d be able to define data types much like we define classes or modules:

Ideally, sure. In reality, no. A quick search on GitHub shows tens of thousands of uses of data and data_type in Ruby code (so much for everything’s an object, eh?). Despite this data construct looking and working extremely well, if we introduced data as a keyword, effectively taking both fun and data out of Ruby, there’d surely be widespread revolt. So data must remain a 2nd-class concept in Ruby (for now).

Another concern is types. Types with functions are really great and I support them 100%, as an option. On the other hand, I’ll go on record saying that any attempt to add static types, or even gradual typing, to Ruby objects will be an unmitigated disaster. More importantly, it’s completely unnecessary with proper support for functions.

But what about my defm for Array example above? Didn’t I essentially use some type annotations there? Sort of.

This syntax compiles fine in regular Ruby today:

But it wouldn’t actually run (that defm feature for the Integer function would be handy here):

However, it wouldn’t be hard to make it work. Let’s expand on the syntax a bit to be sure the ideas are clear:

By considering these more as “behaviors” than as “types”, the coercion semantics are reasonably simple. A method may take an argument that is-a Integer or can be coerced to an Integer. The meaning of the code is explicit, and the existing dynamic typing is unaffected. In my mind, this is still uncomfortably close to adding types to objects, but it doesn’t work against the purpose of objects.

For functions, on the other hand, this same syntax can be used, and could even be checked “statically” for appropriate function definitions. Probably more useful, though, would be to define a set of data types that do not intersect the Ruby class/module space, and preserve this syntax for defining arguments that would be objects. To elaborate:

Aside from that pesky do syntax to work around not having a first class data type in Ruby, the code is reasonably natural and clear.

One final complication remains: where do these functions go? On the lexical scope object, naturally.

Just like you cannot directly access a block in Ruby, you cannot access a lexical scope either, but it exists and is involved any time you look up a constant. In Rubinius, the lexical scope is just another, you guessed it, Ruby object:

In Rubinius, functions are defined on lexical scope and ambiguous method invocations (e.g. puts “hello”) are resolved by checking the lexical scope first in cases where the function definition is not visible at compile time. When the compiler can see the function definition, it can emit a static call to the function.

This feature is especially useful for implementing the Ruby core library in Rubinius. There is no way to manipulate the lexical scope from Ruby. Hence, no Ruby code can change predictions that the compiler makes related to the lexical scope. These features make functions perfectly suited for the following tasks:

1. Field access. In Rubinius, core classes are written in Ruby and use “instance variables” for state, as one would expect of Ruby code. Unfortunately, Ruby programmers are accustomed to arbitrarily mucking with core classes by incompatibly redefining methods, removing methods, or adding their own instance variables. By using functions to access the fields (like instance variables but slightly different), external code cannot break Ruby core classes.
2. Math. Despite tremendous effort in Rubinius, there is a vanishingly small chance Ruby will ever be well specified. Things like the (now deprecated) mathn library arbitrarily redefine things like Fixnum division, without regard for code that depends on well-defined Fixnum division. Of the myriad places that “math” is used in the core library, in Ruby it is just C math, not Ruby math. So mucking with Fixnum is inconsequential in Ruby. But in Rubinius, mucking with Fixnum extensively breaks core classes. By defining math via functions, Rubinius can more robustly implement the Ruby core library.
3. Bytes. As mentioned earlier, bytes are well-defined. They are also extensively used. The flexibility of Ruby’s dynamic types are completely unnecessary for working with bytes, and the cost is prohibitive. In Rubinius, essential String operations can be implemented as functions, allowing for many optimizations while insulating them from arbitrary Ruby level modification in the same manner as math above.
4. Object coercions. Ruby has well-known coercions, like to_str, but unfortunately, they are not well-defined and many (or most) of them exist as numerous C utility functions in Ruby.
5. Utility functions. The object coercions are an instance of the more general category of utility functions for operations like formatting an object’s data a certain way for inspect or JSON or various other manipulations.

Lest you think I whimsically happened across the idea of proper support for functions in Ruby, it was the venerable Yehuda Katz who suggested them to me in late 2013 as a much more elegant way to address the concerns that spawned the abominable Ruby refinements. He was right (he usually is), but it took me several months to see just how elegantly functions fit in Ruby.

But this shouldn’t be surprising. The tension between the opposed forces of completeness on the one hand and minimizing exposure to uncontrollable change on the other requires a balance that is not achieved by lopsided “everything is an object” approaches.

Even worse than “everything is an object” is the attempt to mash together things like objects and types, whose fundamental natures are in opposition. Truly elegant solutions depend on proportion and collaboration between concepts like objects and functions that are fully, unapologetically themselves.

But wait, there’s more (there’s always more).

It’s fairly well known that every Rails application contains a poorly-specified, buggy, slow implementation of most of the Fibonacci function, which explains the prevalence of benchmarks comparing fib() performance. Given that, consider the following code:

Notice that despite looking like quite natural Ruby code, there’s nothing here that depends on anything in the Ruby core library. Code like this in a script should compile to a simple, stand-alone executable.

Perhaps not exactly fib(), but at least code like this should cover a fair number of scripting tasks. And generalizing from this to code that does rely on Ruby core classes or more elaborate Ruby code is possible, given the facilities Rubinius has built to manage information about running code. The possibilities are exciting.