eregon

Many native extensions just work with TruffleRuby, I'd estimate even the majority of the popular native extensions just work. Performance of native extensions is not always good, especially if they do a lot of back and forth between Ruby and native.

To give one example, DB adapters work just fine.

No it wouldn't.

Thanks for the comment.

Cool, but what about replacing the regexp with straightforward parsing code written manually?

If you take a look at the linked snippets of C code, I think it's clear it's all but straightforward. The regexps OTOH are really short and expressive.

Ruby has no access to SIMD. And writing SIMD is basically writing inline assembly, so it's really tedious and messy.

relying on the compiler doing autovectorization to make it fast.

I can relate to that, but Regexps are a much smaller domain, and there it's clear SIMD is always a win, so if the regexp engine uses SIMD it's very unlikely it would ever stop using it (unless something faster comes up).

When did you try? Fixnum/Bignum are gone since many years (before 1.0RC1 which was April 2018, it's all Integer now). So I guess many years ago, and back then TruffleRuby was basically implementing Ruby features, had fairly limited compatibility and likely could not run Rails, very different from today.

There were compatibility issues with BigDecimal, but TruffleRuby now uses the C extension, hence it should be exactly the same behavior as CRuby.

Numerical to string formatting is a bit different too.

AFAIK that was fixed years ago if you mean float formatting.

Regexes behave differently if they worked at all.

TruffleRuby always used Joni, which is literally a translation of CRuby's Regexp engine to Java (by the JRuby team), so that is very surprising and I have a really hard time to believe it. At least "Regexes behave differently if they worked at all" seems harsh and highly inaccurate to me. There likely were a couple Regexp issues but the generalization seems wildly exaggerated.

Yes, some things are slower. Most of them I'd say are unexpected performance bugs. I'd think most are easy to fix once investigated, but some can be hard to fix (recent example, `File.read` is quite fast on CRuby).

Some things are expected to be slower, for instance constantly redefining (monkey-patching) methods or constants is slower on TruffleRuby, but that's typically because the program is broken and so it'd be slow on CRuby as well.

Compatibility is certainly an issue for massive codebases, OTOH I think over time TruffleRuby is getting closer and closer to CRuby behavior to the point it would be fairly rare to find a compatibility issue. Do you have personal experience trying to run such a codebase on TruffleRuby?

Startup is likely to be worse, because it typically runs a lot of different code for not long and the JIT might not have enough time to optimize much of that. OTOH the JIT needs to learn what the program is doing, i.e., profile it in a sense, and that has a cost on interpreter speed. Still these numbers are worse than expected, especially [object validation], so if there is a way to reproduce it'd be great if you can open an issue about it.

Persisting the JITed code is what we think can solve the slower startup entirely: https://www.graalvm.org/graalvm-as-a-platform/language-imple... Also other things mentioned in https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-yj...

That second sentence seems a fair assessment. TruffleRuby does support many native (C/C++) extensions so that's rarely an issue. But indeed in such a large codebase it's likely to depend unexpectedly on CRuby-specific behavior. And while that can be fixed it takes some effort either in TruffleRuby (to match CRuby) or in the app (e.g. to avoid relying on `RubyVM`).

Compatibility is one, it's hard to be 100% compatible with CRuby, and large codebases tend to sometimes depend unintentionally on weird behavior or even bugs in CRuby.

Keeping up compatibility (while keeping things efficient) is a lot of work, TruffleRuby tries to reduce that by reusing as much as possible existing code, including reusing C extensions shipped with CRuby.

Actually TruffleRuby optimizes metaprogramming more than any other Ruby implementation. Rails is not the issue, big codebases is the issue: more code = more chances to use something which is not so fast on TruffleRuby yet = more chance to become a bottleneck and degrade the overall performance.

It's the same benchmark in the blog post, `railsbench`. So TruffleRuby already speeds up some Rails apps like that one but not every Rails app/program. (TruffleRuby 3.27x, YJIT 1.33x on railsbench)

Interesting, I did not know that just calling the original method, even when the refinement method is not used occurs such a cost on CRuby. Seems worth reporting if not already done. No such thing on TruffleRuby though, the only peak performance cost would be for megamorphic calls (rare, even more so in combination with refinements).

Many popular C extension gems just work. About the only popular C extension I can think of which does not work yet is grpc, and there is work on that. There is a status overview at https://github.com/oracle/truffleruby/blob/master/README.md#...

Here is a bit of a critical read of this blog post, from the point of view of alternative Ruby implementations:

instead of having to ship an entire language runtime to production

Except they ship the CRuby language runtime in production. Ruby is not a language that can run without a runtime.

Not only did we not need Java VM-level interoperability

So they almost discard the entire idea because they don't need a specific additional feature?

choosing either alternative Ruby implementation would have made for a difficult migration path.

So what is it?

Stripe relies heavily on gems with native extensions

Yes that's a problem on JRuby (when there is no java extension in that gem), but TruffleRuby supports native extensions, as very clearly stated in many places.

as you can imagine, a multi-million line Ruby codebase over time starts to depend on Ruby-the-implementation, not just Ruby-the-language.

Except all serious Ruby implementations know they need to be compatible with whatever CRuby does, not just an incomplete ISO specification of the language. In fact alternative Ruby implementations match CRuby behavior as much as possible for compatibility, even when it seems weird or makes little sense (they report it in this case but have to match behavior anyway).

TruffleRuby might not be 100% compatible with CRuby yet, but I would say it is [pretty close](https://eregon.me/blog/2020/06/27/ruby-spec-compatibility-re...).

To adopt JRuby or TruffleRuby in Stripe’s most important services, we’d have to be able to run all the code or none of the code [of a service].

Well yes, to get significant performance gains, one needs to try new things. Don't they have a staging environment where they can do experiments?

I've also found that the Computer Language Benchmarks Game benchmarks tend to be less and less representative of real workloads, due to what the benchmarks do and to the philosophy of "write it to make it fast, not idiomatic, and parallelize things that would typically not be parallelized". Just taking a look at the benchmark code makes it clear it's very far from idiomatic Ruby code.

Interesting results, thanks for sharing, that matches my expectations. OptCarrot is probably one of the rare cases where Ruby 3.0+JIT is 3x Ruby 2.0. On most benchmarks, the gains are much smaller.

I'll try to run these benchmarks on TruffleRuby (and maybe JRuby) too, would be an interesting comparison.

BTW, is there any reason you used 2.1.0 and not 2.0.0? Maybe some issue with 2.0.0?

I replied to the tweet. At least for TruffleRuby it's to improve testing, no hidden intent.

And the PR is by someone not at Oracle, and his interest is to run some Rails app on TruffleRuby, and for that he'd like to ensure all gems work fine on TruffleRuby, which seems a very legitimate thing to do, isn't it?

TruffleRuby [can run Rails](https://speakerdeck.com/eregon/running-rack-and-rails-faster...), if that is what you refer to.

That's rather vague. But yes, no matter which JIT you always need some extra memory to run the JIT, and it creates a more optimized version while also needing the unoptimized version of the code, so it needs more memory.

It's exactly the same thing though, isn't it? I use both terms interchangeably. And it seems the term GIL is better known than GVL.

Which comments?

Seems there isn't any warmup accounted for: https://github.com/kostya/benchmarks/issues/246

FWIW, I ran bench.b locally: MRI: 87.83s TruffleRuby Native 20.0: 18.82s truffleruby-dev Native: 14.09s

Why is there is no TruffleRuby Native in that page? (i.e., the default)

Concretely, TruffleRuby never shared much in terms of the core language implementation with JRuby (essentially the parser and encoding stuff, never core methods/operators/etc).

JRuby has no support for C extensions.

TruffleRuby supports C extensions, and has no GIL for Ruby code.

Guilds would have a major performance problem: can't allocate objects without GIL.

Why not? It'd be possible to have TLABs, isn't it? But yes, GC would still be for all Guilds at once.

Racket places don't allow to share objects, only arrays of primitive types, which seems very restrictive. And Racket futures are even more restricted.

I'm not sure about Guilds since it's not there yet, but it already sounds closer to Ruby multithreading and more flexible to me from an usage point of view.

Do you mean every thread would have its own place/isolate for executing C extensions?

I think one difficulty there is some global state in C extensions might expect to be truly process-global.

Also, how would you isolate global variables in the C extension? If the C extension is a dynamic library, it's typically only loaded once per process. Maybe something like dlmopen() to load multiple copies of a native library would help, but it's not portable.

Thanks for the praise on ruby/spec :)

OTOH, I'm sure there are such bugs where the specs don't cover every corner-case behavior. Specs are added when finding bugs in alternative implementations and following MRI's NEWS for new features, but there is no guarantee specs are complete.

Right, I should just rename it to "Classic benchmarks", since it's also from other sources.