hugomg

We figured that it wasn't worth dealing with the hassle of unaligned addresses because the more portable alternatives worked just as well.

tl;dr: For Pallene, Lua is a scripting language. For Terra, Lua is mainly for metaprogramming.

The key difference is that Terra cannot directly manipulate Lua data. Although Terra's syntax is similar to Lua, its type systems and data structures are actually much closer to C. If we want to pass Lua table into Terra, we must first convert it into Terra arrays/structs. This cost adds up if the code uses many Lua data structures; in the worst case it can be worse than just using Lua for everything.

Passing data between Lua and Pallene should incur no cost. If the programmer rewrites a piece of code from Lua to Pallene, it should hopefully make things faster, and never slow it down.

Terra is more focused on metaprogramming. Generating code and executing things at compile-time. Think of C++ templates, but better.

The difference is the Lua-C API. The default Lua-C API is designed for humans: it is stable and safe to use, but every operation must pay the cost of function calls and passing data through the Lua stack. Pallene bypasses the API and reaches into the Lua tables directly. This is much faster, but would be impractical without the Pallene compiler. The internal struct layouts are unstable, and unsafe if you're not very careful.

I can just as good implement it in C using the Lua C API

Pallene beats C when the code uses many Lua data structures. Acessing Lua data from C via the Lua-C API has significant overhead that can erase the gains from rewriting into C. Also, rewriting from Lua to Pallene is much less work than rewriting it in C.

Although Pallene is only a subset of Lua, the idea is that you use it together with Lua. It's not meant to replace Lua entirely.

(I'm one of the Pallene authors)

For compiled code, Pallene's performance can be comparable to LuaJIT[1]. Pallene might be better in code with unpredictable branches, which don't fit in a single trace. LuaJIT has the edge in purely-interpreted code, and can inline indirect or cross-module function calls. LuaJIT is also more featureful, and has a better FFI story at the moment.

[1] https://www.inf.puc-rio.br/~hgualandi/papers/Gualandi-2020-S...

LuaJIT is really good software, it is hard to beat it as its own game. Pallene tries to get its edge elsewhere: it tracks Lua 5.4, whereas LuaJIT diverged around 5.1 and will stay that way forever. Pallene's implementation is arguably more portable, because it compiles down to C and doesn't need hand-crafted assembly.

Pallene and Teal are in a bit of a different space. Teal is focused on compile-time type checking. The intention is to encourage you to add type annotations to your entire Lua codebase. It's another take at the problem that Typed Lua was trying to solve. A Typed Lua 2.0, if you want to call it that way.

Pallene, on the other hand, is more focused on performance. The tradeoff is that it doesn't cover the entirety of the Lua language, just the features that we can generate fast code for. The intention is that you can write the performance-critical parts in Pallene and glue everything together using Lua (or Teal).

That said, I certainly wouldn't be surprised if we saw more cross pollination of ideas between these two languages in the future :)

It wouldn't be the JS we know and love if it had been burdened with a type system designed by a committee sometime in the 90s. That said, one thing we can say for sure is that the dynamic typing doesn't make your job any easier :)

I always love reading more about JavaScriptCore internals although I have to confess that much of the time one of the main lessons I get from it is that life would be much easier if we had types and didn't need to speculate so much in the first place.

If you are curious, this is what the resulting format string looks like after expanding all the macros:

https://gist.github.com/hugomg/73e91ebca7ced3c5c6f955e9e830b...

We chose the Pallene name because it was another one of Saturn's moon names that sounded nice, and is pronounced the same way in English and Portuguese. The Mike Pall thing was just a coincidence although we certainly can't deny his influence, as you have said :)

It is no surprise that the benchmarks on the LuaJIT website are going to show it ahead of PUC-Lua, is it? In truth the performance comparison is a bit more complicated than that. :)

For a pure Lua comparison LuaJIT is typically going to beat Lua by a large margin. For code that can be JIT compiled you might see ~10x performance improvements and for code that doesn't you might still see around a ~2x improvement because the LuaJIT interpreter is written in hand-crafted assembly language (PUC-Lua sticks to portable and standards-compliant C90).

But things get a bit more complicated if you add foreign C code into the mix. Under LuaJIT, the JIT compiled code that uses the LuaJIT FFI interface is the fastest, but interpreted code using the FFI is slower than interpreted code using the traditional Lua-C interface, and can even be slower than PUC-Lua using the traditional Lua-C interface. This means that when you are using LuaJIT you need to be very careful to make sure that your code is the sort of code that can be JIT compiled. If you hit a "not yet implemented" feature you can have a big performance degradation.

------

Nevertheless, Lua 5.4 has brought significant performance improvements compared to Lua 5.3, specially in integer operations (including for loops). Another big change was that the `#` is now faster. For example, on my machine the following loop runs in 1.96 seconds in Lua 5.3, in 1.26 seconds under LuaJIT, and in 0.96 seconds in Lua 5.4

    local a = {}
    for i = 1, 1e7 do
        a[#a +1] = i
    end
    print(#a)

In case others aren't familiar with the termiology: #SAT is the counting version of the SAT problem, which tries to find how many different solutions exist to a set of constraints)

Some SAT solvers allow you to add additional constraints while they are running. So you can find a solution and then add additional constraints to try to get it to find a better one.

Generally speaking, if you ever find yourself coding a brute-force backtracking search for some combinatorical problem, it might be a good idea to consider using a SAT solver instead. This way you get to take advantage of the non-chronological backtracking, and all the other tricks the SAT solvers implement.

I wrote the hexiom solver that got linked above and the motivation in that case was that I saw that someone tried to solve the puzzle with a hand-written backtracking search, but that their algorithm wasn't able to find a solution to one of the levels.

My version of the SAT solver managed to quickly find a solution for every case, including the highly symmetric one where the usual search algorithm got stuck. The nicest thing is that the implementation was very declarative, in that my job was to produce a set of constraints for the SAT solver, instead of to produce an imperative algorithm. The symmetry breaking was one example of something hat was very easy to do in a SAT-solver setting but which would be trickier to do by hand.

Seeing someone cite a little project I did for fun many years ago is a very nice surprise. :)

Yes.

Currently there is no special support for metaprogramming, but maybe we could borrow some ideas from Terra in a future version...

Wow, I hadn't thought of that. Its a complete coincidence, actually!

From the point of view of Lua, Pallene works the same as C

If you want to produce a standalone executable, at the end of the day this executable will contain a copy of the Lua VM.

If you want to produce a library that can be "require"-ed from Lua, then you will produce a ".so" shared object file with just the Pallene functions in it.

If you were writing naive code, it probably wouldn't be that much. Maybe a 2to 3 times difference. But the thing with Terra is that you can be even faster than naively-written C if you know what you are doing.

I talk a bit about this in a sibling comment. Maybe that helps answer your question?

Pallene uses garbage collection, and shares the Lua garbage collector. The idea is to make as easy as possible to call Pallene from Lua and Lua from Pallene. Meanwhile, Terra only uses Lua at compilation time for metaprogramming and template building. At runtime, if you want to call Lua from Terra (or vice versa) you need to use the C API, just like you would do with C.

I wouldn't really try to compare the performance of Pallene with Terra. They are two very different languages, doing different things.

You hit the nail on the head. With a more "Oberon-like" language we can use textbook optimization algorithms and take advantage of existing compiler backends, like GCC and LLVM. For dynamic languages these techniques aren't enough so more sophisticated JIT compilation seems to be the only option.

Don't worry, that is still a fine question. Pallene programs are still able to use Lua's "load" to eval dynamically-typed Lua code. But it is less likely that we would implement a function to load Pallene source code at run-time, for the same reasons why it isn't common to dynamically load C source code: Pallene takes longer to compile than Lua, and dynamically-loaded code usually isn't as performance-sentitive anyway.

Pallene has been designed to seamlessly interoperate with Lua, not to replace it. So you can still use Lua when you need dynamic code loading or magic debug hooks.

Roughly speaking, from the point of view of Lua, Pallene is a system language like C. It is good for performance and calling C system libraries, but isn't particularly suited for dynamic code loading and sandboxing via interpreter hooks.

Terra uses Lua as a metaprogramming language, while Pallene/Titan use Lua as a scripting language.

Terra is designed for high performance numerical computing. As you mentioned, it has no garbage collector, uses C-like datatypes and its executables are completely detached from the Lua runtime.

Pallene, on the other hand, is all about being able to seamlessly interoperate with Lua at run-time. It is possible to directly manipulate Lua tables and functions from a Pallene program. From the point of view of Lua, Pallene modules can be loaded as a C extension module would, except that Pallene is designed to integrate even better with Lua (sharing its garbage collector and having better C API performance).

As the other commenter pointed out, we are currently compiling down to C. However, it is possible that we could switch to LLVM IR in the future.

Yup, this is the idea. :)

If you don't mind, I'm curious about what applications you currently use Lua for.

Pallene is indeed approaching this problem more for performance than for autocompletion. (For just autocompletion it would be more appropriate to use a typesystem that is closer to full Lua, like Typed Lua)

For us, the main motivations for the Pallene approach are performance predictability and implementation complexity.

To get maximum performance with a JIT you need to write your code in a way that ensures that the control flow always stays inside the compiled parts of the code. If you are accidentally "too dynamic" then the JIT will fall back to the general bytecode interpreter, with an order of magnitude slowdown. The Pallene type system should ensure that if your code typechecks then it can be compiled to something reasonably efficient.

The type system also lets us use a simpler ahead-of-time compiler instead of a just-in-time one. JIT compilers are notoriously tricky to implement, which in turn means that it is more difficult to evolve them together with the language, or to port them to additional platforms.

Gabriel's slides are already available here: https://gligneul.github.io/luaworkshop2018/

I'm first author of the linked paper and am working on Pallene for my PhD. Feel free to ask any questions here!