Terra: Low-level system programming language meta-programmed from Lua

https://terralang.org/

by teleforce • 5 years ago

158 32 5 years ago

terralang.org

rektide• 5 years ago

Also runs in GPU via CUDA. That's it's well known claim to fame.

Appears to be going 1.0[1]!

[1] https://github.com/terralang/terra/commit/1ea278278c104580a0...

eslaught11• 5 years ago

We're working on 1.0. The language is basically stable but needs a little polish. As usual everything depends on what time people have to put in. Always looking for more help!

nerdponx• 5 years ago

Am I understanding this right?

Code inside a `terra` function is statically-typed and executed at runtime, while the rest of the code is just plain Lua and is executed at compiled time.

It's a really interesting idea and I'd be interested to try it for something like an X11 application.

hollerith• 5 years ago

I think it is more like this: the Lua code emits assembly code or LLVM intermediate representation or C source code, then arranges for the assembly code to be assembled (then possibly arranges for the code to be executed, then terminates).

Many years ago, someone wrote a Haskell library along the same lines that lets the library's user mimic assembly language with Haskell code. One thing I particularly liked about this library is that (by deviating a little from the standard way of indenting Haskell code) the user could mimic the 2 columns of assembly language. E.g., in

  foo <-  statement
          statement2
  bar <-  statement3

"foo" and "bar" would be a labels, and that mimics the look of assembly language in that the leftmost 15 or so characters (columns) in an assembly-language file are usually reserved for labels. (I.e., if a line has no label, those 15 characters are blank.)

eslaught11• 5 years ago

You can interleave Terra and Lua execution. Terra functions are JITted if you call them from Lua. You can call Lua from Terra too, though the language makes you jump through some hoops to encourage code to avoid Lua dependencies in Terra code. So "execution" is more like you'd expect from standard Lua (or Python or whatever), it's just that functions can be compiled if you want.

Of course Terra functions can also be saved out to a file too, and if you do that they're free of any dependency on Lua (i.e., no need to like liblua*.so).

hkt• 5 years ago

I've been following this for a while and think it is a gorgeous idea. Particularly nice is the fact that one can still use C and C++ libraries from lua, then dip out into Terra for things that require more speed that don't currently have a library.

Lua tying it all together means you have a tiny, dynamically typed language that can be embedded in lots of popular C based software, which can now be accelerated to (even more) incredible speed. I predict big things and wish them well.

w_t_payne2• 5 years ago

Is there anything similar for Python?

canadianfella• 5 years ago

Maybe this

https://pypi.org/project/TMPPy/

w_t_payne2• 5 years ago

Ooooh! Thank you!

test9753• 5 years ago

A Singly Linked List implementation in Terra (just testing out):

https://pastebin.com/xt3Wddtu

Output:

https://pastebin.com/KwSTiN8L

aeldidi• 5 years ago

I remember reading something particularly negative about terra in 2019[1]. Although I haven’t used terra at all since the time I first tried it (not long after I read the above mentioned article), I think it’s relevant to mention that at least at one point, terra has had major problems regarding design. If anyone has used terra in any non-trivial capacity I’d love to hear what your experience with it has been like.

[1] https://erikmcclure.com/blog/a-rant-on-terra/

blackhole• 5 years ago

Please note: The entire "windows" portion of that article has largely been resolved at this point, because I fixed it myself. Some of the more egregious documentation problems have been fixed, but others remain unresolved.

Unfortunately, we have since discovered even more insane behavior: https://github.com/terralang/terra/issues/460

neolog• 5 years ago

Why is that called non-euclidean?

LegionMammal978• 5 years ago

Thanks for pointing that issue out! I had a fun time figuring out the codegen algorithm by putting printf statements in every function I thought was relevant.

eloff• 5 years ago

I used terra at least 6 years ago to create a JIT compiler for database queries.

The language itself is performant and pleasant, but there are not enough people using it, so troubleshooting is really painful.

eslaught11• 5 years ago

I develop Regent, a programming language for distributed computing based on Terra [1]. We're one of the larger (largest?) Terra codebases and indirect source of users.

(I also help maintain Terra, but this answer is more focused on my capacity as a user.)

Terra is a workhorse and it gets the job done. Overall, compared to a previous iteration of Regent (that was built entirely from the ground up in Python), it's been a good experience. Having first-class code generation capabilities has been really nice, especially with built-in support for things like parsing C header files, vectorization, and CUDA code generation. The language is stable and generally doesn't have many surprises. Unlike, say, C++ or Rust, there aren't as any corners to hide odd behaviors. (Obviously, there still is some room for such things, as you can see in the sibling comments. But I still think it's better than most languages.)

The main issue is, as siblings have noted, the size of the community. While most stuff works, if you find something broken you're likely going to have to fix it yourself. This is an area where we're always looking to grow.

[1] http://regent-lang.org/

tooltower• 5 years ago

I wonder how well it handles or rejects syntax-level expansions. E.g. can the outer language produce half an expression for the next level? Do I need to worry about the common pitfalls of macros?

I hope this works out. C++ has been deliberately evolving in the same direction, e.g. using `if constexpr` for conditional compilation. Plus, since the C++ compiler is aware of both levels of the language, it can catch some errors even for compilation paths not taken.

But the C++ syntax for metaprogramming still leaves a fair bit to be desired. So I like that folks are still experimenting in this area.

eslaught11• 5 years ago

Terra macros are hygenic in the sense that they follow the Lua scoping rules. So if you do:

    local add1 = macro(function()
      return `x+1
    end)
    terra()
      var x = 12
      add1()
    end

Then this will give you an error because x is not in scope in the add1 macro.

Instead you'd have to write:

    local add1 = macro(function(y)
      return `y+1
    end)
    terra()
      var x = 12
      add1(x)
    end

Note that I changed the variable name to clarify that only the lexical scope matters (i.e., "x" is a symbol passed to the function as argument y).

You should be able to create towers of languages in Terra. I'm not sure if anyone really does more than 2 levels in practice, but most of the common Terra DSLs are basically written as code generators on top of Terra language expressions.

See e.g. the Regent code generator. Here's an example of some nested metaprogramming---not with a nested language or macros, but with an escape that calls a Lua function to generate more code based on the type of the argument:

https://github.com/StanfordLegion/legion/blob/0cf9ddd60c227c...

FranchuFranchu• 5 years ago

It's like Rust's procedural macros, but more flexible because they're interpreted, right?

eslaught11• 5 years ago

Well Rust procedural macros can run arbitrary code, so that's not such a big difference. I'd say the larger one is having access to actual AST data structures available if you want them. Rust forces you to parse the token stream yourself and serialize it again. Also, Terra allows user defined keywords so it's a lot easier to create nice looking DSLs (which is my personal use case).

adrusi• 5 years ago

You can think of it as clang, but controlled by a lua engine instead of the CLI. It's not just macros, you have control over generation of build artifacts. By default you're running in JIT mode, but you can at any time bundle up some procedures and spit out an object file or link them into an executable.

You can also call back into Lua from the native code, and use it to generate more native code. You get back some of the cool self-modifying executable techniques of the days gone by, without having to completely YOLO it.

a9h74j• 5 years ago

Well-presented in the OP.

Dumb question: It was not clearly instruction-set specific; could Terra compile to webassembly? Would one want to?

eslaught11• 5 years ago

We use LLVM, so hypothetically yes. I don't think anyone has tried though.

eslaught11• 5 years ago

Hi HN,

I help maintain Terra, and also use it to build Regent, my distributed programming language[1].

A bit busy at the moment, but if you leave questions here I'll try to answer them (possibly in some hours).

[1]: regent-lang.org

eslaught11• 5 years ago

In case anyone is still reading this, I'm here now. Still happy to answer any questions.

I'll also add that we're always looking for new contributors. We're an 100% community run project, so everything is driven by people who care about things stepping in to do them. If you're interested in compiler technology, there's a wide variety of things to work on.

reitzensteinm• 5 years ago

I'm curious if it a new syntax was ever considered. It seems as though if you dropped Lua compatibility you might be able to make it a bit cleaner, although it's a big undertaking.

joiguru• 5 years ago

Honest question. Why not just use C/subset of C++ with a template language like Python+Jinja2 for meta programming?

Code generation is already popular with popular C++ libraries like protobuf and in game dev.

eslaught11• 5 years ago

When you use a template language, how do you JIT the code so you can run it in the same process?

How do you parse C/C++ header files so you can introspect their types or functions and take actions based on the results?

How do you write clean abstractions for code generation? Do you build your own C++ AST abstraction entirely from scratch? If not what do you use, and how much power does it expose (and what tradeoffs come with the level of power that you get)?

Terra lets you do all of these things in a clean way with well-defined, predictable semantics that don't leave you scratching your head when you get to 4 nested levels of macros and complex dynamic behavior that depends on program inputs.

eigenspace• 5 years ago

What is the advantage here of doing the metaprogramming in a different language from the one you're manipulating?

eslaught11• 5 years ago

Different languages help because the different parts of the program expose different needs.

You want the generated code to be low-level, because you want it to be fast, free of dependencies (interpreter, JIT, GC, etc.), and you care less about abstractions. You want the metaprogram to be high-level because you want nice abstractions, and it runs at compile-time anyway so you care less about performance.

Sure, people are working on languages that can do both, but it's easier to make it work with two different languages that can be tuned specifically to their niches.

Terra: Low-level system programming language meta-programmed from Lua

Related Stories