One year of Roto, a compiled scripting language for Rust

https://blog.nlnetlabs.nl/one-year-of-roto-the-compiled-scripting-language-for-rust/

by Hasnep • 2 months ago

130 31 2 months ago

blog.nlnetlabs.nl

Does anyone know if the Roto runtime is serde-able?

A big problem I encountered in using Lua in Rust for my game engine was that I wasn't able to serde the Lua runtime such that I can snapshot a game session and save it in a file, and retrieve it in another context.

terts8• 2 months ago

Hi! Author here. What we call the `Runtime` in Roto is not a state of the program, it's only the set of functions, types and constants that are available to the script. Roto scripts cannot really keep state at the moment. The advantage of that is that it allows you to run scripts in parallel. We're thinking about how we can keep that property while also having some state.

pveierland2• 2 months ago

The syntax is of course attractive (coming from Rust), and I'd love to replace more of my posix scripts with something saner. I struggle understanding whether the utility of having language literals for IP addresses, IP prefixes, and AS numbers is worth it though [0]. It seems like the confusion added by having custom built-ins like this for one particular domain, in addition to the unclear scoping (what could later also deserve being a language literal), combined with special-case errors as famous in e.g. the YAML Norway problem, makes it seem like such features are better left as some general extension / macro / library capability.

Nix is a language with built-in support for URI literals typed as strings [1], which is a source of confusion and edge-cases, and I believe the feature is now discouraged in general use.

[0] https://roto.docs.nlnetlabs.nl/en/stable/reference/language_...

[1] https://nix.dev/manual/nix/2.34/language/string-literals

terts8• 2 months ago

Hi! Author here. We are actually planning on removing those literals and allowing applications to extend Roto with their own literals [0]. They should do so with care of course, because indeed adding more literals adds some edge cases. Most applications should be able to get by without any special literals though.

[0] https://codeberg.org/NLnetLabs/roto/pulls/358

pveierland2• 2 months ago

Nice! That sounds like a good change. I'll try to dive a bit deeper through docs once I find some time :)

berkes2• 2 months ago

Have you considered collecting all the literals into domains, but ship them by default?

I could, for example, imagine using roto in some of my current work on svg and visuals generation. In which case I'd be greatly helped with literals like "colors", "vec2", "angle" etc. I'd imagine that as long as other literals which I don't need, like an IP address, aren't in the way, it's still greatly beneficial to have a large lib to pick and choose from, around.

everforward• 2 months ago

That makes a lot of sense to me. The implementation feels odd to me, though. If I’m reading it right, I type in literals normally and then all these hooks decide whether they want to change what I’ve typed in? It feels like I have to remember the custom literals I have installed to make sure I don’t accidentally conform to some spec and end up with a custom literal instead of the string I wanted.

Something like EDN readers seem saner to me where I wrap the value in something that denotes the function to use to parse the value. If I do “192.168.1.0/24” I get a string literal, if I do #cidr{192.168.1.0/24} then it hands the value off to the cidr custom literal.

That’s my 2 cents, I hate when things implicitly modify my literals.

adsharma• 2 months ago

7 years of python -> rust here.

https://github.com/py2many/static-python-skill/blob/main/ass...

ianm218• 2 months ago

Very cool! I have been working on scripting in Rust recently on a Lua project [1].

When you made Roto what kind of workloads were you optimizing for? How are you guys benchmarking performance?

I ran a quick benchmark based on my recent work (Used AI for the code here): ``` fn sum_scalar(n: u64) -> u64 { let total = 0; let i = 0; while i < n { total = total + i; i = i + 1; } total }

  fn sum_list(xs: List[u64]) -> u64 {
      let total = 0;
      for x in xs { total = total + x; }
      total
  }

```

Rust benchmark.rs ```

  use std::time::Instant;
  use roto::{List, Runtime};

  fn main() {
      let rt = Runtime::new();
      let mut pkg = rt.compile("bench.roto").unwrap();
      let sum_list   = pkg.get_function::<fn(List<u64>) -> u64>("sum_list").unwrap();

      let n = 1024;
      let iters = 50_000;
      let xs: List<u64> = (0..n).collect();

      let t = Instant::now();
      for _ in 0..iters { sum_scalar.call(n); }          // adds 0..n with a counter
      let scalar = t.elapsed();

      let t = Instant::now();
      for _ in 0..iters { sum_list.call(xs.clone()); }   // adds the SAME 0..n from a List
      let list = t.elapsed();

      println!("sum_scalar (counter):   {scalar:?}");
      println!("sum_list   (List[u64]): {list:?}");
      println!("-> {:.0}x slower", list.as_secs_f64() / scalar.as_secs_f64());
  }

```

  Output:
  sum_scalar (counter):   28.56ms
  sum_list   (List[u64]): 590.48ms
  -> 21x slower

I'm happy to cut a PR against your repo with some of the benchmarks I run on every commit in my own language projects if that would be helpful! [1]. https://github.com/ianm199/lua-rs/tree/main

terts8• 2 months ago

Thanks!

When you made Roto what kind of workloads were you optimizing for?

We're building a BGP collector with custom filters written in Roto. Imagine a database that constantly receives updates and we want to filter (or transform) those messages based on a script.

How are you guys benchmarking performance?

Actually, we haven't done that much as feature work has been more important than optimization. There's a lot of opportunities for optimization left on the table.

There are a few benchmarks that we have done: - A very naive fibonacci computation, where we were faster than Lua, - There's this benchmark with a lot of string manipulation made by somebody else where we roughly match Lua: https://github.com/khvzak/script-bench-rs - There's the testing done with Iocaine, where Roto is apparently much faster than Lua. The scripts there do a lot of inspection of fairly simple types.

So the nuanced take is that Roto is fast with numbers and other cases which don't involve complex data structures that some other languages have really optimized for.

I'm happy to cut a PR against your repo with some of the benchmarks I run on every commit in my own language projects if that would be helpful!

That would be very helpful! A proper benchmark suite is long overdue. (but do note that we don't accept AI contributions)

sum_scalar (counter): 28.56ms sum_list (List[u64]): 590.48ms -> 21x slower

I think the list is so much slower it's calling out to Rust a lot to get items from the list. Lists currently also have a mutex inside, which would need to be locked for each access.

junon• 2 months ago

Maybe the authors are here to answer this.

The syntax is of course very Rusty, which is cool. However, a sort of obvious question comes to mind - what is the benefit of this over just writing rust, then? Just because the compile times are shorter?

EDIT: should mention I understand why embedded scripting languages exist, having embedded Lua many times. And I love a lot of these features, but to me having an embedded scripting language should simplify the language/API surface area instead of mirroring it almost 1:1. That's what I'm a bit undersold on.

Philpax• 2 months ago

Hot-reloading. You can edit your logic without rebuilding and restarting the host application; this cuts your iteration time from minutes to seconds, especially if the application is in a state that would need to be recreated.

abc42• 2 months ago

Same reason why several projects have integrated Lua to their runtime over the past 30 years. Extensibility and hot reloading.

berkes2• 2 months ago

I'm author of a rust based task manager (not (yet) FLOSS, unfortunately), where we needed "pluggable task sources" (jira, github, trello, etc).

In our setup, the "sources" are more like configuration. Whereas the core, the business logic, is more like code.

Typically, one would configure with e.g. YAML. As we can see in many projects, that have a DSL, in yaml (k9s, GitHub actions, ansible, etc). But, rather than inventing another DSL in yaml, we realized we do need some logic, something very poorly expressed in yaml. And we went for Lua.

Long story to say: if your config typically has some logic in it, it makes sense to go for an embedded scripting language to provide it, rather than building it into the core domain, or to invent yet-another-yaml-amalgation (yayamla?)

terts8• 2 months ago

Other commenters have given most of the reasons already, but since you asked specifically for the author, I'll chime in as well.

The fact that Roto gets compiled at the runtime of the Rust application is very important. That means we can ship a binary and still allow scripting.

We also believe that Rust is too complicated for our use case in some respects, we're trying to make something simpler. Our target audience for Rotonda is not people who necessarily know Rust. We can never be as simple as Lua because of the static typing, but we're trying our best.

And finally, we don't have to ship the entire Rust toolchain with our application. Roto is fully embedded into the binary with no external libraries needed and that's quite nice in practice.

shevy-java• 2 months ago

    fn contains(range: &AddrRange, addr: &IpAddr) -> bool {
      range.min <= addr && addr <= range.max

Looks ugly as fudge.

Syntax is not everything, but it also shows that people too easily think they are great at language design when they really aren't. It's fascinating to watch how people continue with such an approach. How many people are going to use that over, say, python?

ModernMech• 2 months ago

  def contains(range_: AddrRange, addr: IpAddr) -> bool:
    return range_.min <= addr <= range_.max

I don't get it, how is that much better?

hellzbellz123• 2 months ago

you missed the closing bracket.

glad you like python, but a good reason to use this is setup being easier, also for people using rust, chances are the syntax is better compared to python. (also for what its worth, i picked up rust MUCH faster than i ever did with python)

the main reason i like rust is its explicitness in typing along with its syntax choices. memory safety means little imo, outside of being difficult to do strange stuff (which could be good or bad depending on your approach)

cargo add roto

code main.rs

-FILE- main.rs use roto::*;

fn main(){ roto::init_runtime() roto::load_script("hello.roto") }

-FILE-END-

code hello.roto

-FILE- hello.roto fn main() { print("Hello, world!"); } -FILE-END-

cargo run

terts8• 2 months ago

Hi! Author here. You're looking at Rust code there. Roto is very similar but without `&`. I wont pretend to be good at language design, but being similar to Rust has many advantages.

jeorb• 2 months ago

Are there any comparisons to other Rust scripting languages like Rhai and Rune?

ryanshrott• 2 months ago

Rhai is dynamically typed and uses reference counting, Rune has a full VM with a borrow checker, and Roto compiles to bytecode. The tradeoff is basically faster iteration with Rhai vs faster execution with Roto, with Rune sitting somewhere in the middle but feeling heavier.

zengid2• 2 months ago

khvzak has compiled some benchmarks. mlua and roto are some of the fastest: https://github.com/khvzak/script-bench-rs

zengid2• 2 months ago

I've been thinking a lot about a solution to Rust's notoriously slow compile times. From my research, it's not the borrow checker that is the issue, it's the heavy optimizations and monomorphization, and macro expansion [0]. There are efforts to improve Rust compile times, but I realized recently that it will never be lightning fast, so we (Rust users) should really be developing more scripting languages and DSLs that we can put all of the fiddly bits into.

Places where we would benefit the most from this is in the Games and UI space. I know game devs have already started by integrating lua, like with mlua [1]. In the UI space i think Makepad is the best example of a team making a dedicated DSL that can be hot-reloaded [2].

I think we need more of this! Go make a DSL next time you feel crushed by the weight of compiling Rust crates!

---

[0] and by my research i mean Claude. this is a great blog with many posts about improving compile times https://nnethercote.github.io/

[1] https://crates.io/crates/mlua . I don't have a reference for a project using it though so please reply if you know of one!

[2] https://github.com/makepad/makepad

gdcbe• 2 months ago

Congratulations terts and team. In November’25 we had pleasure to interview you about roto and Nlnetlabs [1]. Happy how far it has come already. Before the summer ends We’ll implement support for it in rama [2] to support scripted services and anything else you might want! Looking forward to that day. Until then, keep it up!

[1] https://netstack.fm/#episode-14

[2] https://github.com/plabayo/rama

virtualritz2• 2 months ago

Why do not call a 'record' a 'struct'?

Same with 'List' vs 'Vec'.

And why are List, String & co uppercase?

From what I gather these are built-in types the compiler knows?

'i32', not 'I32' but 'List' not 'list'? In Rust the case is clear cut: built-ins that the compiler 'knows' are lowercase, types expressed in Rust itself are PascalCase.

Roto seems to mix this?

Reads like Rust but then "differs subtley" where it (seemingly?) does not matter at all.

What is the rationale? I'm not complaining, Roto looks great. I'm considering swapping my Koto integration in my project out for this. :)

I'm just genuinely curious. From the outside the above smell like "just because" decisions.

terts8• 2 months ago

This is something we've been backpedaling a bit indeed. 0.11 changed some things to look a lot more like Rust. The reason for this being different in the first place is that were not trying to make Rust-but-simpler necessarily. Or target audience with Rotonda is also not Rust programmers. So we've been exploring different syntax options.

Regarding the List type, as we optimize that, it will probably become a different data structure from a Vec. Also in the Roto-Rust boundary, you cannot pass in a normal Vec. So different names male sense there.

Regarding the case, small primitive types are lowercase. String and List are heap allocated and therefore somewhat different. I'm open to changing that but there is some logic to it.

virtualritz2• 2 months ago

PascalCase == Heap allocated makes a lot of sense to me.

Even if one may argue that scripting languages usually abstract ideas like stack and heap away. :)

fsckboy• 2 months ago

the bash "scripting language" mimics the bash cli, it "scripts" what you can do in bash by hand and it does it by being bash.

I just think that's a nice way to use the word "scripting" instead of calling these scripts "programs" or calling things that are programs "scripts".

Archit3ch• 2 months ago

Why not Julia? Serious question.

For that matter, why not C#?

One year of Roto, a compiled scripting language for Rust

Related Stories