New Rust runtime turned on. What next?

https://mail.mozilla.org/pipermail/rust-dev/2013-August/005158.html

by usea • 13 years ago

186 114 13 years ago

mail.mozilla.org

pshc• 13 years ago

Excerpt from Graydon's (BDFL) reply:

  > Despite all these caveats I have a very strong sense that writing the
  > runtime in Rust will go a long way to validate Rust in the domains it's
  > aiming for: concurrent and systems programming. Even in the task
  > scheduler, where there's quite a bit of unsafe code, the shared-nothing
  > nature of unique types forces you to consciously break the type system
  > to share memory, and that seems to go a long way to making parallel
  > programming easier to reason about.

  Yes, even just reading it seems much clearer since the mutability, 
  lifetime and ownership of each value and reference is spelled out, not 
  just "some Foo* that you have to remember special validity rules about". 
  It's noticeably easier to reason about. Really interesting!

When Rust matures (and this dogfooding-via-self-bootstrap is going to accelerate its maturation) it's going to be seriously revolutionary for, for example, modern multi-core video game development. Really excited.

zhemao• 13 years ago

Yes, please. Anything but C++. The progress of Rust and Go make me hopeful for the future.

charliesome3• 13 years ago

Go (at least as it stands today with its non-generational, stop the world, mark and sweep GC) is wholly unsuitable for video game development where unpredictable latency must be avoided.

hamstergene• 13 years ago

Why is it everytime people talk about replacing C++, someone always comes in and sticks Go into discussion? What does Go have to do with C++?

agumonkey• 13 years ago

dogfooding-via-self-bootstrap

Very rare thing. Ok maybe not in language design. But rare otherwise.

dkhenry4• 13 years ago

I think its much less common in language design then we might give it credit for. Many languages run on a VM that is written in C including the likes of Java, Javascript, C#, Ruby and Python ( even PyPy compiles down to C ). I am sure there are others besides C that are self sustaining, but they are _very_ rare

krichman2• 13 years ago

I, too, think Rust is going to be revolutionary in the video game industry. Were I a game programmer I would start learning now. In two years at least 51% of all new video game code will be in Rust.

usea4• 13 years ago

I think you may be overestimating the pace of change. C++ is quite entrenched for industrial-strength game projects. The network effect is strong, the tooling is mature, and the projects are largely driven by C++ experts. Even if we see Rust 1.0 by the end of the year, it will take a lot of time for that kind of change to occur in the industry, if it ever does.

I think you're much more likely to see 51% of new game code written in C# for Unity. I think the median game budget is shrinking. More and more games are being made as small projects by small teams. Steam greenlight, kickstarter, humble bundle, and other avenues are enabling curation and funding of smaller games. Unity is far more suited for this kind of development than C++. If you look at smaller studios' job listings, a lot of them prefer experience with Unity. The trend is in full swing at this point.

I have no sources to site for the above information. Take it with a grain of salt.

nimrody• 13 years ago

You forget a few points:

* Games require portability. Not many platforms will have a rust compiler (Not to mention that current C++ compilers are very mature and highly optimizing).

* Current C++ libraries. They are probably heavily templates-based and will be difficult to port.

smosher4• 13 years ago

I think it may be about 2 years before the games industry even starts to pick up Rust (or possibly a post-Rust language instead), but I think it will happen eventually.

As someone else mentioned, C# is a big deal now and comes with a degree of safety over C++. I don't expect to see Rust in the pole position on the whole, but it could easily displace C++ at Id Software since Carmack has been shopping for an alternative, and elsewhere (especially if he runs with it first.)

But I think C++'s share is losing ground, and Unity is an impressive platform. The asm.js buzz seems to whisper promises about ES and browsers becoming a serious AAA gaming platform. If FFOS gets anywhere it may supply the necessary voltage. So the lay of the land is anyone's guess, but all the variables eat into C++'s % share even if only by expanding the market around it.

martijn_himself2• 13 years ago

I am intrigued by this. Could you (or anyone) give me a 'explain this to me like I'm a 5 year old' synopsis why you think this is going to revolutionise video game development?

NickPollard2• 13 years ago

Video game development has several constraints that aren't commonly found in other development such as web development. In particular, for real-time games running at 30 or 60fps (frames-per-second), you have only 33 or 16ms respectively to process an entire frame of updates. Most game development targets fixed hardware such as consoles (PlayStation, Xbox) or mobiles (iPhone, Android), so if your code is slow, you can't just put a bigger processor in or distribute the code over an array of servers.

These are other requirements mean that most high level AAA games have to be written to be very highly performant, and in particular, to have reliably consistent latencies. Traditional Garbage Collected languages (Java, C#), whilst achieving high throughput, often struggle with predictable latencies as when GC kicks in, particularly in Stop-The-World collectors, you can get a 10-20ms pause which means you will drop a frame or two. This leads to stuttering which is undesirable. Even modern generational collectors still struggle not to have occasional bad pauses. If something like Azul's continuously compacting collector becomes viable for games, this might be solved, but until it does it is hard (but not impossible) to write high performance (soft) real time games in a GCed language.

The result is that most games are written in C++ (sometimes with a small embedded scripting language, mainly Lua, which uses GC) where memory management can be controlled. The downside of this is developer time - it takes large teams of developers lots of time to write games, because the code is largely written at a low level.

Rust is higher level than C++ and allows many useful and safe idioms, including a more functional style and better use of immutability. In general, like other higher level languages, Rust would allow a developer to be more productive than in C++, allowing games to be developed more quickly, with fewer developers. It does this whilst still allowing manual control over memory - some parts can be handed over to GC, whilst others can be carefully allocated to the stack or heap and managed manually. This allows predictable allocation and cleanup overhead, and therefore more deterministic frame times.

As a games developer, I am extremely interested in Rust as a potential game development language.

Disclaimer: I haven't actually written any Rust yet, only looked at code and thought about the potential. I've spent 5 years doing AAA console development in C++.

batgaijin• 13 years ago

So they are still using libuv after the rewrite?

pcwalton11• 13 years ago

Yes (as the Windows IOCP support is invaluable). But we will probably need to add threading support to it.

fzzzy2• 13 years ago

What do you mean add threading support? Last time I used libev (I thought libuv was just libev plus Windows) it allowed creating multiple loops for use in multiple threads just fine. Do you mean adding the ability to move file descriptors across threads into another loop?

dbaupp3• 13 years ago

The rewrite was essentially to facilitate using libuv more. To paraphrase Brian Anderson: "we needed to rewrite io, and we've just taken a detour to port the runtime."

pkulak• 13 years ago

it does implement TCP and UDP on both IPv4 and IPv6

I thought this sort of thing was usually handled by the OS. Can you even get raw sockets on most systems? Or by "implemented TCP", do they mean they can give you back a Unix TCP/UDP socket?

pcwalton11• 13 years ago

By "implement" it means "integrated into the scheduler".

roryokane• 13 years ago

Here’s a properly formatted version of the link: https://gist.github.com/roryokane/6189765

(The original message is in Markdown; I just put it in a Gist where it would be rendered as such.)

dkhenry4• 13 years ago

I am no language designer, but I wonder why use libuv and have to worry about implementing a scheduler and all the other components of a run time loop in your language when the kernel will do this for you. I think it would make more sense to provide a better interface to existing kernel structures then leverage a third party library and then re implement kernel functions around it ( I am mainly thinking about the paragraph about the current scheduler implementation and how basic it is )

pcwalton11• 13 years ago

Three reasons. First, we don't control the kernel and we don't want to make assumptions about thread spawning being cheap on every OS. Second, it lets us implement work stealing, which is a proven method for dynamic parallelism. Third, it lets us do some operations such as RPC from task to task entirely in userspace with no trip through the scheduler or OS kernel.

jzwinck• 13 years ago

Good points. A counterpoint for #3 is that you could do kernel bypass RPC by installing a driver, but Rust developers probably don't want to write all those drivers, and Rust users wouldn't want to install them.

Work (task) stealing is very compelling, and a little paradigm shift is no bad thing. If Rust or any new systems language stands a chance, it should aim high and not too close to the past.

srisa• 13 years ago

What if someone wants to write a kernel with Rust? This might be a very naive question taking into account my ignorance of language and kernel design.

marijn• 13 years ago

Look at https://github.com/pcwalton/zero.rs . It's not very far yet, but it proves that it is viable to run Rust without a runtime.

ldunn• 13 years ago

I've done some very preliminary (and bumbling) work with that at https://github.com/ldunn/kern/

It's rudimentary, but it does compile and it does run, and I'm not aware of any particular obstacles to a more featureful kernel.

rca• 13 years ago

Someone posted https://news.ycombinator.com/item?id=5771276 a few months ago. I'm not sure we could call that a kernel, but it's a first step.

andrewflnr• 13 years ago

I really wish I had space in my TODO list to start a project in Rust. I don't think there's another dev tool I'm more excited about.

tyleregeto3• 13 years ago

I'm feeling the same way, Rust looks fantastic. There are a few things in the language I'm not huge fan of, I don't like overly subtle things in a language. For example, some of the functionality around semi colons seem like they will be a common source of stupid programmer bugs that are difficult to track down. Perhaps the compiler will catch that stuff.

Go kinda ruined other languages for me with multiple return values, its something I miss when working in every other language now. And reading through the docs I keep hoping I would stumble across that, though that would create a lot of problems when interfacing with C. Type inference is a huge win, standard libraries looks solid, love the potential around marking variables as mutable, and any language that has no null values makes me happy.

Overall, I wish I had more time too.

usea4• 13 years ago

I have limited experience with Rust, but the semicolon stuff you're describing has been one of the most surprisingly positive aspects about it for me. I thought it sounded kind of like a dumb gimmick that just adds subtlety (read: removes simplicity) to something for no reason. However, in practice I've found is really great for making the intent of your code more visible (less cluttered). I miss it when I'm doing C#.

Also, I've never come across a situation with too few/extra semicolons causing any kind of logic errors. The compiler will complain at you if you get it wrong.

pitterpatter• 13 years ago

Multiple return values is possible with n-tuples.

  fn addsub(a: int, b: int) -> (int, int) {
      (a + b, a - b)
  }

  ...

  let (a, b) = addsub(32, 44);

PommeDeTerre• 13 years ago

Rust does have a lot of potential, but don't underestimate how much change it has undergone lately, and will likely be undergoing in the future.

The large amount and rapidity of change makes it difficult to use it seriously. It's nowhere near as stable as other newer languages like Go and Scala are, for instance.

Some experimental programs I wrote a mere 8 months ago are now basically unusable with recent versions of Rust due to language, syntax and standard library changes.

Unless you can constantly track Rust's development on a daily basis, and update your code accordingly, I'd be very hesitant to suggest using it for anything but throw-away code at this point.

copx3• 13 years ago

Go is not the only language with multiple return values. Lua for example had them before Go. In Lua it would look like:

  addsub = function(a, b) return a + b, a - b end

  a, b = addsub(32, 44)

And if you run that code with LuaJIT it compiles down to a few machine code instructions, no object creation at all, no function call.

In contrast I think at least in Python the tuple based solution would be horribly inefficient because it allocates/deallocates a new object every time just to pass values. I don't now if the Rust compiler is smart enough to optimize the tuple creation away.. but I doubt it.

masklinn6• 13 years ago

For example, some of the functionality around semi colons seem like they will be a common source of stupid programmer bugs that are difficult to track down. Perhaps the compiler will catch that stuff.

It will, because the semicolon in Rust actually has a semantics impact: `a` has type T(a) but `a;` has type Unit (~void).

Go kinda ruined other languages for me with multiple return values

That's sad, because Go has one of the worst MRV implementations out there: it's a special case of the language itself.

In most languages with MRV — and that includes Rust, but also MLs, Haskell, Erlang, Python or Ruby — MRV is simply a natural consequence of being able to unpack or pattern match containers (tuples and/or lists depending on the typing discipline).

Anyway Rust has multiple return values, don't worry about that.

BruceIV• 13 years ago

I started porting one of my side projects (written in C++11) to Rust because I thought it looked promising - I still think it does, but I filed two bug reports before I even got argument parsing working, and the documentation is still rather sparse. I don't want to dissuade you, it's a great project, and I hope it's the next C++, but it's very much not production-ready, so don't build anything critical in it yet.

btipling2• 13 years ago

I tried to give Rust a try to build some stuff but my project required HTTP and there's no easy SSL solution in place right now. Hope it comes along. I don't have time to contribute much otherwise I would.

kibwen6• 13 years ago

To quote Brian Anderson (the OP), one of the next steps in the I/O rewrite is "Implementing a new HTTP client on top of rt::io, possibly using Chris Morgan's HTTP code, for use in Servo". So hopefully within the year we'll see the beginnings of a robust HTTP lib that's worthy of a Mozilla-brand browser engine.

btipling2• 13 years ago

That's awesome!

chrismorgan2• 13 years ago

Note that that is HTTP; SSL will be likely to come quite a bit further down the track.

zokier• 13 years ago

Use FFI to use some C SSL library?

http://static.rust-lang.org/doc/tutorial-ffi.html

Dn_Ab• 13 years ago

I haven't written anything in Rust yet but one of its most interesting aspects is its support for linear types. I expect that will share the same kind of perspective enriching attribute of learning logic, array or functional paradigms. While it's certainly not the first language to support substructural types, it looks the only one with a decent chance of developing a meaningful ecosystem.

The linear logic of J.-Y. Girard suggests a new type system for functional languages, one which supports operations that ``change the world''. Values belonging to a linear type must be used exactly once: like the world, they cannot be duplicated or destroyed. Such values require no reference counting or garbage collection, and safely admit destructive array update

http://homepages.inf.ed.ac.uk/wadler/topics/linear-logic.htm...

An interesting bit of trivia about linear types is they are in some sense the closest thing to programming a quantum computer this side of qubits (where no cloning dictates qubit variables can only be used once in a function term).

oconnor02• 13 years ago

I believe Clean also uses linear types.

masklinn6• 13 years ago

tasks are now migrated across threads by the scheduler, whereas in the old scheduler a single task was always run in the same thread.

Is that done by having a single task queue with multiple schedulers, or through work-stealing by schedulers with no ready tasks in their queue?

Would this open the possibility of configuring schedulers (including individually)? E.g. ensuring a given task stays pinned on a specific scheduler, and said scheduler accepts no more task, that kind of things?

smosher4• 13 years ago

Would this open the possibility of configuring schedulers ...

I've been assured that it's the plan. I have no idea how much of it works right now though. I've only done one build with the new rt and it doesn't involve tasks.

masklinn6• 13 years ago

I've been assured that it's the plan.

Excellent [twirls mustache]

I have no idea how much of it works right now though.

Yeah I don't expect it to work at this point, but knowing it's one of the end-goals is good.

pcwalton11• 13 years ago

Is that done by having a single task queue with multiple schedulers, or through work-stealing by schedulers with no ready tasks in their queue?

Right now it's the former, but Aaron Todd has a pull request to switch it to the latter.

brson• 13 years ago

The implementation on master currently uses a single queue, but very shortly it will be converted to work stealing.

Tasks can be 'pinned' to their own scheduler (i.e. thread) with `spawn_sched(SingleThreaded)`, and this is very important for tasks that call foreign code that blocks.

That's about the extent of the configurability at the moment, but I anticipate at least one other 'mode' in the future for coping with blocking tasks that don't want to be pinned to a specific thread.

New Rust runtime turned on. What next?

Related Stories