From Stacks to Trees: A new aliasing model for Rust

https://www.ralfj.de/blog/2023/06/02/tree-borrows.html

by obl • 3 years ago

191 42 3 years ago

    fn two_phase(mut x: Vec<usize>) {
        let arg0 = &mut x;
        let arg1 = Vec::len(&x);
        Vec::push(arg0, arg1);
    }

> This code clearly violates the regular borrow checking rules since x is mutably borrowed to arg0 when we call x.len()! And yet, the compiler will accept this code

Does anybody else wish the compiler wouldn't and would be even more verbose? I know one of the biggest learning curves (personally) for Rust is the borrow checker complaining hardcore and "getting in your way" preventing you from basically doing anything you're used to (passing around pointers in C or objects in JavaScript (even though you should be following immutable practices and not doing object mutation... most of the time))

I'm sure there's probably been discussions on how to make the borrow checker less "mean/rigid/obtuse" but silently passing something as "non mut" and it actually does "mut" stuff, I wouldn't have guessed Rust allowed that.

Edit: gah, I did not realize the function signature is (mut x), I thought it was just (x) and the mut was implied which is what I was trying to call out, apologies.

FpUser6• 3 years ago

"Does anybody else wish the compiler wouldn't"

Compiler being obtuse and not being able to figure when it is safe to "break rules" is the problem. Not twisting brain of the programmer into being "safe compiler". This sounds like a Stockholm syndrome.

"you should be following immutable practices"

No I should not. I should do what makes sense in particular situation and not bending over for some zealots trying to enforce one and the only way.

neerajsi2• 3 years ago

I think one measurable outcome here is what kind of error message you get when you do violate a rule and whether rust users know what to do to fix their code. As a person who loves to explore the complexity behind seemingly simple interfaces, this stuff is really cool. On the other hand, I don't relish having people break their brains to understand why similar code is accepted vs not.

I'm not a rust user myself, but I'm guessing from all the references to raw pointers that a lot of the code referenced here is actually not idiomatic for all but small snippets of high perf code, so maybe the complexity is not going to affect too many people.

Ygg22• 3 years ago

Compiler being obtuse and not being able to figure when it is safe to "break rules" is the problem.

Compiler afaik will never be able to correctly 100% identify you are or aren't breaking some properties due to Rice's Theorem.

That said, you're committing a Nirvana fallacy. Perfect doesn't prevent improvement.

E.g. seatbelts don't prevent being stabbed by a large metal pole, ergo it's useless.

Every week I see newbies coming and asking why won't compiler allow this - and then point a hugely unsafe action.

Hell, I ran into a similar issue. I wanted to expose something mutable as immutable. My argumentation was but it was immutable at time of calling. However as someone in Rust discord pointed, using that you could cause UB trivially.

chlorion• 3 years ago

In C++ or C, you are always twisting the brain of the programmer into being a "safe compiler". I don't think that is an advantage personally.

Lifetimes don't go away just because there isn't a borrow checker or way to define them in the source code.

TazeTSchnitzel• 3 years ago

silently passing something as "non mut" and it actually does "mut" stuff

No, it's the opposite that's happening here: a mutable borrow of the vector is made, and then a non-mutable thing is done with it (getting the length), before finally mutating it (pushing).

wongarsu• 3 years ago

The borrow checker was made for correctness, not correctness for the borrow checker.

You have ownership of a Vec, you get its length, then you push to it through a mutable reference; nothing evil happens here except the order of the statements (which is an implementation detail that people might not think about when writing the short form x.push(x.len())). The code above is perfectly safe if written in C, which is why the borrow checker was extended to also allow it in Rust. You could make the argument that simpler borrow checker rules lead to a simpler mental model. The counterargument (that won in the end) is that "if it's safe, the borrow checker allows it" is a mental model worth pursuing.

denotational• 3 years ago

The original code (which desugars to the snippet you posted) is:

    fn two_phase(mut x: Vec<usize>) {
        x.push(x.len());
    }

This should clearly be accepted (this is self evident in my opinion); if you need to jump through loops to write code like this then the language is too restrictive to write normal code.

The standard implementation of Rust does indeed accept this, and there is no soundness hole here.

The existing semantics for aliasing and borrowing from MPI (Stacked Borrows) don’t allow this, which means the semantics are overly restrictive; we want this to be accepted.

This work “fixes” this issue by extending the semantics to admit the behaviour exhibited by the standard implementation.

The rules for the borrow checker are not fully formalised and to some extent the rustc implementation is the specification; formalising the rules (i.e. RustBelt, Stacked Borrows, etc.) is important, but we don’t want to formalise something that is strictly more restrictive than the reference implementation, especially if there’s no soundness hole.

Osiris• 3 years ago

I’ve been learning rust and I spend the vast majority of my time dealing with lifetimes and borrow checking. Common ways in used to doing things simply don’t work in rust and a lot of effort has to go into keeping track of how and where data is used.

I’ve worked in OOP languages, functional languages, and dynamic languages but all of them were essentially garbage collected, so having to keep track in my head of how data ownership is managed is a big learning curve.

imron• 3 years ago

As a c++ programmer, one of the great things about rust is that I no longer have to keep track of data ownership and management in head.

I can outsource this to the compiler and if I get it wrong the program won’t compile.

In c++ you still need to do all the same tracking and management if you want safe and correct programs, but you don’t get nearly as much help from the compiler if you make a mistake.

saghm2• 3 years ago

If I'm understanding correct, the major change here for Rust users (rather than people who hack on the compiler) is that mutable references will not be considered to be "interfering" with other references being made at the same time until they're actually written to for the first time. This makes intuitive sense to me, but I suspect that there may be a bit of concern that this will make things more confusing when reading code and trying to understand what's going on. I'd be lying if I said that thought didn't occur to me, but at this point being surprised at how much I end up liking the way things turned out has become the norm for me; I remember having misgivings about nested import paths (rather than only being able to use `{`...`}` at the very end), match ergonomics, and `.await` as a postfix keyword but pretty quickly became glad they decided things the way they did after using each of them a bit when they finally got stabilized. I think I did realize that I'd like NLL (i.e. the borrow checker detecting the final use of a reference and not considering it as conflicting for the remainder of the scope) before it landed, but I know a lot of people had misgivings about that as well. I imagine this will be one of those things that in a few years will seem weird it wasn't always how it worked!

PoignardAzur• 3 years ago

To be clear, this doesn't change what programs get accepted by the borrow checker, so for most rust users it changes absolutely nothing.

It changes the abstract rules behind rust's safety model, which impacts which unsafe functions are considered sound, and which optimizations the compiler is allowed to perform.

saghm2• 3 years ago

Ah, I guess I did misunderstand then. I see now rereading that the "don't treat as a mutable borrow until first write" is already how things behave today and have since NLL.

chrismorgan• 3 years ago

To expand a little on this, by pulling out one key piece of the article:

This is what the footnote of the bit about &(i32, Cell<i32>) clarifies (which footnote was added due to this misunderstanding, discussed in https://old.reddit.com/r/rust/comments/13y8a9b/from_stacks_t...).

In particular, for &(i32, Cell<i32>), TB allows mutating both fields, including the first field which is a regular i32, since it just treats the entire reference as “this allows aliasing”.¹
¹ This does not mean that we bless such mutation! It just means that the compiler cannot use immutability of the first field for its optimizations. Basically, immutability of that field becomes a safety invariant instead of a validity invariant […]

This matter of safety versus validity invariants is key (https://www.ralfj.de/blog/2018/08/22/two-kinds-of-invariants...).

saurik2• 3 years ago

...by the time x.len() gets executed, arg0 already exists...

So, I realize that this is the way that Java does it--and, presumably, one still doesn't get fired for doing whatever Java does ;P--but, would it not actually make more sense for the arguments to be evaluated before the target reference, making the argument order more like Haskell/Erlang (but very sadly not Elixir, which makes it awkwardly incompatible with Erlang and breaks some of the basic stuff like fold/reduce)? Particularly so, given that, as far as I can tell from this example, what makes arg0 have the type that it does is the type of the function that hasn't even been called yet? (As in, the semantic gap I am seeing between what the user probably meant and what the compiler wants to do is that "x" shouldn't really be mutably-borrowed until the call happens, and the call here clearly shouldn't happen until after the arguments are evaluated.) (Note: I do not program in Rust currently; I just have spent a number of decades analyzing languages and at times teaching college language design courses. I might be missing something obvious elsewhere that forces Rust to do this, but that one example, in isolation, at least feels like an unforced error.)

Georgelemental• 3 years ago

In Rust, `reciever.some_method(whatever)` is supposed to be relatively thin sugar for `TypeOfReciever::some_method(receiver, whatever)`. So the evaluation order should be the same for those two forms.

saurik2• 3 years ago

Sure, but I am saying it would actually have made more sense to put receiver as the last such argument--as one might expect from having used Haskell/Erlang--given the other design decisions clearly in play here, as the target reference isn't really the first argument for any obvious reason other than visual effect and some historical baggage from implementations of some object-oriented languages (including Java) with different constraints.

codesnik• 3 years ago

interesting. Can you expand on Elixir incompatibility? How is it different?

fredrikholm4• 3 years ago

Elixir does:

  Enum.map(list, func)

This is backwards to how partial application works:

  def mapper(func), do: Enum.map(func)
  incrementer = mapper(fn x -> x + 1 end)
  incrementer(list)

It's not the end of the world to not have it this way, but it removes a lot of patterns that are common in other functional languages.

tsimionescu• 3 years ago

Doesn't Rust support having a function as a field of a struct?

If it does, then the order of evaluation of a.foo(b) would depend on whether foo is a field or a "free-standing" function of a, which seems horrible.

Also, there is a simple elegance in having the order of evaluation match the order the symbols are written that should require a very hight bar to reverse, in my opinion at least.

MereInterest• 3 years ago

It does, but Rust also has separate namespaces for methods and variables. That is, a.foo(b) will always be a method foo and never a field foo, because the syntax is that of a method. In order to access a function object, then call it, you would use (a.foo)(b). The parentheses cause the contents to be parsed as a variable expression.

weitzj• 3 years ago

I have not programmed Rust, yet. But this article gives me a feeling that this looks similar to database transactions. This might be wildly wrong but for me I see an analogy:

Once you get the &mut reference, you have your tree, which then looks to me like you have created a transaction. An in this transaction context you do your things.

oslac• 3 years ago

This is a good intuition, Rust's references can be thought of as a r/w locks.

classified• 3 years ago

Interesting, and a nice demonstration of how a language gets more complicated "all by itself".

From Stacks to Trees: A new aliasing model for Rust

Related Stories