Linux kernel in-tree Rust support

https://lore.kernel.org/lkml/CAKwvOdmuYc8rW_H4aQG4DsJzho=F+djd68fp7mzmBp3-wY--Uw@mail.gmail.com/T/#u

by littlestymaar • 6 years ago

468 491 6 years ago

lore.kernel.org

Jonnax• 6 years ago

So I don't know much about kernel development.

How many other languages apart from C are there in the kernel?

BlackLotus89• 6 years ago

Assembler. Although there is a movement to port assembler to c.

pjmlp15• 6 years ago

Which is impossible with ISO C alone, so it will either stay Assembly, just written inline, or replaced by CPU compiler specific intrinsics, yet another GCC C feature.

rwmj5• 6 years ago

Depends how you define "in the kernel". In terms of what is compiled into the kernel image, it's C plus small pieces of assembler (both inline and in separate files). This proposal (if it ever happens) would be about adding Rust to that. If you mean the tree used to build the kernel, you could say it includes shell script, Makefiles, the Kconfig descriptions, and more. If you mean all the bits you need to have a very minimally working system which boots, you'd want to include all the languages used for the necessary userspace tools like the init system and udev (although the vast majority of that is also C and shell scripts).

correct_horse2• 6 years ago

I'm a bit uninformed as well. Obviously it must use assembly in some places, it has makefiles for building and some scripts in perl. Here's what a cloc clone has to say.

    tokei
    ===============================================================================
    Language            Files        Lines         Code     Comments       Blanks
    ===============================================================================
    ASN.1                  11          484          343           33          108
    Assembly                3         2751         2450            0          301
    GNU Style Assembly   1296       370153       270691        56418        43044
    Autoconf                6          347          303           20           24
    Automake                3           31           23            3            5
    BASH                   54         1991         1438          321          232
    C                   28971     19846129     14634535      2351170      2860424
    C Header            21110      6056898      4465297      1010398       581203
    C++                     8         2313         2016           68          229
    C++ Header              2          125           59           55           11
    CSS                     1          137           80           29           28
    Device Tree          3351       900862       735380        54238       111244
    Gherkin (Cucumber)      1          252          174           50           28
    Happy                   9         5750         5061            0          689
    HEX                     1           86           86            0            0
    INI                     1            7            3            3            1
    JSON                  328       174872       174872            0            0
    LD Script              15          412          317           48           47
    Makefile             2587        64666        43896        10957         9813
    Module-Definition       2          124          109            0           15
    Perl                   53        40583        31126         3931         5526
    Python                118        36932        29072         2980         4880
    ReStructuredText     2621       548840       417105            0       131735
    Ruby                    1           29           25            0            4
    Shell                 556        85666        59725        10972        14969
    SVG                    59        39038        37555         1405           78
    Plain Text           3230       239573            0       197373        42200
    Unreal Script           5          716          446          162          108
    Vim script              1           42           33            6            3
    XSL                    15          300          183           78           39
    YAML                  921        91167        72962         3985        14220
    ===============================================================================
    Total               65340     28511276     20985365      3704703      3821208
    ===============================================================================

nix23• 6 years ago

The Unreal Script is for the new 3D menuconfig ;)

skissane• 6 years ago

I was surprised to see C++ in there, but I understand it is simply some of the user-space tooling (in particular, some small parts of the perf tool are written in C++, in order to embed clang/LLVM.)

Likewise, Perl/Python/Ruby are just scripts included for various purposes (such as building documentation).

Interestingly, some of the assembler code included in the kernel is generated by Perl scripts (crypto code for ARM and MIPS). Perl is also used to compile "SCSI SCRIPTS" which are used by the NCR 53c810 SCSI driver.

rwmj5• 6 years ago

The title is not very accurate, this is a thread about a discussion on this topic which will happen at the upcoming Linux Plumbers Conference in late August.

chrismorgan2• 6 years ago

The title is perfectly accurate, it’s an email thread about Linux kernel in-tree Rust support. Sure, you could misconstrue such a title to be implying that the Linux kernel supports Rust in-tree already it if you wanted to, but half the titles on a site like this could be similarly misconstrued.

jaekash• 6 years ago

The title is perfectly accurate, it’s an email thread about Linux kernel in-tree Rust support.

The title the person you are responding to is complaining about is not the title of a email thread but the title of a hacker news post.

littlestymaar• 6 years ago

I see what you mean. I just posted it here with the same title that was used on /r/Linux, and that I found accurate (for the same reasons chrismorgan exposed in a sibling comment) but now I agree with you that it could cause some confusion.

Maybe a moderator could rename the post to “discussion about Linux kernel in-tree support” or something like that?

pepijndevos• 6 years ago

Pleasantly surprised by Linus response. IIRC his attitude to C++ was that it should be refused if only to keep C++ programmers out.

buster• 6 years ago

Yes, very much so. Not even only in the context of Rust but the insight, to fail fast, integrate early and do work in the open, instead of some hidden work, failing after a long time, when revealed.

kevingadd• 6 years ago

Perhaps the possibility of rust improving kernel security/robustness makes the idea of rust integration seem like it carries its own weight, where C++ has more downsides (perceived or real) and fewer upsides.

Rust's history/origins - loosely, being designed to allow replacing Mozilla's C/C++ with safer Rust that performs well - feel like a good fit for kernel drivers even if the core kernel bits will always be C.

for_xyz3• 6 years ago

His attitude towards C++ also changed when using it for his side project [1].

Initially he started with C and GTK+ and later migrated to C++ and QT Framework.

[1] https://subsurface-divelog.org/

globular-toast• 6 years ago

Did he have an "attitude" about C++ in general? I thought he only commented on it with respect to operating system development. He did make much more general statements about Java, though.

agumonkey• 6 years ago

Did he talk about his thoughts precisely ?

josefx• 6 years ago

I think that was after others took over the UI development. The back end of that program also was still C as far as I remember from their presentation and the move was mostly motivated by the GTK community, the documentation and different priorities on cross platform support.

stjohnswarts2• 6 years ago

I love how the bugtracker on that is "check out our mailing list" :D

xvilka4• 6 years ago

Rust is well designed modern and concise language with sound type system, while C++ isn't. The difference is huge and obvious.

hellofunk5• 6 years ago

Except that one of Linus' most vocal offenses on C++ was due to operator overloading and how basic, seemingly native things like + can actually do a lot of hidden stuff unknown to the programmer. He must have softened on this since Rust offers the same facilities for operator overloading.

https://doc.rust-lang.org/stable/rust-by-example/trait/ops.h...

pjmlp15• 6 years ago

Linus nowadays also uses Qt, so.

mushishi• 6 years ago

Could you give me a pointer to a discussion of type soudness in Rust? I recently watched perhaps 3 years old video about Rust where Simon Peyton-Jones asked about this, and Niko Matsakis answered it was ongoing work back then.

Couldn't find anything proper by googling.

rwmj5• 6 years ago

Assuming this happened and some essential drivers were written in Rust, would this effectively exclude GCC from being used to compile the full kernel?

est315• 6 years ago

GCC can still compile the components written in C. Only drivers for select hardware might be written in Rust in the future, optional components basically. And GCC won't be more disadvantaged than clang, as clang can't compile Rust code either. The only production compiler that can compile Rust is rustc.

rwmj5• 6 years ago

rustc uses LLVM like clang. That seems important because features like LTO require the whole kernel to be compiled with the same toolchain.

xg15• 6 years ago

This answer was a very "no"-sounding "yes". :)

raverbashing2• 6 years ago

I wonder if/when GCC will have a Rust frontend

There seems to be some discussions but not sure if it's a goal

yjftsjthsd-h6• 6 years ago

That would certainly help with some concerns around Rust; compiler diversity and target platform support would both benefit.

loosescrews• 6 years ago

I have a hard time seeing this happen. Rust has most of the same properties which prevent C++'s inclusion in-tree. (Look up Linus' opinions on C++ if you aren't already familiar.)

Further, as far as I can tell, Rust currently lacks mainline GCC support which I believe will be a blocker.

maccam94• 6 years ago

I am assuming that it would work with clang-compiled[1] kernels only at first.

1: https://www.kernel.org/doc/html/latest/kbuild/llvm.html

est315• 6 years ago

Why would it require clang for the C components? The Rust compiler creates native libraries which can be linked to compilation units created by both clang and gcc, unless there's something special going on in the kernel that I'm not aware of.

cesarb4• 6 years ago

Not really. Unless you want LTO, the object files generated by GCC and LLVM are compatible, and can mixed without issues.

photon12• 6 years ago

The thing is the kernel is kinda hurting for maintainers and opening up the path for the next generation of systems programmers (most of whom don't want to touch C any more than they have to) has to be something Linus is more likely to prioritize than in previous decades.

bibabaloo• 6 years ago

The thing is the kernel is kinda hurting for maintainers

Source? AFAIK the kernel is probably one of the most, if not the most, well-funded and maintained open source projects.

Glavnokoman• 6 years ago

I for one do not want to touch Rust more than I have to.

MrBuddyCasino2• 6 years ago

How would requiring learning Rust, which is famously difficult, be the solution to that problem?

atq2119• 6 years ago

The kernel isn't hurting for developers though.

I don't see how a different programming language would help improve the ratio of maintainers vs developers.

jdub• 6 years ago

But you can read his response to Rust in the linked thread.

zozbot2346• 6 years ago

Rust integrates easily with C because it's explicitly designed to fall back to the C ABI for external linkage. Whereas writing C++ with any C++-specific features (hence, any "Modern", guidelines-compliant C++ at all) brings its own bespoke ABI into the picture, that's supposed to be stable but really is not and that one can't easily interoperate with from C. That's a huge non-starter in kernel dev.

bonzini3• 6 years ago

Not really, both in Rust and in C++ the C-compatible parts have to be marked explicitly as extern "C". Even the syntax is the same.

wwright2• 6 years ago

I’m a huge Rust fan, but this is a misrepresentation of C++. You can definitely use the vast majority of modern C++ features while also integrating with a C codebase (exotic control flow such as exceptions and coroutines are definitely dangerous, though).

bonzini3• 6 years ago

Rust has most of the same properties which prevent C++'s inclusion in-tree

It does but it also has some advantages that C++ does not have.

xiphias24• 6 years ago

Here's Linus's answer inside the thread, which is quite positive:

From: Linus Torvalds @ 2020-07-10 23:54 UTC (permalink / raw) To: Josh Triplett Cc: Christian Brauner, Nick Desaulniers, alex.gaynor, Greg KH, geofft, jbaublitz, Masahiro Yamada, Miguel Ojeda, Steven Rostedt, LKML, clang-built-linux, Kees Cook

On Fri, Jul 10, 2020 at 3:59 PM Josh Triplett <josh@joshtriplett.org> wrote: > > As I recall, Greg's biggest condition for initial introduction of this > was to do the same kind of "turn this Kconfig option on and turn an > option under it off" trick that LTO uses, so that neither "make > allnoconfig" nor "make allyesconfig" would require Rust until we've had > plenty of time to experiment with it.

No, please make it a "is rust available" automatic config option. The exact same way we already do the compiler versions and check for various availability of compiler flags at config time.

See init/Kconfig for things like

  config LD_IS_LLD
          def_bool $(success,$(LD) -v | head -n 1 | grep -q LLD)

and the rust support should be similar. Something like

  config RUST_IS_AVAILABLE
          def_bool $(success,$(RUST) ..sometest..)

because I _don't_ want us to be in the situation where any new rust support isn't even build-tested by default.

Quite the reverse. I'd want the first rust driver (or whatever) to be introduced in such a simple format that failures will be obvious and simple.

The _worst_ situation to be in is that s (small) group of people start testing their very special situation, and do bad and crazy things because "nobody else cares, it's hidden".

No, thank you.

AnssiH• 6 years ago

https://www.theregister.com/2020/06/30/hard_to_find_linux_ma...

"C is still one of the top 10 languages," answered Torvalds. However, he said that for things "not very central to the kernel itself", like drivers, the kernel team is looking at "having interfaces to do those, for example, in Rust... I'm convinced it's going to happen. It might not be Rust. But it is going to happen that we will have different models for writing these kinds of things, and C won't be the only one."

aryonoco• 6 years ago

You didn't read Linus's response, did you?

hellofunk5• 6 years ago

Correct, he must have changed his attitude on some of those features that apply to C++ as well that he previously railed against.

https://news.ycombinator.com/item?id=23801076

mijoharas2• 6 years ago

So this surprises me, obviously this is early discussion about a potential topic, but the general consensus seemed to be more positive than I thought.

I thought I'd remembered reading something (maybe from linus) that seemed very against having rust in the kernel, can anyone find a source for that, I searched a little and can't?

(caveat, I obviously realise that linus isn't supporting rust in the kernel, and is only saying something bounded that, if we have it, it shouldn't be completely hidden behind some config options, but it doesn't match my memory)

gtirloni• 6 years ago

Maybe you're remembering Theo's comments about Rust in OpenBSD?

https://marc.info/?l=openbsd-misc&m=151233345723889&w=2

mijoharas2• 6 years ago

I think you're right actually!

Thanks for finding it for me.

raverbashing2• 6 years ago

The sad part is that Theo's answer on the thread seems to be one of the most polite and realistic ones

ComputerGuru3• 6 years ago

Shame I hadn't seen that thread. He's right that most of the rust projects "replacing" standard Unix utils are not feature equivalent and differ in intent, but that doesn't mean there aren't other efforts to do exactly what he is asking.

Eg here's tac (admittedly not cat, but hey, ./goo | tac | tac) published a few months before his email: https://github.com/neosmart/tac

xvilka4• 6 years ago

Immunant folks wrote a blog post [1] about automating conversion of Linux kernel drivers from C to Rust by using their tool c2rust[2].

[1] https://immunant.com/blog/2020/06/kernel_modules/

[2] https://github.com/immunant/c2rust

yjftsjthsd-h6• 6 years ago

I don't see them discussing what I view as the biggest question of such conversion: is the converted code safer than C, or is it a direct translation with all the issues that we were trying to fix by using Rust? (This came up, IIRC, with automatic C to Go; it worked, but was only useful as a first step because it gave you unsafe unidiomatic Go code)

steveklabnik22• 6 years ago

It is currently a direct translation into unsafe.

They are also interested in "unsafe to safe refactoring tools" in my understanding, but they're not there yet.

xondono• 6 years ago

I always find these email threads very hard to follow, specially on mobile.

Is there anyway to have them shown more friendly?

canofbars3• 6 years ago

Firefox reader mode seems to do a decent job

manjalyc• 6 years ago

It is also a blessing for paywalled news sites.

manx• 6 years ago

I'm using the app "Materialistic" on Android to read the comments. It works very well.

golergka2• 6 years ago

Tangential question: what would be a career path for a typical product developer to become one of the programming gods who get invited to such discussions? (Even if it's technically open mailing list, you know what I mean). How many years of experience and how much formal CS education would it take to acsend to that level?

ris• 6 years ago

Heavy reading of LWN.net articles.

stevekemp• 6 years ago

Reading is certainly step one. Steps two to ten probably involve lots of trying things, experimenting, learning, practicing.

jonahbenton2• 6 years ago

They aren't gods, just normal programmers with specific interests. The knowledge and skill base involves details that are close to the machine. It is less of a "deep CS" thing and more just being patient and interested in details of old things, rather than new things. That is why they self-deprecatingly refer to themselves as plumbers.

The big difference between eg product dev and kernel dev is mindset, not skill. Product dev is about novelty, building new features, inventing new tools. Kernel dev is about making old stuff work better.

Very few kernel devs would succeed in a conventional product dev job, instead of creating value they would be tinkering with some superfluous detail.

If one has the mindset, getting into kernel programming is probably a six month to 2-year initial investment, depending on prior background. Most "operating system" CS grad courses introduce kernel programming as a teaching tool, but a whole CS undergrad isn't required, just an understanding of C and some other basics around low level machine operation.

(Edit: side note- for emphasis- kernel programming is highly specific, extremely empirical and experimental. Not abstract or theoretical. Academic CS tends toward the abstract/theoretical. One can be a poor academic but an excellent kernel programmer.)

The best standalone resource is a book by a fellow named Robert Love called Linux Kernel Development. One of the best technical books I have read and a good indicator of the presence of the mindset.

Cheers!

coldpie4• 6 years ago

Formal CS education is pretty much useless for systems programming. Systems programming deals with real-world hardware, CS is theorizing about about fantasy machines. "Computers are to Computer Science what telescopes are to Astronomy." They're just different fields. They intersect, of course, but systems programming isn't very near that intersection. Being good at analyzing a DFA on a Turing machine isn't going to be much help when you need to bang bits to a non-compliant USB device.

If you want to get into these discussions, you need to be a kernel dev. That means you need to write kernel stuff. Probably the easiest thing is to write or improve a driver for some device you have (game controllers; steering wheels; audio devices; drawing tablets). It's not rocket science, it's all just bits and bytes and code, just like any other programming.

Edit: Just for an example, here's where the PlayStation controller driver parses the data it gets from the device into something applications can read over the input subsystem. Don't be intimidated, there's nothing magic here, just C code reading data and transforming it. Suppose you maintained this driver and wanted to re-write it in Rust. You could have some input into the discussion! https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

nikki93• 6 years ago

Re: the first paragraph--a lot of "formal CS education" focuses on systems and there are systems tracks. Where BSD came from, for example. So I wouldn't call it useless (while the parts that do not focus on systems do, indeed, not focus on systems). In fact, even the non-systems stuff isn't entirely useless for systems--eg. how ideas from ML and Cyclone influence Rust and its application of those ideas to systems use cases.

That said, all this is independently learnable, as is true for all things software.

golergka2• 6 years ago

Honest question: why is this downvoted? Is it bad advice? Which of it is?

pjmlp15• 6 years ago

Except many universities don't teach CS like that, I guess that might be an US thing.

In the Portuguese university where I took my degree, for example, I had to write an UNIX floppy device driver, code in 80x86 and MIPS Assembly, implement my own userspace threading library, implement a distributed algorithm for maldebrot calculations across a Linux cluster via PWM (MPI predecessor) and report its scalability varying the amount of available machines on the cluster, create a compiler, stuff SUN RPC generated stubs with our own encryption algorithm, implement a B+Tree indexing library using inodes directly.

All look pretty useful to me.

justinmeiners2• 6 years ago

CS education typically includes OS theory, computer architecture, and compiler theory, all of which are directly relevant to the kernel.

You may be right that EE is perhaps a better fit for this kind of interest.

intc• 6 years ago

Hopefully this will not pass - The requirement of having GCC + LLVM + Rust installed in order to build kernel would be insane.

Multicomp• 6 years ago

As someone new to the dev party who has never built anything more lower level than a win32 exe, may I ask why that's insane?

Adding 1000 npm dependencies, I see. Adding 1 rustc one? Doesn't SEEM terrible.

yjftsjthsd-h6• 6 years ago

We only just got to a point where 2 C compilers could build Linux; adding rustc brings us back to a monoculture.

moreaccountspls4• 6 years ago

I wouldn't be too worried about it. The kernel people aren't going to take a dependency on something that isn't standardized and doesn't have a spec.

kasabali• 6 years ago

Don't be so sure. Rust lobbying is strong and kernel maintainers don't take a tough stance against things like they used to.

klysm• 6 years ago

You could argue not having memory safety when there are tools to do so is insane.

justinmeiners2• 6 years ago

Memory safety is not an absolute good which overrides all other things, like performance, portability, compilation, process, compile time etc.

It is a problem that is well understood and the kernel developers have a lot of experience and tools for dealing with it.

Would it be worth solving well? Definitely, but it has costs and tradeoffs.

roca25• 6 years ago

Rust in Linux will be fantastic except for compile time. Rust (and the world) needs a Manhattan Project to build a fast Rust compiler (where by "fast" I mean both efficient and scalably parallel when compiling a workspace with many crates and long dependency chains).

To put this in perspective, though: increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people. Compared to the size of the projects that are starting to depend on Rust, that's a rounding error.

zoul• 6 years ago

How much of the needed speedup is engineering and how much is research?

roca25• 6 years ago

Depends on what you mean by "research". If you mean "problems with a high degree of novelty" then quite a lot of research, I think. Rust is unusual: it requires a lot of tricky type inference and static analysis from the compiler, and doesn't make developers structure their code with header files etc to make separate compilation easier; BUT unlike most languages with those properties, people are writing large real-world projects in it.

the8472• 6 years ago

I assume dependency trees in the kernel would be much more shallow.

roca25• 6 years ago

That might be true for drivers.

Outside the kernel, I don't think it's going to be true for Android or Windows.

cxr9• 6 years ago

a Manhattan Project to build a fast Rust compiler

Or work on getting what Rust gives you in a different language that isn't Rust (maybe entirely new, maybe not[1]).

One underdiscussed thing about Rust is that it's so clearly a byproduct of the C++ world. And the most disappointing thing about Rust is that it probably won't ever be seen as a prototype that should be thrown away.

1. http://mbeddr.com/

ragnese4• 6 years ago

Can you elaborate on the underdiscussion of Rust as a byproduct of the C++ world?

I'm into Rust, so I guess I assumed it was common knowledge that it was originally written to be like OCaml with better concurrency, and that one of the early goals was to replace C++ in Mozilla code.

But, language-wise, I don't even see that much similarity with C++ except for the philosophy of zero-cost abstraction and the trivial syntax-style of generics being <>.

roca25• 6 years ago

If you want to convince people that Rust should be thrown away in favour of mbeddr or something else, you need to make an argument based on specific design flaws of Rust, not just tar it by association with C++.

That argument would have to not just explain why an improved language is needed, but also why Rust can't evolve into that language via the edition system. It would also have to convince people that whatever benefits the improved language brings justify moving away from the existing crate ecosystem, tool ecosystem, and developer pool, and building new ones (or how you would reuse/extend/coopt them).

(In reality I think the influence of Haskell and even node.js were as important to Rust as C++.)

oconnor663• 6 years ago

And the most disappointing thing about Rust is that it probably won't ever be seen as a prototype that should be thrown away.

Can that ever happen with a language that lots of people use?

pornel3• 6 years ago

• Rust is an ML-family language dressed in a C-family syntax to look palatable to existing systems programmers. It's a byproduct of the C++ world only to the extent that it has to work on the CPU architectures and operating systems influenced by C and C++.

• Rust is mainly based on established research and existing languages from 80s and 90s. It's a distilled version of these, not a prototype.

I recommend checking out the first pitch deck for Rust: http://venge.net/graydon/talks/intro-talk-2.pdf

nilkn10• 6 years ago

Rust is actually a byproduct of OCaml and to a lesser extent Haskell. This becomes obvious the more you use it, especially if you’ve written substantial code in either of those languages before. To really drive this home, I’d wager that you have a better chance of translating OCaml or non-fancy Haskell straight into Rust than you do C++.

d_tr• 6 years ago

You said two different things about Rust but you did not substantiate anything. What do you mean by

"Rust is so clearly a byproduct of the C++ world"?

Furthermore, in which way is Rust defective to the point that it should be seen as a prototype that should be thrown away?

mbbeddr seems really nice and interesting but it is not just a language. It is a set of integrated tools around one.

Rust shares goals with C++, i.e. it wants to be a high-performance and systems language like C++, but I find the differences to be quite substantial and worthwhile.

mekster2• 6 years ago

increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people. Compared to the size of the projects that are starting to depend on Rust, that's a rounding error.

And are you going to pay for the "rounding error" or just expecting someone to pay for millions a year for your idea?

roca25• 6 years ago

It's not my idea. This is well-known.

As for who pays for it: that's a tricky issue! But if we get into a situation where (for example) 10,000 developers spend an hour a day waiting for builds, we know 50 developers could fix the issue in a year, but that doesn't happen because we can't figure out how to structure it economically --- that would be a failure of organizational imagination.

pchanda• 6 years ago

I think OP meant that companies, like Amazon and MS, whose projects are starting to depend on rust should be putting some resources. Although to be fair, Microsoft and Amazon do pay for their infrastructure if I am not wrong.

jodrellblank• 6 years ago

This is such a low effort sneer-comment, it's a dismissive way to attack and putdown any idea that includes any amount of payment. It's the discussion equivalent of "whoever smelt it, dealt it" - if someone mentions an idea which costs money, demand as a first response if they are going to pay for it (assuming they aren't, instant dismissal), or accuse that the alternative is a kind of entitled and unfair expectation held of others, when neither need be the case.

"That volcano looks like it's becoming active, maybe some sensors could give us an early warning of problems" - Are YOU going to pay for your little "idea"??? Then WHO IS?

"That tree is getting dangeoursly high, it's at risk of falling down in a big storm this winter, maybe we could get it cut down before that" - Are YOU going to pay for it?!

"Dumping raw sewage in the river is making people sick, treating it first would take a small amount of space and a small fraction of the council's existing budget" - and you want to TAX ME for YOUR clean river, I suppose?!

As if OP is the only person who would benefit, as if automatically assuming the worst possible intent for who would pay for it, and as if "who would pay" is the first and only thing worth demanding an answer to, before even having a discussion about whether it's worth doing at all - and which of many ways it might be done.

coldpie4• 6 years ago

increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people

I don't think there are 25 people qualified to do that work looking for jobs. It's not something you can just throw money at, you need really qualified people to do that kind of work. Those people are in very high demand.

roca25• 6 years ago

There are a lot of people out there who can do this kind of work. I know a lot of such people. Many of them don't have traditional CVs.

Yes, most of them are employed. Some are under-employed. Some are well employed but might be keen to work on a project this important.

steveklabnik22• 6 years ago

Note that that would be increasing the number for pay, you already have many people who have the relevant qualifications because they’re already doing the work, just only in their spare time.

Second, the compiler team has put a ton of effort into mentorship infrastructure, and you can go from not knowing a ton to being a productive member of the team with their help. It "just" takes time and desire to do it.

CydeWeys2• 6 years ago

There are certainly way more than 25 skilled programmers who are experts in compilers that could be hired easily by offering enough money (on the order of several hundred thousand dollars per year).

crazypython3• 6 years ago

GCC 10 supports Dlang directly. (The support is still immature and it's being updated to the newer version of Dlang.)

Dlang compiles quickly not because the language is simple, but because the compiler authors care about compilation speed:

Lexer skips four spaces at once https://github.com/dlang/dmd/pull/11095

Optimize core logic to 7x compilation speed of language feature https://github.com/dlang/dmd/pull/11303

Cache rarely used types to improve memory locality https://github.com/dlang/dmd/pull/11363

steveklabnik22• 6 years ago

The Rust compiler folks care immensely about compile speed, and the language is a factor in both cases.

roca25• 6 years ago

Nick Nethercote and others have done a lot of work with traditional profiling and optimization like that. They've done a great job, but for my project the fundamental problem seems to be limited parallelism, which I think requires more fundamental design changes (perhaps getting away from the crate-at-a-time approach of rustc).

Though TBH I'm not an expert on the compiler and I'm not confident I know the root problems or the solutions.

PudgePacket• 6 years ago

Slightly off topic, I see D brought up incredibly consistently when Rust is mentioned on HN.. Is it just me?? (it's usually by walter themself but I digress..)

VWWHFSfQ• 6 years ago

but because the compiler authors care about compilation speed

Is this meant to imply that the Rust compiler authors don't care about compilation speed?

erichdongubler2• 6 years ago

It's important to note that run-time tradeoffs were made in the interest of fast compilation speed. When it comes to optimizing code emitted from D source, it's important to leverage the alternate implementations from GCC or the LLVM-based LDC...which don't sport nearly as fast compiles as DMD, the reference D compiler.

geofft8• 6 years ago

The D language uses GC, so it's unlikely to be acceptable for the Linux kernel. (That's not to say that you can't write a kernel in D, just that the Linux kernel maintainers don't want to write a kernel that requires GC.)

Rust's ownership model, which allows you to write straightforward, non-leaky, non-use-after-free-filled code without a GC, is quite complex at compile time. It's definitely possible to make it faster, but it's definitely a thing Rust does that D doesn't do.

zelly5• 6 years ago

The push for Rust is coming from above, top-down. Watch what programmers do, not say. They prefer Go to Rust because it compiles fast.

marta_morena_25• 6 years ago

D was dead by all practical measures since its inception. It was never revolutionary enough to warrant the huge costs of migrating there. This is where Rust comes in...

rumanator• 6 years ago

Rust in Linux will be fantastic except for compile time. Rust (and the world) needs a Manhattan Project to build a fast Rust compiler (where by "fast" I mean both efficient and scalably parallel when compiling a workspace with many crates and long dependency chains).

A basic technique to parallelize sections is to split a project into modules. Is this not an option for the linux kernel?

roca25• 6 years ago

Rust has great support for modules (crates in Rust lingo). It can compile modules in parallel when they don't depend on each other.

The problem is that when you have module A depending on module B, rustc doesn't do a good job of compiling A and B in parallel. In contrast, in C/C++ you can handcraft your header files to ensure that A and B compile in parallel.

Ericson23144• 6 years ago

Replace "faster" with "more incremental".

Trying to get a program to quickly do the exact same thing over and over again is a colossal waste of resource.

hinkley2• 6 years ago

When you’re trying to balance a bad budget, you can’t dismiss many of the cost centers and still succeed.

You can prioritize, but that makes the rest a matter of “when” not if.

It can be hard to tell from the outside if someone is avoiding a problem because they don’t want to solve it, or because they are hoping for a more inspired solution. Tackling a problem when all the options are bad may block another alternative when it does finally surface. I’d lump this in with irreversible decisions and the advice to delay them to the last responsible moment.

Who know what internal plumbing has to be moved around to make the compiler more incremental, parallel, or both.

darksaints4• 6 years ago

I know a lot of people will disagree with me, and I know that the rust compiler has room for improvement, but compile time isn't really that bad for rust when you consider what it actually does for you.

Assuming no unsafe blocks, rust's type system eliminates entire classes of memory safety errors. It fundamentally eliminates data races. It eliminates large swaths of logical errors, due to its use of algebraic data types, and strong type enforcement. No null pointers, explicit error handling, and no undefined behavior.

It boggles my mind that people will balk at compile times for rust, but then not even bat an eye at their C++ testkit that runs for 8+ hours, primarily checking for errors that rust would never let you make in the first place.

benesch2• 6 years ago

Part of what makes rustc's slow compile times so frustrating is that the slow compile times are mostly unrelated to all of the fancy compile-time analysis you listed. You can see this in action if you run `cargo check`. That will run all the special static analysis (typeck, borrowck, dropck, etc.) and complete in a very reasonable amount of time.

To a first approximation, rustc is slow because it generates terrible LLVM IR, and relies on a number of slow LLVM optimization passes to smooth that IR into something reasonable. As far as I can tell this is entirely a result of not having the necessary engineering resources to throw at the problem; Mozilla has a limited budget for Rust, and needs to spread that budget across not just the compiler toolchain, but the language and ecosystem.

Brian Anderson's been working on a blog post series that covers Rust's slow compile times in more detail, if you're curious: https://pingcap.com/blog/tag/Rust

ronjouch• 6 years ago

I think your and OP's perspectives are not exclusive:

A. Yes, as you say, taken from a distance and considering the whole development lifecycle from idea to shipping, "compile time isn't really that bad for rust when you consider what it actually does for you".

B. However, -during- development, you want compile speed, regardless of all the benefits. I don't know about you, but I have a hard time rationalizing "it's okay, rustc is doing many things" at that moment. I just want it to be faster, and as OP mentioned, I hope there's a future where companies / sponsors get to "hiring about 25 people" to crunch and improve things noticeably (and I will do more than hope, I already donate money to Ferrous Systems to improve rust-analyzer, and will donate if a "make rust faster" fund asks for it). Your argument sounds to me a bit like "who cares about making Python faster? It's just glue". To which I answer: maybe (though not always), but even then but I'll take faster glue anytime :) .

To sum up: rust being already awesome and being "fast given all the thing it does" doesn't mean it shouldn't be faster, and doesn't mean speaking about it is futile.

sebazzz• 6 years ago

Typescript allows to compile without type checking at the sacrifice of correctness. If you would have a slow computer or huge code-base, you could consider using the type checker only for tests, pre-checkin check, or CI. (Not that Typescript is slow - but as an conceptual example)

Would something similar be possible for Rust, so you still have the ability for correctness for CI and release builds, but allow for fast compilation when necessary or desired?

justwalt• 6 years ago

I’ve heard someone say about the rust compiler that it will never allow compilation without its safety checks in place because it would decrease faith in any rust binary.

MinusGix• 6 years ago

Typescript would have a far easier time in that scenario because much of it's syntax is simply Javascript with extra types attached. So, it could (I don't know the implementation details at all) 'just' strip away all the typescript-syntax, and give you your Javascript files. Rust has the issue that there's a lot of type-checking done, even in basic scenarios: such as setting a variable `let alpha = some_function(abc);`, you have to do some form of basic typechecking here to get the type of `alpha` (since that affects how it's stored on the stack). The simple case of `fn some_function(abc: u32) -> i16;` would be 'easy', but more complicated scenarios like generics would make your life harder. Though, I'm sure there's parts of the checking that could be eventually written so that they can be turned off, but I don't think it would provide nearly as much benefit as Typescript's non-correct compilation. Personally, I think it would be better to spend that time on just making the compiler faster. This is perhaps easier than I made it sound, but this is my assumptions as to why it wouldn't work that well.

entha_saava3• 6 years ago

Type checking doesn't seem to be major bottleneck at all. The major issues seem to be

1) Technical debt in rustc producing large amounts of LLVM IR and expecting LLVM to optimize it away

2) generic monomorphization producing vast amount of IR

IIRC.

masklinn• 6 years ago

Would something similar be possible for Rust

Yes. The `mrustc` alternative implementation doesn't do things like borrow checking, it assumes the source is "correct".

[skipping typechecking would] allow for fast compilation when necessary or desired?

The real expenses in a "release" compilation pipeline are in the optimisation passes and codegen, so the gains are mostly in avoiding optimisations (debug build) or avoiding codegen entirely (`cargo check`).

For instance on the latest ripgrep head on my machine:

* cargo check takes ~200s user

* cargo build (debug) takes ~300s user

* carbo build --release takes ~900s user

alfiedotwtf• 6 years ago

I’m not sure why everyone cares so much about compile times compared to the only thing that matters - runtime performance.

For the lifetime of a binary, compilation time is like 0.000001% of its existence.

I’m more than happy to wait longer compared to other languages (even zero time at all with scripting languages) while cargo does it’s thing.

mhluongo• 6 years ago

Business reason: When you're in the business of building software, people are your biggest expense. You want to waste their time as little as possible.

Dev reason: tight feedback loops are encouraging, and make it easier to build momentum. I hate being deep in a problem then having to "wait" for a compile.

hellofunk5• 6 years ago

But compile times have a massive effect on developer time, which has a massive effect on the productivity and the amount of work that can actually be done.

Comparing compile time to run time is rather strange, but compared as a percentage of a developer’s time, it’s very obvious why people are worried about it.

zelly5• 6 years ago

Yes, we should have a national politburo assign funds to Rust development. Although most of private industry (proles) disagrees, us in the Administration believe that Rust is superior. The proles--ahem I mean programmers with jobs--keep causing problems by using Unsafe technologies and being too dumb to use them. Rust should be funded by The People. It is a Public Good after all because using anything else would be an act of defiance against the will of The People.

fluffything5• 6 years ago

increasing the number of people paid to work on the Rust compiler by 10x would only mean hiring about 25 people.

Citation needed ?

I know at least 20 people being paid to work on the Rust compiler, which crank's your 10x number from 25 to 200. And this is only the people I can come up with from the top of my head.

crb002• 6 years ago

https://github.com/StanfordSNR/gg - should be almost the same as the GCC/LLVM thunk extractor. You have to pay for the borrow checker, but LLVM IR optimization passes should be the same complexity.

Ericson23144• 6 years ago

Happy to see happening!

But to get the most value out of Rust, one needs to use the Cargo ecosystem too (even for kernel code!) and that will be a much bigger political quagmire.

geofft8• 6 years ago

We've got Cargo wired up in our prototype, which can let you link modules against no_std crates on crates.io: https://github.com/fishinabarrel/linux-kernel-module-rust

Probably one good end state is if the core kernel exposes appropriate safe abstractions, and vendors with out-of-tree modules (which tend to be worse at security than the main kernel) could use those APIs + things from crates.io to build their modules.

But yes, figuring out how to use useful things that aren't in libcore is one of the open questions.

Ericson23144• 6 years ago

Glad to hear it! I just hope Linux doesn't end up with the crates.io vendoring nightmare that the big-corps made for themselves.

doonesbury4• 6 years ago

Formal leaning HN people: Does Rust just have better sales & marketing than C++? Or is it really better for safety and correctness?

* Rust is NOT a programming language with deep, well-integrated support for proof systems or state exploration e.g. TLA+.

* Rust is no Frama-C where static analysis and function contract support is far more apparent and important

* Rust is not ADA

* The previous items also arise and are quite important as a direct result of generality: to the extent Rust imposes (albeit with benefit) language design to avoid certain kinds of functional issues, those features can later become limitations as a pushy, over opinionated language. Therefore to have flexibility in designing fast code that's yet correct code one usually has to have the low-level ability of C/C++ augmented with formal tools.

I'm not seeing Rust do that. Now, I've made a couple of jokes elsewhere in HN about Rust's propensity to show up uninvited to every vaguely language issue and press its case --- ex. I was vacuuming the floor yesterday when Rust somehow got through the front door and was explaining how a Rust vacuum is so much better, or how I'd never need to vacuum again if I used Rust. But I am only half joking.

For the last year I've spent much of my time in GO after getting sick-and-tired of C++ build/language complexity for the previous 10+ years. I am somewhat aligned to others who rightly ask if speed is all that matters? (Recall the Meyers joke about Pavlov training and C/C++ programmers when it comes to speed).

Correctness counts too and if the compiler spends a little bit more I time I'd be fine with that if for distributed/parallel programming I've got good guarantees.

For those who use TLA, SPIN, Frama-C and other formal tools what's your take on this? I think designers of those kinds of formal tools will be surprised to learn some of their correctness aims were solved by Rust.

geofft8• 6 years ago

Ada doesn't do safe dynamic memory deallocation, right? You'd need Ada.Unchecked_Deallocation.

That's fine in cases where you can statically define everything up front (which you can for many of the use cases that people use Ada for), but for a general-purpose OS kernel, it seems like you wouldn't get the level of safety or correctness from Ada that you would from either Rust or C++.

(Disclosure - I'm one of the authors of the prototype mentioned in the thread. I'd happily use Ada or Frama-C if I thought it would solve the problems needed for getting safe code into the Linux kernel, but it doesn't seem like they actually do.)

doonesbury4• 6 years ago

I think this thread is too narrow on correctness. Plenty of memory issues in C/C++ can be solved by library code i.e. not char buf[...] but std::string. Not raw pointers but RC-pointers. The idea that language strictures alone solve a substantial part of code correctness I think is something that people with more experience in particularly CSP computing ala TLA/SPIN would seriously doubt. And even on the nit pickier things ... RUST is gonna help me write lock-free code with CAS, memory fences? Really?

roca25• 6 years ago

Rust makes it easy to write code that's very expressive (dynamic memory allocation etc) and has strong safety properties. Certainly easier than having to verify with SPARK or Frama-C. Rust's type system, especially traits and affine types, lets you encode a lot of important properties (e.g. typestates).

Rust does need tools to verify Rust code correctness and especially to verify unsafe Rust code. There is a lot of work going on in this area: see https://alastairreid.github.io/rust-verification-tools/ for a sample.

doonesbury4• 6 years ago

I'm gonna look at rust; we're a huge c/c++ shop so there's room from improvement here. Now no program is a spec and implementation in one. Thus rust ought to be careful about keeping all it's hammers in language alone. If rust is smart and particularly wants to move to distributed computing or csps correctness will have to take on more complex meanings that bad index or divide by zero.

ComputerGuru3• 6 years ago

These threads always devolve into "rust is too slow" written by developers (or enthusiasts) that have never written no_std code in production. I've written and shipped firmware for embedded devices written in rust, yes, still using cargo and external crates, and had zero issues with compile time because the nature of the dependencies in the deps tree is different and very carefully curated.

Anyway, I really just wanted to point out that from the mailing list we have Linus and Greg endorsing this experiment/effort from the Linux side and a commitment from Josh on behalf of the rust team to grow the language itself with the needs of the kernel in mind. That's quite impressive and more than I could have hoped for.

I've actually played with writing kernel code in rust - for Windows/NT, however - and it's quite weird to be able to use such high-level type constructs in code where you typically manually chases pointers and wouldn't be surprised to see statically allocated global variables used to monitor reference counts.

ksec2• 6 years ago

Linus, normally being C has its quirks but is great and Anti C++ in general is endorsing Rust?

Why is that? Did he ever gave out his reasons?

steveklabnik22• 6 years ago

Linus has commented on Rust twice (that I'm aware of).

First, back in 2016:

Q: What do you think of the projects currently underway to develop OS kernels in languages like Rust (touted for having built-in safeties that C does not)?

A: That's not a new phenomenon at all. We've had the system people who used Modula-2 or Ada, and I have to say Rust looks a lot better than either of those two disasters.

I'm not convinced about Rust for an OS kernel (there's a lot more to system programming than the kernel, though), but at the same time there is no question that C has a lot of limitations.

https://www.infoworld.com/article/3109150/linux-at-25-linus-...

Then second, much more recently:

People have been looking at that for years now. I’m convinced it’s going to happen one day. It might not be Rust, but it’s going to happen that we will have different models for writing these kinds of things.” He acknowledges that right now it’s C or assembly, “but things are afoot.” Though he also adds a word of caution. “These things take a long, long time. The kind of infrastructure you need to start integrating other languages into a kernel, and making people trust these other languages — that’s a big step.

https://thenewstack.io/linus-torvalds-on-diversity-longevity...

I was once in the same room as him at a Linux Foundation event, and really wanted to ask him about it, but also didn't want to be a bother.

Note that the C++ opinion everyone cites is from 2007. I don't know how he feels about C++ today. It seems he's using it for some things at least. I know I've changed a lot since 2007, and so has C++.

geofft8• 6 years ago

I don't know Linus's reasons specifically, but our presentation at Linux Security Summit last year laid out why we think that Linus's past objections to C++ don't apply to Rust. See slides 19-21 of https://ldpreload.com/p/kernel-modules-in-rust-lssna2019.pdf .

His previous objections were:

    In fact, in Linux we did try C++ once already, back in 1992.

    It sucks. Trust me - writing kernel code in C++ is a BLOODY STUPID IDEA.

    The fact is, C++ compilers are not trustworthy. They were even worse in 
    1992, but some fundamental facts haven't changed:

     - the whole C++ exception handling thing is fundamentally broken. It's 
       _especially_ broken for kernels.
     - any compiler or language that likes to hide things like memory
       allocations behind your back just isn't a good choice for a kernel.
     - you can write object-oriented code (useful for filesystems etc) in C, 
       _without_ the crap that is C++.

In brief, Rust does not rely on C++-style exception handling/unwinding, it does not do memory allocations behind your back, and its OO model is closer to the existing kernel OO implementation in C than it is to C++'s model. (There are other good safe languages besides Rust that I personally like in general but do not satisfy these constraints for this particular use case.)

pjmlp15• 6 years ago

Why it is weird? It has been done before in Ada, Object Pascal, C++, Mesa/Cedar, Modula-2, Lisp, Oberon, Sing#, System C#, Objective-C,....

ComputerGuru3• 6 years ago

Right, and it would be "weird" if I were using one of those, too. You don't run across drivers written in any of those languages on a daily basis.

I didn't mean it with any negative connotation, btw.

megous3• 6 years ago

Does rust/llvm even support all the CPU architectures Linux supports?

If not how will that work? Will some drivers be then limited to only some architectures?

ndesaulniers4• 6 years ago

No, but it's already the case that some drivers are already tightly coupled to an architecture. Does it make sense to build a driver for some internal piece of hardware from an ARM SoC for PPC? Generally, no.

megous3• 6 years ago

The kernel contains vast number of drivers that are for universal buses like I2C, SPI, USB, PCI(e), SDIO, UART, writing any of these in rust would make these drivers unavailable to architectures unsupported by llvm. I think that would be quite sad.

daenz3• 6 years ago

6 months ago, as I hopped on the Rust train, I told a friend of mine that I'm convinced that Rust will "eat the world" like Python did[0]. These moves to see Rust enter the Linux kernel only prove to me that this pace is starting to pick up.

0. https://www.techrepublic.com/article/python-is-eating-the-wo...

BlahGod4205• 6 years ago

I've been keeping an eye on crates.io stats - it's definitely booming. ~50 new crates a day (some of them are total nonsense/tutorial work/etc.).

I keep wondering about whether the hype is justified - I must be slowing down because I find myself hanging for long periods of time on borrow checker errors. One of the errors has stopped my progress dead for a week now, I swear it worked a week ago and then Rust decided that a borrow I was doing was no good.

I've also heard the borrow-checker is a common hold up for new Rust developers, so I keep slogging on.

staticassertion• 6 years ago

A week? Might make sense to refactor, or just clone the data? Or ask the subreddit, discord, or whatever place you want for help.

I haven't run into a borrowchecker issue that took me more than a few seconds in a few years. The trick is I'm not crazy about avoiding cloning - Rust is still super fast. The curse of Rust making cloning explicit, and move by default being cheap/ memcopy, is that you often feel like "if only I could express it" and that problem seems to get worse the better you are, and then for me I hit a point where I stopped caring and just got shit done.

It's easy to remove clones later in my experience, once benchmarks show it matters.

coldpie4• 6 years ago

I'm also relatively new to Rust, just about to ship my first ever project written with it. Borrow checker errors are definitely tricky. In my experience, it usually indicates that you've done something "the wrong way," which means you need to re-think it and architect it somehow else. There's often not a quick fix (unless you just made a typo or something, of course). For example, think about who owns the data in question, why it's trying to be stored in two places at once, and how you can re-arrange your data structures or code flow to avoid that condition. It gets harder to understand once lifetimes get involved, I still have to dig into the book to understand the ins & outs of that, and its syntax. Once you get the hang of writing code in a Rust-friendly manner, borrow checker errors mostly just don't happen, because you wrote your code "the right way" to begin with.

I hope I described that well. It's a fascinating language, and now that I'm on the far side of the learning curve, I find it very pleasant to use. I urge you not to give up.

(Have you read through and written, built, and played with the examples in the book? I found it extremely helpful to follow the exercises, and intentionally break them, to test my understanding. https://doc.rust-lang.org/book/ )

_bxg1• 6 years ago

1) Use rust-analyzer if you aren't already. It decreases the time it takes to see compiler/borrow error results by about 10x. This helps enormously when you're banging your head against something, taking guesses to see what might work.

2) It's often better to slow down and think through what's going on. Make sure you have a good understanding of pointers, allocation, ownership, etc, and if you've been fighting with something for more than 10 minutes see if you can actually understand why it's upset. Rust makes it easier not to accidentally mess these things up, but you will still be chronically frustrated if you don't have a good grasp on them.

3) I've found that the more I use it the more I get an intuition for what fixes certain errors. I don't feel great about fixing something without fully understanding it, but for certain common cases it can really help with the problem of iteration speed.

daenz3• 6 years ago

Don't get too hung up on fancy structs containing borrowed data with complicated lifetimes. Zero-copy operations via references are an optimization that you can often forego unless you need really high performance code.

JoshTriplett5• 6 years ago

I remember it taking me a solid month or two (of side-project experimentation, not daily work) before I felt like I'd really internalized the right mental model for ownership and borrowing.

steveklabnik22• 6 years ago

Please don’t hesitate to ask for help! The forums and discord are there as a resource.

incompatible• 6 years ago

Generally a language with automatic garbage collection is easier to work with. Rust can still be useful in areas like the kernel.

estebank6• 6 years ago

Could you share what the problem you're having is? People in https://users.rust-lang.org/ and https://reddit.com/r/rust are more than happy to help, and if I could peek at a repro case running in https://play.rust-lang.org/ I could try to help you too (and potentially fix the diagnostic in the compiler). Also, depending on what the problem is, you might have an easier to understand error in newer versions of the compiler, which is why I recommend newcomers to use nightly and update often: not because they'll use advanced nightly-only features but because we merge diagnostics improvements aimed at newcomers every week.

Also, don't get too discouraged. If you can avoid doing things that require advanced lifetime checking then you can gain enough experience with the rest of the language letting you focus only on that part of the language at a later time. Also, I can confidently tell you that at some point it clicks and you start anticipating things that can cause issues.

nilkn10• 6 years ago

Out of the box, Rust encourages almost everything to be stack allocated, imposes a read-write lock on every value (i.e., you can have multiple immutable readers, but only one mutable writer at a time), and uses move semantics for variable assignments (i.e., values aren't copied but rather ownership of the value changes from one variable to some other variable).

Sometimes you need to drop one or several of these properties. It's very easy to do that -- Rust just makes you be explicit about it. If you want something to be heap allocated, you can use a Box/Rc/etc. If you want a value shared among multiple writers, there are several options; a common one is Arc<Mutex<T>>, which is also thread-safe. If you don't want move semantics, you can just copy the value using .clone().

A few lessons to take away, though:

* Because of that read/write lock I mentioned, it follows that you generally want to avoid taking a mutable reference to an entire data structure, because doing so effectively puts that writer lock on the whole thing. If you can take a mutable reference only to the piece you actually care about, that can help out a lot in some cases.

* Rust makes you feel like you should never clone, but really it's up to you. Exercise basic judgment that you'd use in any other language, and don't let perfect be the enemy of the good. In many cases it's probably not nearly as expensive as you might feel it is, and sometimes it'll just be optimized away in the end anyway.

* Try to stick with structs and data structures that own their data as your default, deviating when it makes sense. I've seen people try to avoid ever cloning values by having many structures only contain references, which leads into complicated lifetime management that they weren't prepared for (and that may not even be possible or make sense).

* Take a moment to understand the various wrapper types (I like [0]). Intuitively, Rust gives you a set of guarantees that you can ask for, and you just pick the guarantees you need and compose them together.

[0] https://manishearth.github.io/blog/2015/05/27/wrapper-types-...

tptacek13• 6 years ago

This is a deeply weird comparison.

Insufferably lazy "eating the world" headline aside, Python is incredibly popular because its runtime performance is "good enough" for a huge fraction of applications, its safety guarantees are leagues better than C/C++, and it's deliberately simple to write (it was designed in part as a pedagogical language).

Rust's performance is leagues better than Python's, it's safety guarantees are better by degrees, and it is substantially harder to write than Python, not just because it requires a more sophisticated understanding how how programs are constructed and verified, but also because it requires developers to think carefully about memory management.

I fully expect Rust to "eat the world" of C/C++ code over the next 10 years, and look forward to a time where we'll look askance at kernel code still written in C. It will no doubt be important in a bunch of performance-critical settings (AAA gaming, browsers, network infrastructure) as well.

But Rust will never "eat the world" of general applications (that is, the overwhelming majority of applications), because Python is good enough across all axes for these applications. It would be engineering malpractice for most companies to rewrite working Python applications in Rust. That was not the case for C-to-Python (or even PHP-to-Python) rewrites.

In case this isn't obvious to anyone reading this thread --- though I'd be surprised --- the reason Rust is in the Linux kernel, while Python isn't, is that Rust was designed to be interoperable with kernels and browsers, and Python, like most other languages that have ambitious runtimes, was not. It's not an indication of Rust's popularity, but rather just Rust doing something it was designed to do.

(Rust is plenty popular.)

baby2• 6 years ago

Javascript did this.

person_of_color• 6 years ago

Ok, you've convinced me. What's the best way to learn Rust in 2020?

daenz3• 6 years ago

I think you're misunderstanding the comparison to Python. I am saying Rust will eat the world in the space that it fits in, which is still relatively large, in the same way that Python did, in the space that Python fits in. Not that Rust will take over very high level languages too.

JoshTriplett5• 6 years ago

As a Rust developer, I completely agree. I'd like to see Rust fully supplant C++ and C, but I don't expect it to fully supplant scripting languages, and I definitely wouldn't suggest rewriting working Python in Rust unless you need to improve performance or scalability, or you're having serious maintainability problems. Let many languages bloom.

blandflakes• 6 years ago

Yes! The number of settings where $RUST_COMPETITORS are applicable is simply smaller than the number of settings where a high-level, dynamic or at least garbage-collected language is applicable. It's hard for me to imagine having a real justification for building services in Rust given the relative productivity of those high-level languages.

This is basically the reason I haven't picked it up for any personal projects, either. I admire the goals, but my interests don't call for a C++ killer.

fctorial• 6 years ago

Python is incredibly popular because its runtime performance is "good enough" for a huge fraction of applications

Is there anything slower than pure python?

cheez• 6 years ago

I'm pretty sure that's exactly what he meant.

jariel4• 6 years ago

Python is high level 'tool' - accessible to grad students from various fields looking to do 'some things' with 'data'.

Rust is effectively for 'software developers'.

Also: ", because Python is good enough across all axes for these applications.", no, Python is good for a variety of things, but not remotely for every thing. Javascript/Typescript, C#, Java and its offspring, and probably Golang, are going nowhere. And of course Swift.

Finally: incumbency is more powerful than anyone expects. If C/C++ 'works' in any given scenario, it will be difficult to replace. So many embedded systems, so much hardware out there.

Think of how much C++ Code Adobe has invested in. Think of the massive array of sophisticated Autocad products all written in C++. They are not going to be refactored. The native parts of the JVM, Oracle DB, the list goes on.

I predict in 10 years 'greenfield' projects might favour Rust, but many won't.

wolco• 6 years ago

PHP to Python rewrites? That seems more foolish then Python to Rudt. I have seen a few Python to PHP rewrites.

Python is taught in school instead of java now which took over c/c++ as the default language.

remexre2• 6 years ago

Python is incredibly popular because its runtime performance is "good enough" for a huge fraction of application
But Rust will never "eat the world" of general applications (that is, the overwhelming majority of applications), because Python is good enough across all axes for these applications. It would be engineering malpractice for most companies to rewrite working Python applications in Rust. That was not the case for C-to-Python (or even PHP-to-Python) rewrites.

I dunno, I sort of hope that there'll be good and resource-light application development frameworks in Rust that do "eat the world." I spent most of a summer two years back using an (at that point) 12-year-old machine without X for both my personal and work computing. I did browsing with w3m+imlib2, images and PDFs with fbi and fbpdf (iirc), and fbterm (and tmux) as a terminal emulator. That was leagues more responsive than the firefox/eog/zathura/konsole setup I reverted to once the school year started (and I needed to run webapps again).

If you're familiar with https://en.wikipedia.org/wiki/Wirth's_law I think it applies really strongly to this case, and it's gotten remarkably worse as programmers have switched to making general applications with Python and Electron at the same time as single-threaded clock speeds have slowed down.

ncmncm3• 6 years ago

Now there is no legitimate reason to lock out C++.

josephcsible3• 6 years ago

A lot of the reasons that Linus doesn't want C++ aren't true of Rust.

arcticbull3• 6 years ago

The biggest issue with C++ is it's an absolute atrocity -- an agglomeration of every feature under the sun, but half-built as many of those features can't possible co-exist.

I mean, for example, std::move doesn't move... anything. std::move "just" static_casts the receiver as a template<class T> typename remove_reference<T>::type&&. Not that any sane human knows what that means. The source object remains alive and potentially invalid. Who even knows? Certainly not the compiler. This is just one example. All of C++'s new features are built like this. And the old ones are just so fragile.

It's an incredibly dangerous, convoluted and absurd language, and every time I turn around it gets worse. It's time to leave C++ in the dustbin. It is an octopus made by stapling extra legs onto a dog.

geofft8• 6 years ago

Here's our presentation at Linux Security Summit last year outlining why I think you can legitimately object to C++ and not Rust for this particular use case: https://youtu.be/RyY01fRyGhM?t=313 (alternatively, slides 19-21 of https://ldpreload.com/p/kernel-modules-in-rust-lssna2019.pdf)

jhauris• 6 years ago

Those slides are great. I love modern C++, but reading those slides makes me excited about trying out rust.

rhinoceraptor• 6 years ago

Much of C++ is inappropriate for a kernel. And defining which parts of C++ could be safely allowed would take so much time and bikeshedding that it's just probably not worth it.

JoshTriplett5• 6 years ago

There are plenty of reasons. As just one of them: C++ is unsafe by default. There's theoretically a safe subset, but there's no precise or consensus definition of that subset. There's no type-system or language support for helping you stay strictly within that subset (and requiring some kind of marker for if you don't). Often, the most obvious way of doing something is unsafe; for instance, in C++, arr[i] is unsafe, and you have to write arr.get(i) to be safe. (In Rust, arr[i] and arr.get(i) are both safe, and you have to write arr.get_unchecked(i) in an unsafe block if you truly want an unsafe indexing operation.)

tptacek13• 6 years ago

Can you help me understand why locking C++ out of a kernel written almost entirely in C, and which receives huge volumes of new C code every year, wouldn't be an instance of making the perfect the enemy of the somewhat marginally better?

jariel4• 6 years ago

The 'unsafe' part of C++ is not the reason.

C is already unsafe enough.

C++ is a Frankenstein of a language, can't really be parsed, so many idioms, nobody agrees on the common subset.

If there was a C++ totalitarian who ruled with an 'iron set of idioms, conventions and best practices' maybe 20 years ago and forced the world into his benevolent bidding - then C++ might be useful there, even if 'unsafe'.

The memory safety value of Rust is way overblown. That's an important feature but not 'the' feature. Most C++ programs are not dying from memory problems. Rust's value in everything else as well.

ausjke• 6 years ago

While I like Rust I still picked Modern C++(17) for my next project. All modern languages are converging these days.

Adopted C++ because of its libraries at large. Will still learn Rust on the side. For the moment, boring and verified for product especially when they're updating and remaining up to date.

29athrowaway• 6 years ago

The problem I perceive with Rust as a language for Linux, is not that Rust is not an excellent language. It is that it evolves rather quickly and code may break when you switch versions.

staycoolboy• 6 years ago

Hmm... Might be time to start looking into Rust. Is it popular in college? I think that's what made Python displace Java.

EDIT: lol. Decided to dive in a hit this almost immediately: https://docs.rust-embedded.org/book/start/io.html

steveklabnik22• 6 years ago

You can count the college courses on two (or maybe three) hands. Not quite there yet.

coldpie4• 6 years ago

That's the guide to embedded programming. If you're just looking to learn Rust, start here: https://doc.rust-lang.org/book/

leoh• 6 years ago

No, but C/C++ is widespread and the relative understandability of Rust for those coders and advantages of Rust will certainly aid its adoption.

roca25• 6 years ago

Rust is probably never going to be the ideal first language for anyone to learn. This is a problem because a lot of people will turn to the language they first learnt, which they know best, when faced with any given problem :-(. But it makes sense that the easiest language to learn might not be the best language to use to get real work done...

asveikau2• 6 years ago

Just stating something obvious since I don't see it noted here: Linus is a smart guy with this idea of not wanting it to be some niche feature that nobody enables and hence nobody cares about and sees breakage from.

merb• 6 years ago

I think what he did has nothing to do with smart. it's experience that matters here, he learned such stuff, sometimes even on the hard way.

armitron2• 6 years ago

It takes smarts to learn / abstract from experience. The smarter you are the better you do.

asveikau2• 6 years ago

The smarts, I guess, is in recognizing which pieces need this treatment and which do not.

Also, he is suggesting this change to experienced people who did not do it that way. So experience is not 100%.

blondin2• 6 years ago

what does linux kernel in-tree support mean? that drivers can be written in rust or something?

ndesaulniers4• 6 years ago

Yes. In tree means their sources are in the Linux kernel git repository, as opposed to out of tree drivers. In tree drivers are maintained by the maintainers, so API changes are done together for all in tree drivers. Out of tree drivers don't get this benefit, and are more difficult to build in my experience.

blondin2• 6 years ago

thank you!

Shorel• 6 years ago

My only complaint about Rust is terminology.

"Immutable variable" what an oxymoron!

That's what is known as a constant!

A variable is mutable, a constant is not. Was it that hard to understand?

zelly5• 6 years ago

To bootstrap Rustc you have to compile every single version. Say goodbye to reproducible builds. Say goodbye to civilians with cheap laptops being able to compile their kernel.

hisham_hm• 6 years ago

This is a strawman argument. To be reminded of what it takes to truly "bootstrap" a C compiler, I recommend everyone to read Ken Thompson's Turing Award lecture "Reflections on Trusting Trust".

zelly5• 6 years ago

gcc can still be bootstraped from just an assembler and linker[1] with very few binaries you have to trust and it is possible on comoddity hardware. Bootstrapping Rust is so difficult that adding it to Linux would for all purposes eliminate this feature (being able to be built completely from source) from Linux, probably leading to a fork of Linux without Rust for those that care about it.

[1] https://guix.gnu.org/manual/en/html_node/Reduced-Binary-Seed...

Linux kernel in-tree Rust support

Related Stories