ltratt

The downside of manually narrowing is more code, but the upside is clarity.

This, IMHO, is where things get tricky: quite often, writing more code is at odds with clarity. I don't think there's a simple relation here -- more sometimes is better! -- and for me at least I don't know how to define it any better than "I know it when I see it".

We did have a project some years ago looking at extending LuaJIT on which Tom Fransham did excellent work. Alas, the funder's priorities moved on (as is their right!), so we didn't get to finish it. It was a bit sad, as we (well, mostly Tom) had built up a real head of steam, but c'est la vie. Still, either way, I would never have claimed enough personal expertise with LuaJIT to take over maintainership!

This looks interesting, and definitely useful for non-C/Python languages, which existing reducers I know of mostly don't have explicit support for! I can't get it to build though (I've filed an issue).

I was also wondering: is there a UI while reduction happens? I've found Shrink Ray's UI improvements in the last year to be much more useful than I first expected: not just because it gives me something to look at, but because it really helps me understand if reduction is on the right path or not. [Some of the new Shrink Ray extras like being able to rewind reduction to a past point and to skip passes are also really useful too.]

It would really be good if someone could provide an updated overview of all of the "GCs for Rust" created thus far -- for a while I tried to keep up with them, but there are just too many! When we wrote the Alloy paper, we took Manish's survey as a starting point, and covered as many GCs of different kinds as we could squeeze in [1]. But even then there was no way we could squeeze _everything_ in, and I've seen at least 2 or 3 new ones since (safe-gc is older than the Alloy paper, so I'm not including it in that count)!

[1] https://soft-dev.org/pubs/html/hughes_tratt__garbage_collect...

If this project would be able to detect the interpreter hotspots itself and completely automate the procedure, it would be great.

I don't think that's realistic; or, at least, not if you want good performance. You need to use quite a bit of knowledge about your context to know when best to add optimisation hints. That said, it's not impossible to imagine an LLM working this out, if not today, then perhaps in the not-too-distant future! But that's above my pay grade.

You're quite right that since we're working with LLVM IR, adapting to other languages is probably not _that_ difficult, though these things always end up taking more time than I expect! Since the majority of real-world problems in this area depend on C interpreters, we put our limited resources to that problem. You're also right that "interpreters" is a pretty vague category, and there are other parts of C (and other) programs that could be yk-ified, though I suspect it would be a fairly specialised subset of programs.

Our fork of LLVM does add a pass, amongst other changes, but we also have to do things like change stackmaps in a way that breaks compatibility. Whether stackmaps in their current incarnation are worth retaining compatibility for is above my pay grade! So some of our changes are probably upstreamable, but some might be considered too niche for wider integration.

I'm assuming you're referring to the Python finaliser example? If so, there's no syntax sugar hiding function calls to finalisers: you can verify that by running the code on PyPy, where the point at which the finaliser is called is different. Indeed, for this short-running program, the most likely outcome is that PyPy won't call the finaliser before the program completes!

The only way to forbid it would be to forbid creating pointers from `Gc<T>`. That would, for example, preclude a slew of tricks that high performance language VMs need. That's an acceptable trade-off for some, of course, but not all.

We don't exactly want Alloy to have to be conservative, but Rust's semantics allow pointers to be converted to usizes (in safe mode) and back again (in unsafe mode), and this is something code really does. So if we wanted to provide an Rc-like API -- and we found reasonable code really does need it -- there wasn't much choice.

I don't think Rust's design in this regard is ideal, but then again what language is perfect? I designed languages for a long while and made far more, and much more egregious, mistakes! FWIW, I have written up my general thoughts on static integer types, because it's a surprisingly twisty subject for new languages https://tratt.net/laurie/blog/2021/static_integer_types.html

If you've used Chrome or Safari to read this post, you've used a program that uses (at least in parts) conservative GC. [I don't know if Firefox uses conservative GC; it wouldn't surprise me if it does.] This partly reflects shortcomings in our current compilers and in current programming language design: even Rust has some decisions (e.g. pointers can be put in `usize`s) that make it hard to do what would seem at first glance to be the right thing.

As Koffiepoeder suggests, since the vast majority of content on my site is static, I only have to compress a file once when I build the site, no matter how many people later download it. [The small amount of dynamic content on my site isn't compressed, for the reason you suggest.]

As an example, I like to point people at https://doc.rust-lang.org/std/cell/struct.UnsafeCell.html which for many years now has contained this line:

The precise Rust aliasing rules are somewhat in flux, but the main points are not contentious

I've sometimes found myself in situations where the only way I've been able to deal with this is to check the compiler's output and trawl forums for hints by Rust's developers about what they think/hope the semantics are/will be.

Historically speaking, this situation isn't uncommon: working out exactly what a language's semantics should be is hard, particularly when it has many novel aspects. Most major languages go through this sort of sequence. Some sooner or later than others --- and some end up addressing it more thoroughly than others). Eventually I expect Rust to develop something similar to the modern C spec, but we're not there yet.

Because Morello is an experimental platform, only a small number were manufactured. They are/were allocated mostly to people involved in early stages CHERI R&D and, AFAIK, none were made available to the general public. [That said, I don't know whether there are still some unallocated machines!] One can fully emulate Morello with qemu. While the emulator is, unsurprisingly, rather slow, I generally use qemu for quick Morello experiments, even though I have access to physical Morello boards.

You're quite right, I over-simplified -- mea culpa! That should have said "often unify these phases". FWIW, I've written recursive descent parsers with and without separate lexers, though my sense is that the majority opinion is that "recursive descent" implies "no separate lexer".

I not only welcome reasonable scepticism, but I do my best to facilitate it. I have accrued sufficient evidence over time of my own fallibility, and idiocy, that I now try to give people the opportunity to spot mistakes so that I might correct them. As a happy bonus, this also gives people a way of verifying whether the work was done in the right spirit or not.

To that end we work in the open, so all the evidence you need to back up your assertions, or assuage your doubts, has been available since the first day we started:

* Here's the experiment, with its 1025 commits going back to 2015 https://github.com/softdevteam/warmup_experiment/ -- note that the benchmarks are slurped in before we'd even got many of the VMs compiling. * You can also see from the first commit that we simply slurped in the CLBG benchmarks wholesale from a previous paper that was done some time before I had any inkling that there might be warmup problems https://github.com/ltratt/vms_experiment/ * Here's the repo for the paper itself, where you can see us getting to grips with what we were seeing over several years https://github.com/softdevteam/warmup_paper * The snapshots of the paper we released are at https://arxiv.org/abs/1602.00602v1 -- the first version ("V1") clearly shows problems but we had no statistical analysis (note that the first version has a different author list than the final version, and the author added later was a stats expert). * The raw data for the releases of the experiment are at https://archive.org/download/softdev_warmup_experiment_artef... so you can run your own statistical analysis on them.

To be clear, our paper is (or, at least, I hope is) clear to scope its assertions. It doesn't say "VMs never warmup" or even "VMs only warmup X% of the time". It says "in this cross-language, cross-VM, benchmark suite of small benchmarks we observed warmup X% of the time, and that might suggest there are broader problems, but we can't say for sure". There are various possible hypotheses which could explain what we saw, including "only microbenchmarks, or this set of microbenchmarks, show this problem". Personally, that doesn't feel like the most likely explanation, but I have been wrong about bigger things before!

Hello Filip -- I hope life is treating you well! I'm happy to clarify a couple of things that might be useful.

First, VM authors I've discussed this with over the years seem roughly split down the middle on microbenchmarks. Some very much agree with your perspective that small benchmarks are misleading. Some, though, were very surprised at the quantity and nature of what we found. Indeed, I discovered a small number had not only noticed similar problems in the past but spent huge amounts of time trying to fix them. There are many people who I, and I suspect you, admire, in both camps: this seems like something upon which reasonable people can differ. Perhaps future research will provide more clarity in this regard.

Second, for BBKMT we used the first benchmarks we tried, so there was absolutely no cherry picking going on. Indeed, we arguably biased the whole experiment in favour of VMs (our paper details why and how we did so). Since TCPT uses 600 (well, 586...) benchmarks it seems unlikely to me that they cherry picked either. "Cherry picking" is, to my mind, a serious accusation, since it would suggest we did not do our research in good faith. I hope I can put your mind at rest on that matter.

For those of us with Unix-y mail setups the move to OAuth2 can be a bit tricky, but there are now several different programs to help (spurred, I suspect in no small part, by Microsoft/Exchange's stance). The ones I know about are:

Email OAuth 2.0 Proxy <https://github.com/simonrob/email-oauth2-proxy>; mailctl <https://github.com/pdobsan/mailctl>; mutt_oauth2.py <https://gitlab.com/muttmua/mutt/-/blob/master/contrib/mutt_o...> (some suggestion that it might not always work these days?); pizauth <https://github.com/ltratt/pizauth>; oauth-helper-office-365 <https://github.com/ahrex/oauth-helper-office-365>. Disclaimer: I wrote pizauth and it's just about to move into the alpha stage.

Honestly, I haven't tried Xvfb in years. That said, I did have another motive by keeping things simple. Even though xwininfo is very X11 specific, I hope that it's easier for people to work out an alternative for other platforms and adapt the recipes from my post to their situation.

I always try to be respectful towards other software in my writing and I fell short this time. I hope you'll accept my sincere apologies!

the tool chain target names, which I think was your real complaint

Yes, this was solely what I was referring to (slightly thoughtlessly) as "ugly", and even then only in the sense of "where did those magic names come from?" I certainly wasn't referring to objcopy itself!

It's absolutely possible, because the hardware doesn't care about your compilation model: you can mix normal pointers and capabilities as you wish. A challenge is that it's easy to go from capability -> pointer, but harder to go from pointer -> capability -- where do the extra capability bits come from? CHERI C provides a default ("inherit capability bits from the DDC") but I'm not sure that's what I would choose to do.

I agree that you don't want non-CHERI Rust code to have to know about capabilities in any way. However, if you are using Rust for CHERI, you need to have some access to capability functions otherwise, even in pure capability mode, you can't reduce a capability's permissions. For example, if you want to write `malloc` in purecap Rust, you'll probably want to hand out capabilities that point to blocks of memory with bounds that only cover that block: you need some way to say "create a new capability which has smaller bounds than its parent capability."

As to the utility of hybrid mode, I politely disagree. For example, is it useful for safe Rust code to always pay the size penalty of capabilities when you've proven at compile-time that they can't misuse the underlying pointers? A very different example is that hybrid mode allows you to meaningfully impose sub-process like compartments (in particular with the `DDC` register, which restricts non-capability code to a subset of the virtual address space; see also the `PCC` register and friends). Personally, I think that this latter technique holds a great deal of potential.

[Warning: over-serious response coming.] There's a big "cattle grid" sign a little way before it, so I don't think that's a realistic worry. As an aside, that road is long, but a dead end: it leads up to a beauty spot / hiking point. The road also largely lacks dividing lines (i.e. it's sort-of single track, though often wide enough for two vehicles), and the cattle grid comes not long after a sharp corner, so I can't imagine many vehicles would have been able to get to flying speeds. Besides, that neck of the woods has plenty of interesting driving roads including the main A39 out of Porlock just a couple of miles away, that I imagine are of more interest to speed demons.

I'm not an IDE person myself (I'm a neovim person), but I'd love to see someone integrate such an approach into an IDE! The algorithm is free available and the implementation in Rust (grmtools) the same. If you wanted to reuse the Rust code, and it needs some tweaks, then I'm sure that we can find a way to support both batch and IDE-ish use cases well.

Is this a separate thing that would be integrated into an IDE or would it make more sense as part of the compiler?

As things stand it's most easily used in a batch setting (e.g. a compiler). I don't think it would take much, if any, work to use in an incremental parsing (for an IDE), but I haven't tried that yet!

Alas, most parser generators don't have very good error recovery (and some have such terrible error recovery that I think it's worse than not having any!).

It turns out that this isn't inevitable: there's been a long strand of research on decent error recovery for LR parsers, at least, but it needed a bit of a refresh to be practical. If you'll forgive the blatant self promotion, we tackled this problem in https://soft-dev.org/pubs/html/diekmann_tratt__dont_panic/ which is implemented in our Rust parsing system https://github.com/softdevteam/grmtools/. It won't beat the very best hand-written error recovery routines, but it's often not far behind.

I also have a C920: it's fairly infamous for the "Smurf effect", where it tends to make everything look unnaturally blue. Setting the "white_balance_temperature" setting manually improves this situation substantially! Depending on your OS, you might have to first set a "white_balance_temperature_auto" to "off" and then set a "white_balance_temperature" in Kelvin. [In OpenBSD, we recently added this ability, but conflated the two controls into one. YMMV will vary by OS!]

PHP is an... interesting... language to implement. The de facto standard PHP interpreters have become very fast over time. It's been a long time since I measured HHVM, but it always used to have very good warm-up time. I always felt it was a bit of a pity that no-one took on HippyVM (PHP in RPython https://github.com/hippyvm/hippyvm): it implemented a very large chunk of the language, had decent performance, but still some low-hanging fruit to improve (I had some fun making fairly substantial performance improvements to some aspects of it). It would be interesting to compare HippyVM and Graalphp once they implement equivalent chunks of the language, especially if someone takes on maintainership of HippyVM!

My understanding is that Wagner's thesis is mostly concerned with "history-sensitive" error recovery in an IDE: using earlier syntax trees as an additional input to the error recovery process in order to better understand the user's intent when editing.

That matches my understanding (note that Wagner's error recovery algorithm is probably best supplemented with Lukas Diekmann's thesis, which corrects and expands it a bit https://diekmann.co.uk/diekmann_phd.pdf).

Because Wagner's algorithm relies on history and context, it does sometimes do some unexpected (and not always useful things). However, I don't see any fundamental reason why it couldn't be paired with a "traditional" error recovery approach (I have one to sell!) to try and get the best of both worlds.

For error recovery, do any parsers have a notion of which tokens are more likely to be wrong?

lrpar has a %avoid_insert declaration which allows users to say "these token types are better avoided if there are other equivalently good repair sequences". It's simple, but it works well https://softdevteam.github.io/grmtools/master/book/errorreco...