BrainFlood: Runtime code generation via reflection in .NET

https://sbox.game/churchofmiku/brainflood/news/brainflood-compiling-via-reflection-8089c180

by thunderbong • 1 year ago

86 25 1 year ago

Ah, yes, this is delightfully useless! Well, maybe good for stress-testing language tooling (that single huge type expressing the Mandelbrot generator is... something to behold), but mostly just cool-that-this-can-be-done.

Another cool thing that I learned about via this post (as I was wondering about the .scene files in the GitHub repo) is this whole ecosystem:

S&box is coded in C#. Under the hood, it uses the Source 2 engine (CS2, HL:Alyx, DOTA2) and some of its systems: rendering, resources, physics, and audio

And people say there is no more Good Stuff on the Internet...

pjc50• 1 year ago

This is quite something. I suspect this person has a C++ background, because the thing where they're abusing the generic type system to do arithmetic is very much more like C++ template libraries than something you normally see in C#. The advantage seems to be that you can use MakeGenericType() in situations where the normal bytecode Emit() isn't available?

kittoes2• 1 year ago

I wouldn't exactly call it "normal", but that's partially because a lot of C# developers still don't know about the relatively new generic maths feature. Those who are familiar regularly "abuse" the system in a similar way because that's one of the major benefits of it. We don't necessarily reimplement basic arithmetic using the type system, but we regularly hoist generic constants into static properties in order to convince the compiler to behave the way we expect. For example: T.PopCount(value: T.AllBitsSet) is a safe and generic way to express the size of a binary integer as a constant.

neonsunset2• 1 year ago

For example: T.PopCount(value: T.AllBitsSet) is a safe and generic way to express the size of a binary integer as a constant.

Despite the name, 'Unsafe.SizeOf<T>()` is the safe and preferred way to do this - it is guaranteed to be a compiler constant (in the past you'd say "JIT constant" but you can no longer assume that with the advent of NativeAOT).

It's a bit unfortunate because much like with MemoryMarshal, not all methods are equally unsafe. Some are benign or safe like this one, same are Unsafe.As or, worse, Unsafe.AsPointer.

_cogg2• 1 year ago

Author here. My background is in rust if anything. Honestly, I prefer to avoid playing these clever games with type systems unless it's really necessary. It just ended up being the perfect tool for this very specific problem. (How do I compile code in a sandbox that exposes a limited reflection API?)

Timwi• 1 year ago

Sounds like an awesome article, but it's unreadable on mobile (cut off on both sides) so I'll have to wait to read it until I get to a PC. Might be worth fixing

nguyenkien• 1 year ago

Your mobile browser doesn't have reader view?

gwbas1c• 1 year ago

You shouldn't have to jump through hoops to read an article on a phone. It's 2025; mobile browsers have been mainstream since 2007.

maushu• 1 year ago

Desktop mode in mobile seems to work fine.

recursive• 1 year ago

The point is that normal mode doesn't.

saurik• 1 year ago

Maybe it has been fixed, but the article currently looks fine for me on an iPhone.

souenzzo• 1 year ago

Feels like a brainfuck interpreter version of magic(1), a clojureCLR JIT compiler implemented via macro.

1 https://github.com/nasser/magic

neonsunset2• 1 year ago

Practical .NET Optimization Abuse

This is not abuse, it's intended behavior, standard library itself likes to use it a lot internally. That's also how generics work in Rust!

This can be pushed further by passing data by ref's. You can also easily allocate structs in unmanaged heap (aka malloc and free) and then pass them by `ref` without using unsafe (save for construction and alloc/free). Just don't do it for structs which contain gcrefs. C# can lean very heavily into systems programming :)

There have also been a large amount of struct optimizations across the last few versions. So it is not a coincidence that RyuJIT/ILC can trade blows with LLVM (not always but still!).

Edit: Oh god, I saw what this refers to. I take it back. There is likely a better, idiomatic and compiler-friendly way haha.

MrLeap• 1 year ago

This is fascinating! Code generation is a thing that's solved a couple problems for me recently. The way I do it never feels great. Whenever I want to cache a big dictionary literal, I tend to generate the code as a string and write it to a file. I've yet to find a compelling upgrade that checks all the boxes. (The generated side has run in the kneecapped .net for unity, although the generator doesn't necessarily.)

I'm going to dig into your implementation.

kkukshtel• 1 year ago

You could source generate this directly into your code instead of dealing with file IO. Newer versions of Unity support source generators:

https://docs.unity3d.com/6000.0/Documentation/Manual/create-...

whizzter• 1 year ago

This is quite cursed, but does show that the .NET runtime system and JIT seems fairly robust even in the presence of this stupidity.

MarkSweep• 1 year ago

This is great and despite what the article says, an actually useful technique. The reason this performs well is the .NET runtime does monomorphization when the generic type parameters are value types (primitive types like int or user-defined structs). In modern .NET you would use static abstract methods interface methods to implement this feature. But if you have to run on the old Windows-only .NET Framework, you need to use the approach used in BrainFlood. See this article [1] for more details. An example of an official .NET library that uses this technique on .NET Framework is TensorPrimitives. See this file [2], the structs are defined at the bottom.

[1] https://blog.stephencleary.com/2022/10/modern-csharp-techniq...

[2] https://github.com/dotnet/runtime/blob/main/src/libraries/Sy...

NWoodsman2• 1 year ago

I am a fan of this particular repository by Jules Jacobs who similarly uses struct based generics (to create immutable collections.)

https://github.com/julesjacobs/ImmutableCollections/discussi...

It has similar "source generator" hallmarks like in OP's post, for example calling a generic constructor which is JITted to a deeply nested struct type.

It's (in my opinion) very hard to reason what the code is doing because it has that flavor of "writing code to generate source to generate code"

It's impressive. And i'm pretty sure it's only real hinderance, why it could not be adopted into the base library, is that the repo needs a phd holder to finish implementing the `IList` interface because deleting is really difficult and theoretical.

NWoodsman2• 1 year ago

This post has a lot of similarities to the 'object algebra' paper by Bruno Oliveira.

https://www.cs.utexas.edu/~wcook/Drafts/2012/ecoop2012.pdf

If the OP author has any interest in doing a 1000x service to the C# community, the community needs a translator like you to convert the paper's compiler example into other domain specific examples. I'm pretty sure this paper went nowhere because it lacked a hello world example. something easier to digest and smaller and scope. The classic "dog is an animal and cat is an animal" level of explanation.

WorldMaker• 1 year ago

you aren't in a sandboxed jail, you should probably just use System.Reflection.Emit

System.Linq.Expressions is a useful intermediate (that also works in sandboxes). People overlook its compilation power all the time, especially this long after the DLR (System.Dynamic and especially System.Dynamic's IDynamicMetaObject) has been forgotten, but it is a far easier API than System.Reflection.Emit.

I was half-hoping in this article to see a comparison with BF implemented in System.Linq.Expressions.

_cogg2• 1 year ago

Yeah, I was totally unaware of this. The namespace appears to be blocked in the sandbox -- probably rightfully so -- but now I'm curious as well. I might have to test it out and write a little addendum.

Edit: Looks like it's slower by a pretty good margin (~3.6 seconds). It's certainly possible I'm doing something stupid. This would be really nice to have for webassembly, where I'm convinced I need to implement some sort of virtual register allocation, and I'm still not entirely sure about control flow.

BrainFlood: Runtime code generation via reflection in .NET

Related Stories