Your code is fast if you're lucky

by chrka • 12 days ago

136 85 12 days ago

jdw647• 12 days ago

I really envy programmers who are so skilled at this kind of low-level optimization.

The same meaning, but different performance based on notation—it's ultimately about entering LLVM's optimization pass, which likely comes down to differences in the internal IR pattern. It almost feels like a difference in innate talent...

I feel like I can build CRUD applications well enough, but I still seem to be weak at low-level processing.

Where can I learn these kinds of techniques? I'd appreciate any book recommendations.

po1nt• 12 days ago

I personally learned a lot from cpp conference videos. I would highly recommmend it.

jdw647• 12 days ago

I should study that. Thank you

khuey2• 12 days ago

I'm not sure it's a "technique" but the general insight worth taking away from this is that compiler authors often write optimizers to recognize specific patterns so writing your code in a more idiomatic form increases the odds an optimizer will be able to optimize it.

In this specific instance, at the hardware level it helps to understand how the branch predictor works and why quicksort in particular is essentially the worst case for the branch predictor, and then you'll understand why the cmov/csel optimization is such a big win.

jdw647• 12 days ago

About that kind of 'technique', I guess I should make it a habit to dig into the compiler, which is still a black box to me. I should study a few techniques myself. Have a good day

jasonjmcghee• 12 days ago

Casey Muratori's https://computerenhance.com is a good resource

jdw647• 12 days ago

thanks!

ivanjermakov• 12 days ago

Expose yourself to lower level technologies (compilers and optimization techniques, hardware history and design) and let curiosity guide you. Learn how to profile and analyze program performance.

nikhizzle• 12 days ago

Got to my first job out of college and they gave me core dumps and had me debug the kernel for a year. Not my kind of fun, but definitely got me skilled in the art of low level.

tux32• 12 days ago

You could read compiler books, but I would actually recommend reading about CPUs and computer architecture directly. If you understand how the hardware works, then the optimizations are all very natural and fit into the picture perfectly, instead of being some arcane compiler magic that you have to take as a disconnected fact.

Personally I actually haven't read too many books on optimizations, I just absorbed information over years one thing at a time, but something like Computer Organization and Design is a pretty good intro to the low-level picture. If you want to drown in extremely dense technical topics that will give you a lot of jumping off points to search, read Agner Fog's microarchicture optimization guide (https://www.agner.org/optimize/). It won't tell you what LLVM is doing, but it'll tell you why it's doing it. Fair warning, it's dense and pretty dry.

Then it depends how interested you are in doing low-level nonsense. If you spend a lot of time writing performance oriented systems code, you'll come to use profiling tools that show you the assembly. If you stare at it long enough, you sometimes start to question why the compiler wrote it this way. And you're naturally led as you try to optimize your code to wonder how LLVM is coming up with this ASM that it spits out and why it sometimes gets it wrong.

There's nothing magical or that requires innate talent. You can learn all of this very naturally if you work close to the metal and take the time to question how the abstraction layer below you actually works. If you keep doing this, you eventually find out it's not that deep, it's just a lot of stuff accumulated over time, but none of it particularly difficult or inaccessible.

vlovich1232• 12 days ago

I also agree that computer architecture is more important - it grounds your understanding of how to write efficient code regardless of platform since most machines today share very similar ideas (OOO execution, caches, NUMA etc).

How ever, I will disagree slightly that all the optimizations compilers do are about optimizing for a given architecture; some transformations are just weird algorithmic black magic about optimizing the underlying code itself. Knowing how to make sure the compiler sees through a given construct to give you the low level expression you want is too much art and randomness; we need better ways to express optimization expectations so that if the compiler fails to match expectations it becomes a loud compiler error.

xyzsparetimexyz6• 12 days ago

Thee optimization in question here was not obvious at all. It's a bs clang codegen quirk

maCDzP• 12 days ago

I did Nand2Tetris and then I understod why I need to vectorize. You get a view from nand-gate to software and get to see all the interfaces. Its a wonderful course.

jandrewrogers• 12 days ago

A large part of optimization is understanding the hardware architecture in detail and then making your software architecture mirror that hardware architecture as closely as possible. A compiler can't do this for you. Much of "performance engineering" is applying your understanding of how the hardware components work and are connected to the software design.

Most software introduces a large number of unnecessary stalls where one part of the hardware is waiting on or bottlenecked by a different piece of hardware. Optimization is often about removing or minimizing these stalls to the extent possible. But to do this you need to understand why specific code choices cause the hardware to stall in the first place. The most basic form of this is CPU cache locality but there are many levels.

Compilers are great at localized micro-optimizations, except for SIMD. Most idiomatic code can be detected and transformed by the compiler into something that takes advantage of the hardware design. You don't need to do tricky bit-twiddling stuff, the compiler is better at it than you are in most cases (this wasn't always true). I always recommend people interested in optimization experiment running snippets of code through the compiler using godbolt with optimization turned on. You can learn a lot about how the compiler sees and understands your code by studying this. It is also a good way to became familiar with assembly.

Leveraging SIMD and vector ISAs is a mess. Different hardware architectures rely on different idioms and compilers cannot auto-vectorize most code that can be vectorized. You need to learn the idioms of each vector ISA and how to write them using intrinsics. A great way to learn this is reading other peoples' SIMD code. Modern SIMD code is wide in that you need to memorize a lot of things but conceptually pretty shallow. It mostly comes down to learning the idiomatic tricks and gadgets for an ISA -- code is composed from these conceptual primitives.

jdw647• 12 days ago

These are concepts I've come across in books I've read, and your explanation is easy to understand. But I can tell my understanding is lacking in some parts because I haven't actually tried it in practice. As you said, I think using Godbolt to disassemble things myself is the right approach.

Thanks for the advice.

superjan• 12 days ago

In order of priority, I’d say:

1. If you write CRUD apps, make sure that the database does the heavy lifting.

2. Take note of algorithm complexities, use hashtables as appropriate, and write good hash functions when you start using hashtables/dictionaries.

3. Avoid pointer-heavy datastructures. In high level languages like Java, an object reference is a pointer dereference that can stall the CPU waiting for memory. This is sometimes optimized, but you can’t depend on it. The true zealots call this “data oriented design”.

4. If you write C/C++,rust, or the like, you might want to learn to read assembly. Godbolt.com is a fun way to learn. Note that not all instructions are equally fast: Long division and trigonomic functions are slower than integer adds, even when they are both a single instruction.

5. The next level is probably going for vectorized instructions: SIMD (ARM Neon, AVX). The most original applications can be found at lemire.me: a professor exploring optimizing things like JSON parsing using the latest processor features.

Sesse__• 12 days ago

I don't have a book, but my best recommendation would be: Learn how to measure. Whatever language you are using should have a profiler. Learn how to use it and look at how your code holds up. Look for surprises; those are optimization opportunities and learning opportunities. Make a change that you think has a reasonable shot at optimizing performance, look at how the profile and benchmarks change. If you're using a compiled language, look at what the generated code looks like. Is it what you'd expect, or did the compiler do something that blindsided you? Can you get rid of that overhead? Can you use a faster algorithm? Can you make things faster without making it an unmaintainable mess?

In general, you must expect to try ten things and then one of them will help you; fewer if your ideas are bad. :-) Occasionally, you learn new things (either about your machine or your language or your code base) and then you can try that elsewhere in the code (but don't go overboard, every technique has its limits).

DO NOT FALL PREY TO SUPERSTITION. Always measure in one way or the other. Don't do stuff blindly just because someone on the Internet told you (there's a _lot_ of bad performance advice out there).

karussell2• 12 days ago

First important step is the need to make your application faster.

Second step is to actually make it faster.

And I would say 99% of the time you can make applications faster without going down so deep on this low-level (which is still not easy) but sometimes you can only get faster via low-level optimizations.

jdw647• 12 days ago

thanks!

IshKebab2• 12 days ago

Wow that is quite surprising. Almost seems like it could be a compiler bug tbh. Very fragile optimisation if not!

adgjlsfhk13• 12 days ago

it's arguably more of a cpu bug than a computer bug. The problem is that predictable data determined whether a cmov or a branch is faster. cmov is only faster than if when the branch is unpredictable. Summer the compiler doesn't know what values your program will be called with, it can only pick and hope. To fix this, cpus could have an instruction like cmov but that learns whether speculation would be profitable and converts to treat it like a branch if better.

delamon• 12 days ago

You can, in fact, give compiler a hint wether the branch will be predictable or not. E.g. clang has __builtin_unpredictable().

astrobe_• 12 days ago

tbh, I'm not sure if we need "more clever" CPUs since they cause more and more unpredictable performance outcome -- and more pressing issues like Spectre. Besides, looking at the evolution of technology tells us that performance increases mainly comes from parallelism, like multicore and GPUs. Which doesn't apply to "annoyingly not parallel" problems of course. But besides "gambling" approaches like branch prediction, there's not much you can do about them in the general case.

IshKebab2• 12 days ago

There's no reason CPUs can't do prediction and speculative execution on conditional moves just like they do with conditional branches. But with conditional moves the cost of a misprediction is going to be much lower because you don't have to wait for an instruction fetch.

xlii2• 12 days ago

Is it only me..?

Quicksort is supposed to be an algorithm that has O(n) to O(n²) performance and O(n log n) being only an average performance case. Test was made on random data coming from different archs (so I doubt it's characteristic would be remotely identical).

Given input size of 50M it means that performance could be between 50M (5e7) up to 2.5e15. That's like performance instability of 8 orders of magnitude.

I'm not sure here if we can't write instead that "Your code is fast if you picked fast case for it" especially since fix of 6 OOM is smaller than algorithm's performance range.

sashank_1509• 12 days ago

I’m assuming he measured time by averaging on 100s of instances, or he maintained the exact same input for both versions of code. Would be a big oversight if not!

chrka3• 12 days ago

Both versions use the same input data. I also tried different random initial values and got essentially the same result. I didn't test hundreds of inputs, since that would have been mostly a waste of time in this case. The algorithm and the data distribution remain practically the same. What I'm measuring is the machine code that Clang generates for the hot loop.

gmm1990• 12 days ago

I hope the test data is the same when comparing the different runs. So the big o notation should be the same across different runs.

adgjlsfhk13• 12 days ago

if you have decent (randomized) pivoting, you never hit the worst case or anything like it

SkiFire132• 12 days ago

You don't need randomized pivoting for this, there are deterministic ones like median of median that will also result in a O(nlogn) worst case.

Also note that with a randomized pivoting you _might_ hit a O(n^2) worst case, it's just that it's incredibly rare and cannot be forced by an attacker controlling your input, so for most practical purposes can be ignored.

marginalia_nu4• 12 days ago

Looking at big-O isn't very informative. We have plenty of statistical tools for telling whether there is an effect even with noisy data.

xlii2• 12 days ago

I ran 10k test locally on 2e5 and I'm seeing 4 orders of magnitude instability, but very high local stability (i.e. runs within specific second are very stable, showing almost no deviation, runs couple second later are the same, but results are within 1 OoM of the prior results (smaller batches, 500 tests).

I'm not saying that optimization isn't valid, what I'm saying is that Quicksort shouldn't be optimized over randomized per run data set.

cogman104• 12 days ago

It's a good rule of thumb that can be quite useful without additional analysis. It's not always the right way to do performance tuning, but I can't count the number of times I've changed an O(n^3) to an O(n) and seen massive performance gains as a result.

chrka3• 12 days ago

You're talking about the complexity of the Quicksort algorithm, whereas the article is about code generation.

Both versions sort the same data using the same algorithm. Just a tiny change in the source code caused Clang to generate different machine code.

Using different seed values - (srand(1), srand(2), srand(time(NULL))) essentially leads to the same result. With a good choice of pivot, Quicksort is very close to O(n log n) in practice, so that’s not the key factor here.

The interesting thing is that the generated machine code changes significantly.

Sardtok• 12 days ago

Look at the source, Luke!

It's just rand() in test.c

Validark• 11 days ago

The Big-O for sorting algorithms is typically on the number of comparisons. The problem with that is that a comparison takes 1 cycle and you can do 4 per cycle in scalar code or much more in SIMD. Obviously comparisons of doubles are going to be a bit more expensive. Memory reads, on the other hand, take 4 cycles from L1, 10 from L2, and it goes up from there. If you have a predictable memory access pattern, the CPU can preload everything into cache for you, and if you have a deep enough pipeline you can cover up memory reads. Memory access is often completely ignored when analyzing sorting algorithms.

Heapsort has the minimum number of comparisons. Search up "An optimal algorithm for deleting the root of a heap" by Svante Carlsson. Notice how Heapsort is commonly used only as a fallback sort? That's because minimizing comparisons isn't the most important.

jimaway1232• 12 days ago

Does anyone know exactly what is going on here to cause this difference? I am extremely puzzled that the "beginner friendly" code is not at some point in the compilation pipeline in EXACTLY the same representation as the non-"beginner friendly" code. I would imagine they'd be in the same form very early on, perhaps even at the point of generating an initial syntax tree. And once they take on the same form in the compilation pipeline, the resulting compiler output should be identical. So what is really going on here?

141134• 12 days ago

Representing code in a compiler is not precisely trivial, and the two statements are actually quite different from a compiler or AST perspective. Just looking at the first branch:

  *lwr = x; lwr++;

This could be be represented with something like this (and this is a very vague approximation of an AST):

  block
    statement (assignment)
      expression
        operator (dereference)
          variable
      expression
        variable
    statement
      expression
        operator (post-increment)
          variable

The second form, that looks this in the source:

  *lwr++ = x

might look like this in the AST:

  block
    statement (assignment)
      expression
        operator (post-increment)
          operator (dereference)
            variable
      expression
        variable

If these two ASTs are to take the same form in the compilation pipeline, there needs to be some kind of pass that transforms one into the other.

As for why the second form is faster (or rather, why the compiler can generate faster code): There is likely an optimisation pass somewhere in llvm that recognises a pattern that this fits into, which allows it to generate branchless instructions. For instance, in the second form, there is a pattern of:

  operator (post-increment)
    operator (dereference)

that might be recognised by a pass. In the former form, the two operators are far apart in the tree, so a pass would have to "look further" to match them up. A single pass likely won't do this, either for (compiler) performance reasons, or for correctness reasons.

Finding which pass that is can be non-trivial, as it's more than a matter of enabling individual passes until one works. It might be that an earlier pass does some code reshaping that allows the relevant pass to work. My suggestion would be to dump the llvm ir at the end, and find the rough pattern that you're looking for, then re-run the compilation with `-mllvm -print-after-all` to see what the IR looks like after each pass, and then manually "look back" until you can't see the pattern any more.

jimaway1232• 12 days ago

  block
    statement (assignment)
      expression
        operator (post-increment)
          operator (dereference)
            variable
      expression
        variable

I don't think this could possibly be a valid AST for '*lwr++ = x' because the increment is not an operation on the dereferenced value, its an operation on the pointer. So in this case I don't see how it could help but be transformed into a form similar to the "beginner friendly" case.

Or perhaps I am wrong and it would generate an AST like you describe and rely on later passes to actually create a proper dependency graph. My mental model of how these kinds of postfix operators work always assumed it must very early on turn it into two separate statements. Thank you for the suggestions.

rokob• 12 days ago

Have you considered what happens in the presence of multiple threads? I know aarch is weakly memory ordered but the assembly output by those two versions is quite different when multiple threads are involved. There must be a reason spreading the mutations across multiple lines causes the compiler to pessimize to the branch version.

Edit: not saying they are different just that it is harder for the compiler to see the safety of the transform when they are quite different.

wat10000• 12 days ago

The compiler doesn’t have to consider how other threads will see this. If you don’t use atomics, the compiler is free to reorder things however it wishes as long as the single threaded behavior is the same in the end.

akoboldfrying• 12 days ago

Complete speculation, but it could be simply that clang only considers applying the branchless optimisation when the code inside the "if" block "looks like" a single instruction/statement -- and this "looks like" check might be done quite early, before "*x++ = y;" and "*x = y; x++;" are converted to the same thing.

Why restrict the optimisation unnecessarily like this? It's never the intention to artificially restrict optimisations, but they are difficult to test and debug as they can interact with other optimisations, and applying an optimisation too broadly leads to horrible correctness bugs. So if you cannot be certain that you understand all possible interactions now and in the future, it makes some sense to be conservative in applying them.

wiml2• 12 days ago

Yeah, that's an interesting question. I would expect the two to look the same once converted to SSA.

You can ask the compiler to dump its intermediate representations, so it might not be too hard to answer the question.

khuey2• 11 days ago

In the one statement case, clang emits LLVM IR where the last operation in each branch is the store. In the two statement case, clang emits LLVM IR where the last operation in each branch is the [pointer math](https://llvm.org/docs/GetElementPtr.html).

The SimplifyCFG pass in LLVM pulls the identical-in-all-but-one-operand store IR instructions into a successor block with a [select instruction](https://llvm.org/docs/LangRef.html#select-instruction) that will ultimately become a conditional move. But the pointer math differs by two operands (because there are two different pointers and one is being decremented and one is being incremented) so SimplifyCFG stops immediately and never reaches the stores.

This sort of brittle behavior where semantically identical code produces different optimization results is common in optimizing compilers unfortunately.

sylware3• 12 days ago

Don't forget to disable all "spectre and friends" mitigations in your linux kernel, and some workloads will become much faster.

Can you do the same on the windows kernel or apple kernels?

dyauspitr• 12 days ago

Interesting, did not know you could do that but it does make sense. Make sure it’s air gapped.

xyzsparetimexyz6• 12 days ago

https://wiki.archlinux.org/title/Improving_performance#Turn_...

Arch wiki says its okay.

xyzsparetimexyz6• 12 days ago

Only really on CPUs with the vulnerability

sylware3• 11 days ago

mitigations do slow all CPUs, even those without the vulns.

You need alternative linux kernels with their modules.

tredre32• 12 days ago

Yes you can do the same on Windows.

https://www.grc.com/inspectre.htm

sylware3• 11 days ago

You have to recompile the kernel to remove some mitigations about "indirect branching", namely you must have alternative windows kernels.

Do you have those windows kernels?

sigbottle• 12 days ago

Auto-vectorization is not a programming model!

jagged-chisel• 12 days ago

    else *rwr-- = x;

No. Make that obvious and the PR can pass. Argue, and you're off the project.

xyzsparetimexyz6• 12 days ago

Fine. I'll use inline asm then.

CodeMage• 12 days ago

I sometimes wonder whether that's a better idea for these micro-optimizations, rather than looking at the assembly code and trying to coax the compiler into generating what you want.

That said, I'm not keen on "argue and you're off the project" work environment.

bigstrat2003• 12 days ago

That would unironically be easier to understand than doing *rwr-- = x.

wiml2• 12 days ago

*foo++ (and --) is an extremely common C idiom. I'd argue it's clearer than the separated version.

141134• 12 days ago

As I experienced while trying to write out an AST for this pattern, the operator precedence makes it harder to read. I would at least prefer that it's written as *(foo++).

tom_• 12 days ago

    else{
        *rwr--=x;
    }

xyzsparetimexyz6• 12 days ago

What if you wrote this in a branchless way?

bool v = BLQS_CMP(x, piv);

int* ptr = v ? lwr : rwr;

*ptr = x;

ptr += int(v) * 2 - 1;

tredre32• 12 days ago

The ternary isn't guaranteed to be branchless. In your case it should almost always be on a modern compiler, but it really shouldn't be present in what one would call branchless code.

misev• 12 days ago

You could rewrite it to

    int* ptr = ((v != 0) * lwr) + ((v == 0) * rwr);

I doubt it would be faster than the ternary though.

xyzsparetimexyz6• 12 days ago

Does any standard library have a glsl-like select(condition, true_val, false_val) that is more or less guaranteed to correspond to a csel?

shevy-java• 12 days ago

My code is not fast. Writing efficient code takes a lot of brain power. My brain is of the lazy type - it wants the computer (but not AI) to solve things. I only write code so I can be lazier lateron.

I think with this approach, we will only win if a language allows for:

1) ease of writing, and 2) fastness

Right now languages don't really combine both. We have ease of writing e. g. ruby or python, but they are slower than C, our godfather language. So far all languages that try to solve both problems, become mega-verbose and tend to gravitate more towards one than the other - usually e. g. "let's write a replacement for C". I wonder if combining both 1) and 2) is possible, kind of like select on your own what to combine, so if my time is precious, I write a quick prototype. If this must become faster, I write it with more details. That's still not really a language that combines 1) and 2) genuinely but perhaps it is an acceptable trade-off. Right now we kind of mix two languages here, say, ruby+java or python+C or any other similar combination.

karussell2• 12 days ago

Coincidentally a few days ago, also for a sorting algorithm, I stumbled over a situation where replacing the branchless cmov with branching instructions actually made the code 30% faster: https://github.com/graphhopper/graphhopper/pull/3380 Or at least the AI explained it this way to me, but I'm unsure if this is correct.

fsmv• 12 days ago

But it's not exactly a cosmetic change. x++ is semantically different from x; x++; I wonder if clang would make it branchless if you instead write

  if (BLQS_CMP(x, piv)) { *lwr = x; ++lwr; }
    else { *rwr = x; --rwr; }

The difference is post-increment has strange semantics. While the compiler should be able to understand that the value wasn't used and post increment and pre increment are the same I wouldn't be surprised if it tracks that it was post increment and misses some optimizations because it's trying to garuntee post increment semantics.

Although it's true compilers can be very sensitive to exact phrasing triggering specific optimization passes. So it still might not give the branchless version by changing it to pre increment (which is the same as a normal +=1).

The only way to really know is to dig into what optimization passes clang took in both cases and analyze the difference.

pixelesque• 12 days ago

Note: in some cases (not completely sure if it applies in this situation as I haven't looked into the details of this code enough), using pre-increment AND when the value is used is a dependency on that operation, so "+= 1" is actually faster for out of order execution.

So yeah, as you say, compilers can be very sensitive to this, and in the past (although more than 10 years ago now, so might not be relevant now), ICC often used to generate better (faster executing) code when using "val += 1" vs "++val".

chnoblouch• 12 days ago

Looking at the optimization pipeline in Compiler Explorer (https://godbolt.org/z/rd3qber3b) after the code is converted to SSA (SROAPass), it seems that the fast version computes the pointer increment (`lwr++`) before storing `x` (`*lwr = x`), whereas the slow version does it the other way around. This happens because Clang chooses a different instruction order when generating a single statement compared to two statements.

The computation is the same, but apparently, this tiny change prevents SimplifyCFGPass from turning the code into the branchless version later on. I'm not sure why this happens, perhaps because it messes something up in the pattern recognition of the pass?

Your code is fast if you're lucky

Related Stories