marshallward

I hope you can list all of the groups of order < 8 without having to think too hard.

Welp, guess I'm out.

Here is more recent video that beats those records: https://www.youtube.com/watch?v=jPyf5kIGoKA

Except! Even this is now out of date, and I think BlueScuti has broken some of these records! The Tetris leaderboards are very active at the moment.

“The pyramids seem like pretty monumental work”

You don't say...

I ran with OMP_NUM_THREADS=1, but your point is well taken.

As for the original post, I felt a bit embarrassed about my original comments, but I think the compilers actually did fairly well based on what they were given, which I think is what you are saying in your first part.

I'm sure there's more to it, but just comparing the profile output shows aggressive use of prefetch and broadcast instructions.

I just did a test of OpenBLAS with Intel-compiled BLAS, and it was about 6 GFLOP/s vs 150 GFLOP/s, so I must admit that I was wrong here. Maybe in some sense 4% is not bad, but it's certainly not good. My faith in current compilers has certainly been shattered quite a bit today.

Anyway, I have come to eat crow. Thank you for your insight and helping me to get a much better perspective on this problem. I mostly work with scalar and vector updates, and do not work with matrices very often.

You are right, I just tested this out and my speed from BLAS to OpenBLAS went from 6 GFLOP/s to 150 GFLOP/s. I can only imagine what BLIS and MKL would give. I apologize for my ignorance. Apparently my faith in the compilers was wildly misplaced.

I don't disagree, but where are those techniques presented in the article? It seems like she exploits the particular shape of her matrix to align better with cache. No BLAS library is going to figure that out.

I am not trying to say that a simple 50+ year old matrix solver is somehow competitive with existing BLAS libraries. But I disagreed with its portrayal in the article, which associated the block with NumPy performance. Give that to a 2024 Fortran compiler, and it's going to get enough right to produce reasonable bytecode.

How do you feel about Nvidia endorsing do concurrent migration to GPUs? Would that be classified as parallelization?

Fair enough, this is not meant to be some endorsement of the standard Fortran BLAS implementations over the optimized versions cited above. Only that the mainstream compilers cited above appear capable of applying these optimizations to the standard BLAS Fortran without any additional effort.

I am basing these comments on quick inspection of the assembly output. Timings would be equally interesting to compare at each stage, but I'm only willing to go so far for a Hacker News comment. So all I will say is perhaps let's keep an open mind about the capability of simple Fortran code.

There is an implication here that the Fortran implementation of `SGEMM` is somehow inadequate. But any modern Fortran compiler will quite easily apply the AVX and FMA optimizations presented here without any additional changes. Both GNU and Intel make these substitutions with the correct flags.

The unrolling optimization is also just another flag away (`-funroll-all-loops`). The Intel Compiler will even do this without prompting. In fact, it appears to only do a modest 2x unroll on my machine, suggesting that the extreme unroll in this article would have been overkill.

Parallelization certainly a lot to ask of Fortran 77 source, but there there is little stopping you from adding OpenMP statements to the `SGEMM` function. In fact, modern Fortran even offers its own parallelization constructs if you're willing to go there.

Which is to say: Let's not belittle this old Fortran 77 function. Yes it is old, and does not even resemble modern Fortran. But the whole point of Fortran is to free the developer from these platform-specific details, and hand the job off to the compiler. If you don't like that approach, then you're welcome to go to C or C++. But this little block of Fortran code is already capable of doing just about everything in this article.

Unfortunately I don't think the compilers will add it in time.

I have only looked at the GCC Fortran frontend. I would not say it is necessarily difficult but rather very ad-hoc and disconnected from the rest of gcc. I don't see many tools for specifying a grammar or tokens. Instead, there are a lot of constructs custom-written to handle Fortran. Looking around, the Go and Rust frontends feel equally disconnected.

I'm told that you eventually make your way to the "GCC IR", but it seems like you are largely on your own with how to get there.

I did not invest a lot of time here, so I could be wildly off base, but thought I'd at least try to give a very-slightly-hands-on perspective.

I know someone who took that course. They did not have fond memories of it.

My impression is that it can be a very frustrating way to learn mechanics if you don't have much interest in functional programming.

It's an excellent essay, and the Fortran community owes you a major gratitude for promoting these issues. But surely there loops can be safely parallelized if the iterations do not interact, e.g. per-element array arithmetic, and a compiler ought to safely identify such arithmetic.

Also, isn't your employer promoting do concurrent as a method of GPU parallelization? Has this been controversial within Nvidia?

Very true, and it opens the door to new optimizations which are not possible when going through a library (at least in the case of MPI).

I agree that it is not a parallel construct. But it can nonetheless be parallelized if certain criteria are met.

Do concurrent loops can also be parallelized.

I remember going to jai alai games in Miami as a kid. I was never very into sports, but it was fun to watch.

As far as I could tell, the adults were there to bet on the games.

What is it about?

I have not yet seen anything like this, but I have observed the Intel compiler produce different bytecode on Intel and AMD machines.

In one case, the compiler would use masking vector instructions on Intel but not on AMD. It was also the correct decision: Those instructions have very low latency on Intel, but are many times slower on AMD. On each platform, it produced the best bytecode that it could.

We work hard to retain floating point reproducibility in climate models. I have a presentation on this, if anyone is interested.

https://www.marshallward.org/fortrancon2021/#/title-slide

https://m.youtube.com/watch?v=wQQuFXm6ZqU&t=27s

True but every kind is considered a type.

Fortran is no longer in ALL CAPS.

Fortran supports both of these, with `mod` as the C-like truncated modulo and `modulo` as the floored modulo. Having both is convenient, but you do get errors from people who don't realize the difference

I think the statement has different meanings to different people, but one "unsolved" problem is the ability to model the dynamics of small scales (the "turbulence") in terms of the large-scale "observed" flow.

The question has very strong analogies to thermodynamics. For example, one can average over microscopic motion to yield something like a diffusion equation, where transport of the (microscopically averaged) density is pushed from high to low concentrations, and all of the microscopic details get wrapped into a single number, the diffusion coefficient.

In fact, averaging over molecular dynamics works in so many contexts that the details end up not being terribly important. You will always end up with something like a diffusion equation.

It's very reasonable to think this ought to work more generally, averaging over the turbulence to produce a similar expression for the dynamics of the large-scale. But if you try to apply similar averaging techniques to the Navier-Stokes equations, the averages never end, no clear solution emerges, and the only hope to terminate the exercise is to insert some kind of "closure approximation".

Some consider a robust theory for such a closure approximation, or any method to resolve the impact of the turbulent flow on large-scale flow, to be an unsolved problem of turbulence.

These questions have been researched for many decades, and a great deal has certainly been learned, but a rigorous closure theory has been elusive. Meanwhile, the computers keep getting bigger and faster, to the point where the turbulence can in many cases be modeled reasonably well. And as the questions around turbulence become relegated to smaller and smaller scales, one starts to wonder if this is a problem that will even need to be solved in the future.

We use biharmonic diffusion operators in oceanography as a basic turbulence model, which is effectively a fourth order derivative in space. It is like an accelerated diffusion which aggressively dissipates small scales but preserves the large physical scales more robustly than normal diffusion.

I had no idea this was a thing, but I did notice Chris Lattner (of LLVM) is a woodworker.

https://nondot.org/sabre/Woodworking.html

Same for me, it was laughably bad.

I find autoconf to be good at its job - building a standalone script which runs micro tests - and still use it, but that automake and libtool add very little value for the reasons you described.

I do agree that m4 is rather unpleasant to write, and that the resulting tests are too slow (often because user defined macros do a poor job of cacheing) but I also think that these things could be addressed (newer macro languages, bundled or threaded micro-tests) while still preserving the good bits of autoconf.