charles_irl

Very cool!

Unless you've actually watched tokens stream at those rates, the numbers are hard to internalize. This is the rendering.

I built something similar recently, for the same reason: https://modal.com/llm-almanac/token-timing-simulator.

I like that the output rendering is closer to typical UIs -- syntax highlighting in code mode, tool calls, dim-italic reasoning.

One feature mine has that the author, or anyone else who vibe codes their own version after seeing this, might like to steal is modeling the distribution of output latencies. My implementation is hacky (log-normal roughyl estimated from p50, p90, and p99 values), but still, when you set those to realistic values, it recreates the "jitter" you see in many LLM UIs.

antirez is right that generation tok/s isn't flat as a function of context length, which is a weakness of both simulators.

To clarify: we do content-based hashing, and when we say "shared bytes aren’t guaranteed to be in the exact same container image layer", what we mean is that

FROM some/image RUN pip install torch==2.7.1

and

FROM another/image RUN pip install torch==2.7.1

will produce images with very high overlap in contents, which will be shared by a content-based cache, but those images' final layers are disjoint from the perspective of a layerwise cache.

You're absolutely right!

Yep! That should start in ten seconds or so -- about a second per gigabyte of weights, plus a second to start the container and a few seconds to load the memory snapshot.

There are a few limitations with snapshotting, e.g. it generally fails when using multiple GPUs, which we document here: https://modal.com/docs/guide/memory-snapshots.

Cutting latencies by 40x! Unfortunately couldn't fit the whole title in the character limit :<

Ed Zitron also called out the business model of GPU-as-a-service middleman companies like modal deeply unsustainable, and I also don't see how they can make a profit if they are only reselling public clouds.

You got a link for that? I work on Modal and would be interested in seeing the argument!

We think building a proper software layer for multitenant demand aggregation on top of the public clouds is sufficient value-add to be a sustainable business (cf DBRX and Snowflake).

Sorry to lead with a bunch of jargon! Wanted to make it obvious that we'd give concrete recommendations instead of palaver.

The technical terms there are later explained and diagrammed, and the recommendations derived from something close to first principles (e.g. roofline analysis).

oof ty, willfix

Inspired by https://float.exposed, which was on the front page recently, I put together this visualizer for lower precision/quantized floating point numbers -- specifically, all of the formats in the Open Compute Project's Microscaling Formats Spec (https://www.opencompute.org/documents/ocp-microscaling-forma...).

Thanks! I think computers are fun and I want reading about them to be fun too.

I was also reminded of HazyResearch's MegaKernels. Didn't want to distract from the main thrust of the post, but definitely think that's a promising approach.

Hey, one of the authors here!

Reductively, software engineering means taking an idea and mapping it into code. So one form of "reverse" engineering would be taking the code and extracting the ideas. That's what we did here.

Because the source is public, there's quite a lot to work with from the start -- the warp specializations are named and there are helpful comments in many places.

But for many components, we didn't have much. Maybe the clearest case of "reverse engineering" explained in the post is with the cubic approximation for the rational part of the exponentiation. That required staring at some inline assembly and doing math.

Cool paper! The authors use the fact that the M1 chip supports both ARM's weaker memory consistency model and x86's total order to investigate the performance hit from using the latter, ceteris paribus.

They see an average of 10% degradation on SPEC and show some synthetic benchmarks with a 2x hit.

Author here! Worked on this project generating QR codes with diffusion models and ensuring they actually scan.

Ended up having to manually scan a few thousand generated codes by hand in order to build up a dataset to evaluate fully automated systems against.

We landed on QReader. It was such a good match for a human with an iPhone that we were able to use it to scale up inference -- basically, generate eight codes, each of which has a ~30% chance of scanning, then show the user any that did scan. That gives you a (1-0.3)^8 ~= 5% chance of failure.

That's a fair point! I was thinking of the Yamaha chips in the Sega consoles mentioned in that comment -- which certainly defined the sound of the 1990s for me as a child. But my small town Midwestern up-bringing was behind the curve!

Will replace with the lore-accurate "late 1900s".

We looked into this at Modal! We put out vGPUs but didn't see demand and our internal benchmarks for MPS and Green Contexts didn't indicate a big win.

The tricky thing here is that many GPU workloads saturate at least one of the resources on the GPU -- arithmetic throughput, memory bandwidth, thread slots, registers -- and so there's typically resource contention that leads to lowered throughput/increased latency for all parties.

And in a cloud (esp serverless/auto-scaling) computing context, the variety of GPU SKUs means you can often more easily right-size your workload onto whole replicas (on our platform, from one T4 up to 8 H100s per replica).

When I'm feeling sassy, I like to tell people that Modal is "Enterprise Java Beans for AI".

We've talked to them and there's some impressive technology there!

Hmm, hard to say!

In the few months since I originally wrote this, I've come to an even greater appreciation of just how hard it is to maximize utilization of the Tensor Cores. It's a lot more than just kernel parameter tuning and using a few parallel programming tricks (parallel reduce, unrolling). It really borks your CUDA code -- you need warp specialization, you need to break warp uniformity, you need to work with explicit asynchrony. Hoping to write about this for the Modal blog/GPU Glossary soon!

I also spent a bit of time working with ncu/"NSight Compute". I'd probably include a bit about it in the section on how to improve your MFU if I rewrote the article today. But tl;dr use the profiling tool, Luke! And a good way to learn is to watch NVIDIA's GTC talks.

That said, I've also noticed even more cases where GPU kernel utilization is well below target. I think (and Horace He has argued) that that comes in part from optimized GEMMs running so fast on Tensor Cores that host overhead becomes the bottleneck (classic Amdahl). This unfortunately means more host logic needs to be compiled -- either graph-compiled as in torch.compile or moved into a compiled language.

Nice article! I had to restrain myself from ranting on our blog :)

Oh, I wrote this! Thanks for sharing it.

I worked on this project a bit! These are great questions.

Fine-tuning works very well for style (as opposed to factual knowledge), which is all we're trying to achieve here -- as another commenter put it, it's a "comedy bit" for a company Slack.

In fact, fine-tuning for style works well enough that we find it pretty easy to just re-train when new models come out. There's sometimes some YAML-fiddling required to get training frameworks to work with different model series (e.g. a DeepSeek series model vs a LLaMA series model), but it's not too onerous. IMO, the ideal ML pipeline looks less like the bespoke process common these days and more like a materialized view (shouts to pgML). That's easy to automate and so reduces the gap with prompting.

On the other hand, it seemed harder to craft a generic system prompt and a generic retrieval system that would work across organizations to define user communication style.

Thanks! We think that just because something is deeply technical doesn't mean it has to be ugly.

Coming from the neural network world, rather than the shader world, but: I'd say you're absolutely right!

Right now NNs and their workloads are changing quickly enough that people tend to prefer runtime optimization (like the dynamic/JIT compilation provided by Torch's compiler), but when you're confident you understand the workload and have the know-how, you can do static compilation (e.g. with ONNX, TensorRT).

I work on a serverless infrastructure product that gets used for NN inference on GPUs, so we're very interested in ways to amortize as much of that compilation and configuration work as possible. Maybe someday we'll even have something like what Redshift has in their query engine -- pre-compiled binaries cached across users.

Thanks for the kind words! I still feel like one of those newcomers myself :)

Now that so many more people are running workloads, including critical ones, on GPUs, it feels much more important that a base level of knowledge and intuition is broadly disseminated -- kinda like how most engineers basically grok database index management, even if they couldn't write a high-performance B+ tree from scratch. Hope this document helps that along!

100% -- there's basically no substitue for benchmarking! I find the empiricism kind of comforting, coming from a research science background.

IIUC, even CuBLAS basically just uses a bunch of heuristics that are mostly derived from benchmarking to decide with kernels to use.

Yeah, there's a tension between showing enough information to be useful for driving decisions and hiding enough information.

For example, "compute capability" sounds like it'd be what you need, but it's actually more of a software versioning index :(

Was thinking of splitting the difference by collecting up the quoted arithmetic (FLOP/s) and memory bandwidths from the manufacturer datasheets. But there's caveats there too, e.g. the dreaded "With sparsity" asterisk on the Tensor Core FLOP/s of recent generations.

A few seconds is way longer than we intended! When I click around all pages after the first load in milliseconds.

Do you have any script blockers, browser cache settings, or extensions that might mess with navigation?

would also like to see a PDF that has all the text in one place, presented linearly

Yeah, good idea! I think a PDF with links so that it's still easy to cross-reference terms would get the best of both worlds.

Good callout! I'll work on the captions.

Custom! Took inspiration from lynx, lotus, and other classic terminal programs.

Thanks!

intels SMT does do something very similar (2 hw threads)

Yeah that's a good point. One thing I learned from looking at both hardware stacks more closely was that they aren't as different as they seem at first -- lots of the same ideas or techniques get are used, but in different ways.