What Every Developer Should Know About GPU Computing (2023)

https://blog.codingconfessions.com/p/gpu-computing

by mooreds • 2 years ago

179 44 2 years ago

blog.codingconfessions.com

cgreerrun3• 2 years ago

This video is a great explainer too:

How do Graphics Cards Work? Exploring GPU Architecture (https://youtu.be/h9Z4oGN89MU?si=EPPO0kny-gN0zLeC)

penr0se• 2 years ago

Wow, this is one of the best videos I've ever watched. Thanks for sharing

Hunpeter• 2 years ago

I've been binging Branch Education the last week or so, and I concur that the videos are exceptionally well made. Some commenters noticed one or two mistakes in some of them, but nothing major.

cgreerrun3• 2 years ago

You're welcome! I highly recommend the CPU/microchip one too.

ferbivore• 2 years ago

These videos always give me an unpleasant feeling that I don't know how to express in a useful way. Almost everything in this one is oversimplified, misleading, or wrong, yet I feel like any attempt to argue for this would come off as pedantic; any individual complaint could be countered by either "it's technically true" or "it's just entertainment".

Like, the "cowboy hat" example is wrong on multiple levels - GPUs are not SIMD machines, model-to-world translation doesn't work like that in practice - but you can maybe excuse it as a useful educational lie, and he does kind-of explain SIMT later, so is objecting to it valid? Or: the video claims to be about "GPUs", but is in fact exclusively about Nvidia's architecture and the GA102 in particular; is this a valid complaint, or is the lie excusable because it's just a YouTube video? Or: it overemphasizes the memory chips because of who's sponsoring it; does this compromise the message? Or: it plays fast-and-loose with die shots and floorplans; is a viewer expected to understand that it's impossible to tell where the FMA units really are? Or: it spends a lot of time on relatively unimportant topics while neglecting things like instruction dispatch, registers, dedicated graphics hardware, etc.; but is it really fair to complain, considering the target audience doesn't seem to be programmers? And so on.

Did you actually get anything out of this video? Any new knowledge? The article seems like a much more useful intro, even if it is also specific to Nvidia and CUDA.

cruffle_duffle3• 2 years ago

Honestly, I think Branch Education does a solid job with topics like this. Yes, it glosses over details and sometimes simplifies to the point of oversights, but that’s often necessary in educational content to avoid getting bogged down. It’s a balance: if you dive too deep, you risk losing the main points.

Branch Education is designed to introduce complex concepts, often for high schoolers or newcomers to the subject. Even my first grader finds it interesting because it’s visually engaging and conveys a general understanding, even if much of the terminology goes over their head. Their video on how computer chips are made, for example, managed to hold the whole family’s attention. That is hard to do for most of the nerdy shit I watch on YouTube!

It’s not meant to be a deep dive—Ben Eater is better suited for that. His work on instruction counters, registers, and the intricacies of “how CPUs work” is incredible, but it’s for a different audience. You need a fair amount of computer science and electrical engineering knowledge to get the most out of his content. Good luck getting my family to watch him breadboard an entire graphics system; it’s fascinating but requires a serious commitment to appreciate fully.

cgreerrun3• 2 years ago

The only time I really feel like I understand something is by building something with it. So actually writing a CUDA kernel to do grayscale conversion and then tweaking the code.

BUT... both the video and the article are useful before you do that. They both allow you to build a mental model of how GPUs work that you can test later on.

ChrisArchitect• 2 years ago

(2023)

Some discussion then: https://news.ycombinator.com/item?id=37967126

winwang2• 2 years ago

Makes me consider writing a post on misconceptions of GPU computing, such as requiring the problem to be fully data-parallel.

etdeagle2• 2 years ago

Please do! I would love to read about it. I have been playing with GPU hash tables shared between all the threads using locking (Interlocked.CompareExchange) and such in a Compute Shader. I have been wondering if there are better ways than locking.

jms554• 2 years ago

Hash tables on GPUs are cool. You can use them for meshless radiance caches, that automatically (for better or for worse) adapt to the surrounding geometry.

winwang2• 2 years ago

Until I get around to it, there's this blog post I wrote: https://wiwa.substack.com/p/can-we-10x-rust-hashmap-throughp...

Not an "expert" on GPU hash tables, but I'm mildly aware of other hashmap things like a novel(?) hash-indexed sorted array (HISA): https://arxiv.org/html/2311.02206v3

jms554• 2 years ago

In my opinion, the biggest misconception around GPUs I see people have is that they don't realize it's an entirely separate device with it's own memory, compiler, scheduler, etc.

You don't call functions to tell the GPU what to do - you record commands to a buffer, that the GPU later executes at some indeterminate point. When you call dispatch/draw(), nothing actually happens yet.

Another kind of misconception: data transfer is a _really_ overlooked issue. People think "oh this is a parallel problem, I can have the GPU do it" and completely discount the cost to send the data to the GPU, and then get it back. If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU (which again, is a completely separate device unless you're using an iGPU), and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time.

physicsguy• 2 years ago

I found that, I recently worked on a project where there was a Python library using pymc3, scikit-learn and pytensor that is used for time sensitive calculations. The performance wasn't very good and someone was pushing for us to spend a lot of time moving it to GPU, but as I had to point out, that means almost a total rewrite, not simply switching to the GPU enabled version(s) of those libraries, where they even exist.

wiz21c• 2 years ago

Another one I had when starting was to underestimate the power of the GPU. I really needed to increase the size of my toy problems to actually see the benefits of using a GPU.

For data transfers, my experience is that you rather quickly hit that bottleneck, and it's a tough one. And it's not proportional to the number of transferred bits: transferring one byte naively can be extremely costly (like half the performance, I'm not joking)

ElFitz2• 2 years ago

Another kind of misconception: data transfer is a _really_ overlooked issue. […] If you want to write 20mb of data to a buffer, that's not just a memcpy, all that data has to go over the PCIe buss to the GPU […], and that's going to be expensive (in real time contexts). Similarly if you want to read a whole large buffer of results back from the GPU, that's going to take some time.

Does having a unified memory, like Apple’s M-series chips, help with that?

trq01758• 2 years ago

And if one is using iGPU, one might think I'll have a great bandwidth, but reality is that DDR memory for CPU is optimized for low latency not bandwidth and they'll probably have a 64 bit channel (or 2x32 bits) from a single DDR module or 128 bit in dual channel configuration, while something like RTX 4090 will have onboard graphics-DDR GDDR memory on 384 bit channel very much optimized for bandwidth and not latency pushing according to specs a terabyte per second. Apple really needed their memory architecture - having a high memory bandwidth for onboard GPU to have reasonable performance.

amelius3• 2 years ago

What is holding manufacturers back to create a bus for the GPU that is as fast as main memory?

zokier• 2 years ago

While I do understand conceptually that GPU is basically its own computer, I struggle to understand how this works in terms of operating systems and multitasking. Fundamentally managing resources between tasks is one of the core functions of operating systems, and stuff like CPU schedulers and virtual memory are fairly well understood. But how are the resources on GPUs managed? If I have n processes doing GPU compute (/graphics), how are the limited GPU time and memory allocated between them? Can you set priorities and limits like you can with other resources?

I feel these sort of "operational" questions are often neglected in the discussions, but considering how GPU is increasingly being used in wide range of applications (both for graphics and compute) I think it's becoming relevant to think how they play together.

cruffle_duffle3• 2 years ago

When will “they” invent a better bus to go between the system memory used by the CPU and the GPU? It seems like that is a pretty major bottleneck.

It took a while before SSD’s stopped using SATA despite SATA being a huge bottleneck. Now it uses something else whose name I cannot recall. Surely there is work to do something like that for the GPU.

Because while it’s been a long while since I’ve built a computer I do know that the video card has always been the peripheral that pushed the limits of whatever interconnect bus existed at the time. There was all kinds of hacks for it like AGP, Video Local Bus, and then even PCI express to a large degree.

forgotpasagain• 2 years ago

Yes! I always thought shared memory is bigger and registers+l1s are per core instead of per SM.

Miniminix• 2 years ago

If you are not writing the GPU kernel, just use a high level language which wraps up the CUDA, Metal, or whatever.

https://julialang.org https://juliagpu.org

schmidtleonard• 2 years ago

The big problem I've had historically with non-native CUDA wrappers is that they always seem to omit or bug some feature that is critical for my application, and the amount of debugging pain and implementation or bugfix work to get around this problem exceeds the effort "savings" of a high level interface by an order of magnitude or three.

esperent• 2 years ago

Unrelated but I absolutely love this reply from the previous time this was posted and someone complained about the line "most programmmers ...":

Try this on: "A non-trivial number of Computer Scientists, Computer Engineers, Electrical Engineers, and hobbyists have ..."
Took some philosophy courses for fun in college. I developed a reading skill there that lets me forgive certain statements by improving them instead of dismissing them. My brain now automatically translates over-generalizations and even outright falsehoods into rationally-nearby true statements. As the argument unfolds, those ideas are reconfigured until the entire piece can be evaluated as logically coherent.
The upshot is that any time I read a crappy article, I'm left with a new batch of true and false premises or claims about topics I'm interested in. And thus my mental world expands.

https://news.ycombinator.com/item?id=37969305

ImHereToVote• 2 years ago

That's beautiful

dmichulke• 2 years ago

I know this process as steelmanning.

https://umbrex.com/resources/tools-for-thinking/what-is-stee...

If you see people doing this, befriend them because it means they're valuing knowledge higher than their ego.

aquariusDue• 2 years ago

Kinda reminds me of Rogerian rhetoric: https://en.wikipedia.org/wiki/Rogerian_argument

I suppose it always depends on the goals of the conversation and the participants' greater view of the world. Your advice also seems similar to the sage wisdom along the lines of "It's the mark of an educated mind to entertain a thought without accepting it" but goes one step further.

I generally agree with you, it truly is rare to find people who can put aside their ego for the pursuit of a higher (or common) goal.

amelius3• 2 years ago

Seems in-line with this HN guideline:

Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

BillLucky• 2 years ago

great

amelius3• 2 years ago

Title reminds of:

https://people.freebsd.org/~lstewart/articles/cpumemory.pdf

hermitcrab3• 2 years ago

GPU are optimized for number crunching. Do they get used at all for string processing? I ask because I develop data wrangling software and most of it is string processing (joins, concatenations, aggregations, filtering etc), rather than numerical.

TheDudeMan• 2 years ago

Do you have millions of strings that need to be manipulated in the same way at the same time?

hermitcrab3• 2 years ago

Yes. For example, you might want to change a column of 10 million strings from upper case to lower case. Or concatenate 2 columns to create a third column. It is not clear to me this would be any faster on a GPU.

hermitcrab3• 2 years ago

Also, you might want to create a hash table from a million values in a column, so you can use this for a join.

What Every Developer Should Know About GPU Computing (2023)

Related Stories