49pctber

Anyone who would like to run an LLM would need to perform their computations on hardware. So picking hardware that is good at matrix multiplication is important for them, even if they didn't develop their LLM from scratch. Knowing the basic math also explains some of the rush to purchase GPUs and TPUs on recent years.

All that is kind of missing the point though. I think people being curious and sharpening their mental models of technology is generally a good thing. If you didn't know an LLM was a bunch of linear algebra, you might have some distorted views of what it can or can't accomplish.

Yeah, that's a better way of thinking about it. That way you don't have any phantom bookkeeping like the way I was taught. The units are still right there.

The idea that helped make decibels click for me is that they're a way to quickly do both dimensional analysis and gain/attenuation calculations at the same time.

"Plain" decibels are simply (power) ratios. These can describe multiplicative changes in power. These are positive for gains (like in a power amplifier) or negative for attenuations (like path loss). They are unitless quantities.

Decibels add. A ten 10 dB gain (x10) followed by a 20 dB loss (x0.01) is -10 dB (x0.1).

"Flavored" decibels are in reference to some power quantity. For example, dBm uses one milliwatt as its reference. So 2 mW / 1 mW = 2 = 10^(3/10) = 3 dBm. These quantities have associated units, but they're still technically dimensionless.

Here's the key insight. You can only have one "flavored" decibel value per computation. Say you have some 3 dBm signal (2 mW). You can add as many regular decibel values as you want, but the unit is still dBm. 3dBm + 4 dB - 7 dB = 0 dBm. In linear units, 2 mW * 2.5 * 0.2 = 1 mW

If you were to do something like 3 dBm + 0 dBm, the linear units would be 2 mW * 1 mW = 2 mW^2, which is probably not what you want.

dBs are confusing. Different fields have slightly different conventions. People talk about any factor of 2 as a 3 dB change, when technically it should only be relative to power-like quantities. It's weird that some of these "units" can be added together, while others can't. The factors of 10 and 20 can be confusing.

But if you consider the units from a dimensional analysis standpoint, decibels are much more sane and intuitive than they appear.

25 and 50 contribute two 5 factors each. The multiplicity of 2 factors is even greater, as every other even factor contributes at least two, every fourth at least three, etc.

I absolutely love this. Brilliant!