I agree, and I don't even know math.
Most of things that are impenetrable in programming aren't about... programming. They are about some actually complex field like math being applied to programming.
For example, a library that does stuff with geometry. You need to know geometry to understand the program, but the program itself will never be complicated. It's the geometry that is complicated.
In cryptography, it's not the program that is complicated, it's the field of cryptography. In AI, it's statistics.
In graphics programming, math is the most impenetrable part, not programming anything. You can be a very good programmer in the sense that you know how to architect information systems and still fail to write a shader because shader programming requires you to know what a "dot" product is and you haven't heard about that since high school.