Yes the performance boost in training is critical.
So what I'm saying is this performance boost is thanks to the switch to non-binary neuron (which allows a reversible operation to transmit a magnitude) - that's MOST important.
And separately ReLu are just better at this, because they are linear they don't have "vanishing edges" (which prevents the vanish/explode)
Separately I'm glad you brought up CNNs because CNN's are old, and go back to rosenblatt (1958), his perception had a first layer of local connections in it based on the findings in biological systems.
and of course that's because nature has found it's more efficient, and so the efficiency is huge.
but the point is CNN = fewer knobs to train.
and there are lots of simple ways to help reduce knobs
- fix some connections
- drop some connections
These are great points. You'll notice that video is actually a "bonus" step which isn't required to complete the final exercise in that lesson. We are still experimenting with how to go about "bonus" steps. In other lessons you'll see they are done in a multi-step article style. Such as in Animation:
https://www.khanacademy.org/partner-content/pixar/animate/pa...
We also do this for the bonus step in Character Modeling (my favorite lesson!)
Our plan is to continue collecting more feedback on these different styles and find out what works best.
The initial idea for this content came from Pixar (Tony DeRose had given talks in the past showing connections between math and the movies - see here a video he did with numberphile a while back: https://www.youtube.com/watch?v=2NzTAaYgk4Q). The goal was to turn that sort of thing into a series of interactive lessons.
Hey HN! My name is Brit Cruise and I'm the lead content developer on the Khan Academy side of this equation (working alongside Tony DeRose from Pixar...plus a village of others), if you have any specific questions about this project I'll be happy to answer as best I can.
Hey there, I created this video. It was just one part of a longer lesson I was trying on "Random Algorithms" because it involved Fermat's primality test...so in this case we really needed that formula to try it out which is why I added it at the end.