Ask HN: Those learning about neural networks, what do you find most difficult?

About a decade ago I ran a blog focused on AI where I would try to post interesting practical tutorials for people to learn from. But around 2012ish with the launch of professional e-learning sites like Coursera I figured I was probably doing a disservice making content when people would probably be better served learning elsewhere from well regarded experts in the field.

However, since the launch of ChatGPT I've been thinking about reviving my blog partly to discus AI risk, but also to educate those new to neural networks how they work. To this day I still find the vast majority of the content on neural networks online to be extremely poor. You tend to find content is either highly academic and therefore inaccessible, or too high level, eg, "build a CNN in pytorch".

The first I dislike because it's not accessible or useful for those new to the field. The latter I dislike because they rarely touch on why things are done – sure anyone can copy paste some Pytorch code, but how does the Adam optimiser work? Why do we use ReLU activation vs sigmoid? How should weights be initialised? These are things that those inaccessible academic articles are actually useful for.

In my experience it's only when you try to build your own ANNs (without libraries) that you realise how much hand holding the libraries do and how little you actually understand. I had this issue when I was learning about AI at university too. It was mostly focused on the theory and it was only when trying translating that theory to something practical I realised how little I understood.

I recently watched Andrej Karpathy's introduction to neural networks and I was really impressed since it was both practical and informative. There's no magic libraries and everything discussed is demonstrated with practical examples helping develop intuition – exactly the kind of content I find difficult to find online.

Anyway, I guess this is a really long-winded way of asking if anyone learning about ANNs at the moment shares this frustration? Or if there's anything else you've found difficult?