trybackprop

A blog for learning all things ML – concepts, interview tips, online ML courses: https://www.trybackprop.com

I made a list of all the free resources I used to study ML and deep learning to become an ML engineer at FAANG, so I think it'll be helpful to follow these resources: https://www.trybackprop.com/blog/top_ml_learning_resources (links in the blog post)

Fundamentals Linear Algebra – 3Blue1Brown's Essence of Linear Algebra series, binged all these videos on a one hour train ride visiting my parents

Multivariable Calculus – Khan Academy's Multivariable Calculus lessons were a great refresher of what I had learned in college. Looking back, I just needed to have reviewed Unit 1 – intro and Unit 2 – derivatives.

Calculus for ML – this amazing animated video explains calculus and backpropagation

Information Theory – easy-to-understand book on information theory called Information Theory: A Tutorial Introduction.

Statistics and Probability – the StatQuest YouTube channel

Machine Learning Stanford Intro to Machine Learning by Andrew Ng – Stanford's CS229, the intro to machine learning course, published their lectures on YouTube for free. I watched lectures 1, 2, 3, 4, 8, 9, 11, 12, and 13, and I skipped the rest since I was eager to move onto deep learning. The course also offers a free set of course notes, which are very well written.

Caltech Machine Learning – Caltech's machine learning lectures on YouTube, less mathematical and more intuition based

Deep Learning Andrej Karpathy's Zero to Hero Series – Andrej Karpathy, an AI researcher who graduated with a Stanford PhD and led Tesla AI for several years, released an amazing series of hands on lectures on YouTube. highly highly recommend

Neural networks – Stanford's CS231n course notes and lecture videos were my gateway drug, so to speak, into the world of deep learning.

Transformers and LLMs Transformers – watched these two lectures: lecture from the University of Waterloo and lecture from the University of Michigan. I have also heard good things about Jay Alammar's The Illustrated Transformer guide

ChatGPT Explainer – Wolfram's YouTube explainer video on ChatGPT

Interactive LLM Visualization – This LLM visualization that you can play with in your browser is hands down the best interactive experience with an LLM.

Financial Times' Transformer Explainer – The Financial Times released a lovely interactive article that explains the transformer very well.

Residual Learning – 2023 Future Science Prize Laureates Lecture on residual learning.

Efficient ML and GPUs How are Microchips Made? – This YouTube video by Branch Education is one of the best free educational videos on the internet, regardless of subject, but also, it's the best video on understanding microchips.

CUDA – My FAANG coworkers acquired their CUDA knowledge from this series of lectures.

TinyML and Efficient Deep Learning Computing – 2023 lectures on efficient ML techniques online.

Chip War – Chip War is a bestselling book published in 2022 about microchip technology whose beginning chapters on the invention of the microchip actually explain CPUs very well

I wrote a post about how to build image search using OpenAI's CLIP, and I included a Google Colab notebook that walks you through step by step the coding process. My article also gives the reader all the fundamental linear algebra and ML they need to know. Enjoy!

Thanks for the feedback!

From my experience with 5 years in “software engineering” and then 7 years in “machine learning”, what matters is the most is that you like what you do so that you can bring your A game to work. That’ll separate you from the average engineer and management and peers will take notice. Yes it’s very tough to find a job right now, and there will always be down cycles. But I’ve noticed the best engineers are able to stay afloat even during downturns because they’ve built up a reputation for being a good engineer. Plus, you can always transition into ML if you work hard enough. Even with an ML degree, you’re not guaranteed to find an ML job these days. I actually wrote a blog post about how folks transitioned into ML that you might find useful: https://www.trybackprop.com/blog/2024_06_09_you_dont_need_a_...

I just posted part 2: https://news.ycombinator.com/item?id=40846513. It covers the dot product and embeddings and features more visuals and two interactive playgrounds to reinforce the concepts learned. Hope you find it useful.

Thanks! I built the interactive playgrounds with React in TypeScript, a lot of SVG manipulation. All of this was built with either Python for visuals or React in general.

Learn the linear algebra you need to know for AI/ML. This covers the dot product both algorithmically and visually and applies it to machine learning embeddings. This article also contains visualizations and two interactive playgrounds, the Interactive Dot Product Playground [0] and the Interactive Embedding Explorer [1] (best viewed with laptop or desktop!) to reinforce the concepts that are taught.

[0] https://www.trybackprop.com/blog/linalg101/part_2_dot_produc...

[1] https://www.trybackprop.com/blog/linalg101/part_2_dot_produc...

I actually wrote a blog post about this for experienced software engineers like you who are thinking of transitioning to ML, so I wanted to share it here: https://www.trybackprop.com/blog/2024_06_09_you_dont_need_a_...

I write about various engineers who now work at Meta, Google, Amazon, and OpenAI who made the switch. You can see what strategies and tactics they used to do it.

1) It's "wise" if you find during your personal hours you are enjoying hacking on it. Before I made the switch, I spent a year studying the material on nights and weekends so that was m my first data point that perhaps this is something I wanted to do full time.

2) Yes, I have! And I've been an ML engineer for 7 years now after I made the switch. For context, I'm an ML tech lead at FAANG. Prior to that, I worked in infrastructure and product.

3) One piece of advice I got on this years ago is to join a team adjacent to ML work so that you can get familiar with what production ML looks like. You can also start practicing ML thinking on Kaggle.com.

P.S. You can check out other posts in my blog for resources to learn AI/ML and the math needed for this career, such as my Linear Algebra 101 for AI/ML series: https://www.trybackprop.com/blog/linalg101/part_1_vectors_ma... (includes interactive quizzes, fundamentals of vectors/matrices, and a quick intro to PyTorch, an open source ML framework widely used in industry)

Thank you for the kind words! Please feel free to leave any feedback if you have any. Thanks!

thank you for the feedback! Ah, I can see how switching from column form to row form right away is confusing, especially if someone isn't familiar with all the notation and the code yet. Same with introducing the tensor method without talking about what a tensor is until later. Thanks for pointing this out! I'll make some adjustments.

I'm working on this series of articles for folks like you! So glad to hear that you found it useful. Stay tuned for part 2!

I love 3Blue1Brown's Essence of Linear Algebra series! Yeah, I wrote this article knowing it'd be review for a lot of folks, but I wanted to at least get readers on the same page before diving deeper into the more interesting aspects of ML. Thanks for reading! Stay tuned for part 2 of my Linear Algebra 101 series.

Thank you for the kind words! I currently working on Part 2 of the series (dot products, matrix multiplication, similarity search) and hope to have it out soon. Hope you continue reading!

Thanks! Part 2 (in the works) will cover visual explanations of the dot product and matrix multiplication, and I'll discuss how to build a visual similarity search engine. Stay tuned!

Was not expecting to be on the front page of HN! Please feel free to leave feedback on the website (i.e., mobile experience, content, quiz, etc.) Thanks!

Here are three sources:

Mistral AI employee: https://x.com/dchaplot/status/1800601722722935153

TechCrunch: https://techcrunch.com/2024/06/11/paris-based-ai-startup-mis...

Wall Street Journal: https://www.wsj.com/tech/ai/french-startup-mistral-ai-raises...

It looks like Silicon Valley really values the experience of execs from Instagram:

OpenAI CPO: Kevin Weil, Instagram VP of Product 2016-2018 [1]

Anthropic CPO: Mike Krieger, Instagram Cofounder and CTO 2010-2018 [2]

OpenAI VP Consumer Product: Peter Deng, Instagram Head of Product 2013-2015 [3]

[1] https://www.linkedin.com/in/kevinweil/

[2] https://www.linkedin.com/in/mikekrieger/

[3] https://www.linkedin.com/in/peterxdeng/

+1 the ultra-wide viewing experience is better than what I was expecting: dual monitors

I select papers based on references from coworkers, Twitter posts by prominent ML researchers I follow, ML podcasts, and results.

The research becomes relevant immediately because my team is always looking to incorporate it into our production models right away. Of course it does take some planning (3-6 months) before it's fully rolled out in production.

In a given week, I usually do the following:

* 15% of my time in technical discussion meetings or 1:1's. Usually discussing ideas around a model, planning, or ML product support

* 40% ML development. In the early phase of the project, I'm understanding product requirements. I discuss an ML model or algorithm that might be helpful to achieve product/business goals with my team. Then I gather existing datasets from analysts and data scientists. I use those datasets to create a pipeline that results in a training and validation dataset. While I wait for the train/validation datasets to populate (could take several days or up to two weeks), I'm concurrently working on another project that's earlier or further along in its development. I'm also working on the new model (written in PyTorch), testing it out with small amounts of data to gauge its offline performance, to assess whether or not it does what I expect it to do. I sanity check it by running some manual tests using the model to populate product information. This part is more art than science because without a large scale experiment, I can only really go by the gut feel of myself and my teammates. Once the train/valid datasets have been populated, I train a model on large amounts of data, check the offline results, and tune the model or change the architecture if something doesn't look right. After offline results look decent or good, I then deploy the model to production for an experiment. Concurrently, I may be making changes to the product/infra code to prepare for the test of the new model I've built. I run the experiment and ramp up traffic slowly, and once it's at 1-5% allocation, I let it run for weeks or a month. Meanwhile, I'm observing the results and have put in alerts to monitor all relevant pipelines to ensure that the model is being trained appropriately so that my experiment results aren't altered by unexpected infra/bug/product factors that should be within my control. If the results look as expected and match my initial hypothesis, I then discuss with my team whether or not we should roll it out and if so, we launch! (Note: model development includes feature authoring, dataset preparation, analysis, creating the ML model itself, implementing product/infra code changes)

* 20% maintenance – Just because I'm developing new models doesn't mean I'm ignoring existing ones. I'm checking in on those daily to make sure they haven't degraded and resulted in unexpected performance in any way. I'm also fixing pipelines and making them more efficient.

* 15% research papers and skills – With the world of AI/ML moving so fast, I'm continually reading new research papers and testing out new technologies at home to keep up to date. It's fun for me so I don't mind it. I don't view it as a chore to keep me up-to-date.

* 10% internal research – I use this time to learn more about other products within the team or the company to see how my team can help or what technology/techniques we can borrow from them. I also use this time to write down the insights I've gained as I look back on my past 6 months/1 year of work.

From the blog post: > We observe that with AI-based suggestions, the code author increasingly becomes a reviewer, and it is important to find a balance between the cost of review and added value. We typically address the tradeoff with acceptance rate targets.

In the past year since GPT-4 came out, I've also found this to be the case. I'm an ML/backend engineer with little experience in frontend development. Yet, I've been able to generate React UIs and Python UIs with GPT-4 in a matter of minutes and simply review the code to understand how it works. I find this to be very useful!