Omar's Journey: Learning Cuda, Triton, and Flash Attention
0:00: Introduction to a special 50th episode, highlighting the GPU mode community and its importance in learning. Omar J is introduced as a guest speaker to share his learning journey with Cuda, Triton, and flash attention.
0:30: Talk is about the learning journey in Cuda Triton and flash attention because of the question "How do I learn Cuda how do I learn machine learning".
0:40: Quit job to focus full time for learning stable diffusion.
0:51: Challenging ancient wisdom of teaching to fish. The challenge is by the time one learns to fish, they end up in the jungle, where there's no point in fishing. The modern way is teaching new skills so he can build a fishing boat to the entire village.
6:43: Definition of Flesh Attention, Cuda, and Triton.
12:40: Before starting a journey ask yourself, why do I want to study this? Don't do that because it is hype.
13:00: Follow Practical Problems. Always had an issue with new architectures so decided to go to Triton Website.
14:40: How to fill knowledge gaps? GPT can help with understanding terms that are being presented on the website.
16:37: Asking GPT for a roadmap to understanding is important. The road map is not as important as what steps you need to take to learn.
17:24: Next step is to get your hands dirty.
17:48: Difficulty Level.
18:48: Build necessary knowledge to understand stable diffusion. It is good to have a objective to get through tutorials.
20:24: Like to go "hardcore mode" by only reading papers and challenging themselves. Not watch tutorials.
22:08: Tutorial Vector Addition.
24:20: Always make sure where you are and you need to come back there because you need to doing TRON to learn flashh attention so you should always try to stay on your path unless you really want to change your goal
27:50: The best approach for papers is to read the entire paper top to bottom. Highlight the important information and annotate what are the things that you don't understand.
32:00: Examples: Self-attention does not need o and squared memory paper, Online Normalizer calculation for softmax.
32:56: Test it on paper, code it by yourself or have GPT code it. It is very important to understand it for active learning. Read the proof and test yourself on hypothesis and prove it by yourself. This way you will learn.
36:10: Go back and forth to reinforce understanding.
37:56: Talk about Block Matrix Multiplication.
39:56: Boss battle: Tensor Shapes and Strides.
40:40: Advantages of strid.
44:06: Final boss battle: the "back propagation". Compute the gradients. Chain rule.
48:51: Recommend you do a challenge and do it and move on. What should I learn next for career?
49:42: Follow curiosity, not hype.
50:45: Focus on consistency because it doesn't matter how you start, but with enough time, consistent trials, you will get there.
52:41: Avoid the noise and start learning dual streams. Also have something to build on for a long term.
53:58: Final and secret ingredient: It's You! First need to be able to convince yourself that you can do and learn these things to be successful. Need to build confidence and this comes by doing hard things.
56:43: Don't just follow tutorial. Push yourself and use modern tools like GPT, etc. You can get better code and do it faster with AI. Learn how to promote things the right way. There has to be actual learning or the bot can take over.
57:16: Recommend to do better than what you saw on book or video. Build Confidence to take you far.
58:11: Answering questions about quitting job for learning, advice for people, steps to learning Cuda, etc.
1:00:45: Don't have to quit job but do something to build on what was being done.
1:00:53: Make sure the tools help get the job done faster.
1:02:39: Set up a problem you have to solve.
1:04:23: Contribute to Open Source to improve, get yourself noticed, and start engaging.
1:06:28: Don't always think in terms of X or Y, instead learn both.
1:06:46: Optimizations.
1:07:19: Software stack can be optimized.
1:07:32: Fan of 10,000 hours of learning.
1:07:50: Learn language, AI, and how to get there.
1:19:20: Last question was about what was his AGI timeline.