Deep Dive into LLMs like ChatGPT: A Comprehensive Overview by Andrej Karpathy
0:00 Introduction: The video aims to provide mental models for understanding LLMs, addressing their capabilities, limitations, and how they work.
1:02 Pre-training Data (Internet): LLMs are first pre-trained on vast amounts of text from the internet, including sources like Common Crawl and datasets like FineWeb (44TB). Filtering processes are applied to ensure data quality, removing unwanted content (malware, spam) and personally identifiable information (PII).
7:47 Tokenization: Text is converted into a sequence of tokens, using techniques like Byte Pair Encoding (BPE). GPT-4 uses 100,277 tokens, which are fed into the neural network.
14:27 Neural Network I/O: LLMs model statistical relationships by taking windows of tokens (up to 8,000) and predicting the next token. The input is a sequence of tokens, and the output is a probability distribution for the next token.
20:11 Neural Network Internals: The inputs are mixed with parameters in a mathematical expression. A Transformer neural network with 85,000+ parameters consists of layers with mathematical expressions, and then needs to find a good setting of these parameters
26:01 Inference: Generating new data from the model involves sampling tokens based on the probability distribution, and feeding the output back into the model.
31:09 GPT-2: Training and Inference: Karpathy discusses OpenAI's GPT-2 (1.5B parameters) as a recognizably modern LLM.
31:09 GPT-2: Training and Inference: GPT-2 was trained on 100 billion tokens, and one can reproduce it today for about $100.
42:52 Llama 3.1 base model inference: Karpathy demonstrates how to interact with Meta's Llama 3.1 405B parameter base model, highlighting its limitations as a glorified autocomplete until refined.
59:23 Pretraining to post-training: After pre-training, the base model has to undergo post-training to be made into an assistant. This happens in three steps.
1:01:06 Post-training Data (Conversations): LLMs are fine-tuned on datasets of conversations between humans and assistants, enabling them to respond to questions, reject harmful requests, and adopt a specific persona.
1:20:32 Hallucinations, Tool Use, Knowledge/Working Memory: LLMs can hallucinate if they lack specific knowledge, and their inability to say “I don’t know” can compound the problem. Tool use, like web search, enables LLMs to retrieve information and reduce hallucinations.
1:41:46 Knowledge of Self: The presenter discusses the question of self knowledge in LLMs and how it is engineered with the help of data or a special system message.
1:46:56 Models Need Tokens to Think: To solve problems, LLMs should distribute computation across many tokens, as there is a limited amount of computation per token. They should be pushed to generate intermediate steps in chain of thought, and be asked to use tools in a toolbox whenever possible.
2:01:11 Tokenization Revisited: Models Struggle with Spelling: Since the LLMs don't see characters they are bad at spelling related tasks.
2:04:53 Jagged Intelligence: It’s important to treat LLMs as stochastic systems because they will randomly do dumb things. An example of this is where they fail to compare two simple numbers and get the correct answer.
2:07:28 Supervised Finetuning to Reinforcement Learning: The video describes the step-by-step process of going from pre-training, to supervised finetuning, to reinforcement learning, and what type of data would be used in each step.
2:14:42 Reinforcement Learning: Models try various solutions and are rewarded for correct answers. They learn cognitive strategies like re-evaluating steps and approaching problems from different perspectives.
2:27:47 DeepSeek-R1: DC-R1 demonstrates how the model's responses get a lot more verbose, and does additional reasoning in its head.
2:42:07 AlphaGo: Karpathy draws a parallel to DeepMind's AlphaGo, where reinforcement learning allowed the system to surpass human players. In AlphaGo, the system played against itself and discovered plays to win.
2:48:26 Reinforcement Learning from Human Feedback (RLHF): The video describes the way LLMs can use human feedback to train reward models, then improve their performance.
3:09:39 Preview of Things to Come: Multimodality (audio and images), long-running agents, and computer use are cited as future directions.
3:15:15 Keeping Track of LLMs: El Marina, AI News Newsletter, and X (Twitter) are all recommended for staying up to date.
3:18:34 Where to Find LLMs: LLMs can be found at the website of that provider (e.g. ChatGPT), inference providers such as together.ai, or by using open weights models locally on LM Studio.