Abstract:

This stream provides a deep dive into two recent research papers that apply reinforcement learning (RL) to Vision Language Models (VLMs), exploring how these models can achieve sophisticated reasoning on images and long videos. The papers reviewed are "GLM-4.1V-Thinking" and NVIDIA's "Scaling RL to Long Videos." A central theme is the widespread adoption of techniques from the DeepSeek paper, including the GRPO (Group Relative Policy Optimization) algorithm and the "think/answer" chain-of-thought format. Both papers emphasize a multi-stage training pipeline (pre-training, supervised fine-tuning, and RL) and the critical importance of meticulous data curation and curriculum learning. Key innovations discussed include GLM's "Reinforcement Learning with Curriculum Sampling" (RLCS) for more efficient training and NVIDIA's MRSP system, which uses sequence parallelism and embedding caching to enable RL on hour-long videos. The stream concludes that applying scalable, RL-driven methods to the vast amount of visual data represents the next major frontier in AI development.

From Text to Vision: Summarizing the Latest in Reinforcement Learning for AI

• 0:02:51 Two Foundational Papers: The stream focuses on two papers advancing Vision Language Model (VLM) reasoning through Reinforcement Learning (RL): "GLM-4.1V-Thinking" and NVIDIA's "Scaling RL to Long Videos." Both aim to extend the reasoning capabilities seen in text-based LLMs to models that can process visual data.
• 0:14:52 A Common Technical Stack: Both research projects rely on a similar set of tools, including Hugging Face transformers, PyTorch, TorchVision, and notably vLLM for highly efficient inference. This efficient inference is critical for the RL phase, where the model must rapidly generate its own data (called "rollouts") to train on.
• 0:27:59 The DeepSeek Influence (GRPO): A key takeaway is that both papers use the GRPO (Group Relative Policy Optimization) algorithm from the influential DeepSeek paper. Instead of relying on a complex, absolute reward for a single answer, GRPO generates a group of candidate answers and optimizes the model based on their relative correctness within that group.
• 0:35:48 Letting the Model "Run Free": The GLM paper discovered that for VLMs, removing the standard KL-divergence penalty (which prevents the model from changing too much) was beneficial. This allows the RL process to significantly transform the model, rather than just slightly tuning it, leading to better performance.
• 0:52:37 Solving the Video Bottleneck: The NVIDIA paper introduces two key optimizations for handling long videos:
  • Sequence Parallelism (MRSP): The video is split into chunks, and each chunk is processed on a different GPU simultaneously, drastically reducing the time needed to create vision tokens.
  • Embedding Caching: The generated vision tokens for each video frame are saved (cached). Since the vision encoder is "frozen" (not trained), these tokens can be reused for every RL training step, eliminating the need to re-process the video frames repeatedly.
• 1:00:59 The Central Role of Data Curation: A massive part of both projects is curating the training data. This involves a multi-step pipeline: filtering public datasets, using models like CLIP to ensure text descriptions are relevant to images, re-balancing the data to cover diverse concepts, and even using other AI models to write more detailed, factual captions for images.
• 1:22:24 The Three-Stage Training Pipeline: Both papers follow a similar training curriculum:
  1. Pre-training: A standard VLM is trained on a massive dataset of image/video and text pairs.
  2. Supervised Fine-Tuning (SFT): The model is "warmed up" on a smaller, high-quality dataset of long reasoning examples to teach it the "think/answer" format.
  3. Reinforcement Learning (RL): The model's reasoning is sharpened using GRPO, where it learns from its own self-generated attempts.
• 1:48:54 How Much "Warm-up" is Needed? An interesting finding from the GLM paper is that while the SFT "warm-up" stage is necessary to get the model into the right format, doing more of it doesn't necessarily lead to a better final model. A powerful RL process can elevate a model to its peak potential even if it starts from a less-optimized SFT checkpoint.
• 2:22:15 The Next Frontier: The Visual World: The streamer concludes that these papers demonstrate the future of AI. The same "next token prediction" principle that created powerful text-based AI can now be applied to the vast, untapped world of video. Training models to predict "what happens next" in a video will unlock a new level of general intelligence.