This livestream provides a detailed review and comparison of two recent research papers advancing the application of reinforcement learning (RL) to Vision Language Models (VLMs): "GLM 4.1V thinking" and NVIDIA's "Scaling RL to long videos." The host breaks down the core methodology shared by both papers, which is heavily influenced by the DeepSeek model and its Group Relative Policy Optimization (GRPO) algorithm. A key insight is the adoption of a multi-stage training pipeline: initial pre-training, followed by a supervised "warm-up" on Chain-of-Thought (CoT) data to instill a reasoning format, and finally, a reinforcement learning phase to refine performance. The stream highlights crucial technical contributions, such as NVIDIA's sequence parallelism for efficient long-video processing and the GLM paper's finding that removing traditional RL constraints like KL divergence improves VLM performance. A central theme is the critical importance of data curation and curriculum design, extending from high-level training stages down to the composition of individual data batches.
Applying Reinforcement Learning to Vision Language Models: A Paper Review
0:02:27 - Today's Topic: VLM RL: The stream focuses on a deep dive into two recent papers that apply reinforcement learning (RL) to Vision Language Models (VLMs) to enhance their reasoning capabilities.
0:05:01 - Paper 1: GLM 4.1V thinking: This paper from JIPU AI introduces a VLM trained with a "reasoning-centric" framework. It utilizes reinforcement learning with curriculum sampling (RLCS) to improve performance on tasks like video understanding, coding, and long-document comprehension.
0:08:37 - Paper 2: Scaling RL to Long Videos: This NVIDIA paper focuses on the computational challenges of applying RL to long videos. They introduce a system (MRSP) that uses sequence parallelism to achieve a 2.1x speedup and avoid out-of-memory errors on long video inputs.
0:12:26 - Key Theme: Data Curation & Curriculum: Both papers dedicate significant effort to creating high-quality datasets and designing a curriculum. This involves careful data filtering, balancing, and structuring the training stages to effectively teach the model.
0:27:59 - The Core Algorithm: GRPO: A crucial commonality is that both papers use Group Relative Policy Optimization (GRPO), the RL algorithm from the DeepSeek paper. This method works by generating a group of possible answers ("rollouts") and training the model to increase the probability of producing the higher-reward answers relative to the lower-reward ones within that group.
0:35:48 - Removing RL Constraints: A key finding from the GLM paper is that traditional RL constraints, like KL divergence (which prevents the model from changing too much), are detrimental to VLMs. They found that removing this loss and allowing for larger gradient updates (higher clipping) let the model "run free" and achieve better performance.
0:45:29 - How Sequence Parallelism Works: The NVIDIA paper's efficiency gain comes from splitting a long video's frames across multiple GPUs. Each GPU processes a chunk of the video's vision tokens in parallel. These tokens are then gathered, significantly speeding up the most computationally expensive part of the process.
0:52:37 - The Importance of Caching: To further accelerate the RL process, the pre-computed vision tokens for each video are cached and reused. This avoids having to re-encode the same video frames multiple times during the numerous RL rollouts, which is a major training bottleneck.
1:22:24 - The Three-Stage Training Pipeline: Both papers use a similar training structure:
Pre-training: Building a base VLM on general image/video and text data.
Supervised Fine-Tuning (SFT) on Chain-of-Thought (CoT): A "warm-up" phase using a highly curated dataset to teach the model the think... answer... format.
Reinforcement Learning (RL): Using GRPO to further refine the model's reasoning capabilities by learning from its own generated data.
1:33:05 - What is Curriculum Sampling (RLCS)? The GLM paper's RLCS method makes training more efficient. It dynamically adjusts the difficulty of the problems in each training batch to match the model's current skill level, ensuring the model is always learning from data that is challenging but not impossible.
1:48:54 - SFT as a "Warm-Up": An interesting finding is that while some SFT on CoT data is essential to teach the model the reasoning format, extensive SFT may not be necessary. The RL phase is powerful enough to elevate models to a similar peak performance even if they start from a weaker SFT base.
2:22:30 - The Untapped Potential of Visual Data: The host concludes that applying the core principles of AI, like next-token prediction, to the vast and rich world of visual data (YouTube videos, game replays, etc.) is the next frontier and will unlock incredible advances in model intelligence.