This presentation introduces verl (Volcano Engine Reinforcement Learning), an open-source framework from Bytedance designed for building flexible and scalable end-to-end reinforcement learning (RL) pipelines for Large Language Models (LLMs). The talk details verl's core "hybrid controller" programming model, which combines the flexibility of a single-controller architecture for defining complex RL algorithms (like PPO) with the efficiency of a multi-controller paradigm for high-performance distributed execution within each stage. Key features include tight integration with VLLM for efficient generation, support for training massive Mixture-of-Experts (MoE) models, and mechanisms for optimal model sharding between different pipeline phases. Recent developments focus on enabling complex agentic tasks through an asynchronous rollout server for efficient multi-turn tool calling and a one-step asynchronous training pipeline that improves efficiency by pipelining decoupled generation and training workers. Future work includes a more modular design, more advanced asynchronous algorithms, and additional reproducible recipes.
Verl: A Framework for Scalable LLM Reinforcement Learning
0:53 Motivation for RL in LLMs: Post-training with Reinforcement Learning (RL) is increasingly critical for enhancing advanced LLM capabilities like reasoning, coding, and tool-use. Many state-of-the-art models and agentic products rely on end-to-end RL to learn how to interact with external environments like browsers and code sandboxes.
2:47 The Complexity of RL Pipelines: RL training is a complex data flow involving multiple models (actor, critic, reference, reward) and distinct computational workloads (generation, inference, training). This creates a challenging cyclic dependency where the newly trained model is immediately used for the next generation step, requiring a sophisticated system to manage.
8:18 Verl's Hybrid Controller Paradigm: To balance programming flexibility with execution efficiency, verl uses a hybrid model. A "single controller" manages the high-level algorithm logic, making it easy for researchers to program complex data flows. This controller then dispatches tasks to "multi-controller" groups that execute specific stages (like training or generation) with maximum parallelism using libraries like FSDP or Megatron.
21:54 Supporting Agentic Tasks and Tool-Calling: For complex agent interactions, verl has an asynchronous rollout server. This handles each generation request in a streaming fashion, preventing the entire batch from being bottlenecked by slow, multi-turn tool interactions (e.g., waiting for a code sandbox to execute).
27:32 Scaling to Huge Models: The framework is designed to train large-scale Mixture-of-Experts (MoE) models like DeepSeek. It includes integrations with distributed training backends like Megatron to effectively run on hundreds of GPUs.
29:21 Asynchronous Training Architectures: To overcome the overhead of synchronous pipelines, verl now supports a "one-step off" asynchronous training recipe. This uses a disaggregated architecture (separate devices for generation and training) and allows the generation model's weights to be slightly "stale," creating a pipeline that boosts throughput by over 20%.
34:37 Project Roadmap: Future work on verl focuses on creating a more modular, algorithm-agnostic design, developing more advanced asynchronous pipelines (e.g., partial rollouts where training occurs mid-generation), improving performance with techniques like FP8, and adding more reproducible recipes for agent benchmarks.
42:21 Why Token-Based APIs are Preferred: For agentic RL, using token-ID-based generation APIs is more stable than text-based chat APIs. This is because repeatedly encoding and decoding text during multi-turn conversations can introduce subtle inconsistencies that destabilize the training process.
55:17 Debugging RL Training: Debugging RL is notoriously difficult due to its inherent instability. Verl provides tools to help by allowing users to dump and visually inspect generated samples at each stage and by logging key metrics, such as log probability discrepancies between the generation and training phases, to identify numerical or algorithmic issues.