This paper introduces Group Sequence Policy Optimization (GSPO), a novel reinforcement learning (RL) algorithm designed to address the instability and inefficiency of existing methods like GRPO when training large language models (LLMs). The authors identify that the core problem in previous algorithms is a flawed, token-level application of importance sampling, which introduces high-variance noise and can lead to catastrophic model collapse. GSPO resolves this by aligning the optimization with the reward signal, defining its importance ratio, clipping, and optimization objective at the sequence level. Empirical results show GSPO achieves significantly better training stability, efficiency, and performance. Critically, it inherently stabilizes the training of Mixture-of-Experts (MoE) models without needing complex workarounds and has the potential to simplify RL infrastructure. These benefits were instrumental in the performance improvements of the latest Qwen3 models.
Group Sequence Policy Optimization (GSPO): A New RL Algorithm for LLMs
The Problem: Instability in Current RL Algorithms
Existing state-of-the-art reinforcement learning algorithms, such as Group Relative Policy Optimization (GRPO), exhibit severe stability issues when training large-scale language models.
This instability often results in "model collapse," an irreversible failure of the training process. The paper identifies the root cause as a flawed application of importance sampling at the token level, which introduces high-variance noise that accumulates over long sequences.
The Solution: Group Sequence Policy Optimization (GSPO)
GSPO is a new RL algorithm that corrects this fundamental issue by performing optimization at the sequence level.
Key Innovation: Instead of calculating an importance ratio for each token, GSPO calculates a single importance ratio for the entire generated sequence. This aligns the optimization unit with the reward, which is also given for the whole sequence.
Benefits for Mixture-of-Experts (MoE) Models
Training MoE models with traditional RL is challenging due to "expert-activation volatility"—the set of activated experts changes between updates, destabilizing token-level metrics.
GSPO fundamentally resolves this issue because its sequence-level likelihood is robust to changes in individual expert activations. This eliminates the need for complex and costly workarounds like "Routing Replay," allowing for more stable and efficient MoE training.
Superior Training Efficiency and Performance
Experiments show GSPO delivers continuous performance improvement and is more efficient than GRPO. It achieves higher training rewards and better scores on benchmarks like AIME’24 and CodeForces with the same amount of computational resources.
This efficiency is achieved despite GSPO clipping a much larger fraction of tokens, suggesting its sequence-level approach provides a more reliable and effective learning signal.
Potential to Simplify RL Infrastructure
Because GSPO uses sequence-level likelihoods, it is more tolerant of precision differences between separate training and inference engines.
This could allow models to use likelihoods directly from an inference engine, eliminating a costly recomputation step and simplifying the overall training pipeline.
Impact and Application
The stability, efficiency, and performance improvements from GSPO were a key factor in the exceptional advancements of the latest Qwen3 series of models.