This research addresses the challenge of aggressive, time-optimal quadcopter control by introducing a novel end-to-end reinforcement learning (E2E RL) approach that directly generates motor commands, bypassing traditional inner-loop controllers. To mitigate the significant sim-to-real gap inherent in robotics, the method integrates a learned residual model and an adaptive compensation mechanism for modeling errors in thrust and moments. Comparative analysis against a state-of-the-art controller, which utilizes an INDI inner loop, demonstrates E2E RL's superior performance, achieving a 1.39-second advantage in simulation and a 0.17-second lead in real-world flight. The observed performance disparity between simulation and reality highlights promising avenues for future research, including enhanced sim-to-real strategies and the application of offline reinforcement learning using real flight data.
Summarizing the Transcript: High-Speed End-to-End Reinforcement Learning for Quadcopter Control
Aggressive Quadcopter Control Challenge: The core problem is achieving aggressive, time-optimal control for quadcopters, which is a significant robotics challenge.
Limitations of State-of-the-Art (SOTA): Current SOTA methods use reinforcement learning (RL) with robust inner-loop controllers (like INDI), but this abstraction can limit optimality and necessitates adding margins for disturbances.
Novel End-to-End RL (E2E) Approach: The proposed solution is a high-speed quadcopter control system using end-to-end RL, which directly outputs motor commands, removing the need for an abstract inner-loop controller.
Bridging the Sim-to-Real Gap: To ensure practical application, the E2E approach incorporates a learned residual model and an adaptive method designed to compensate for modeling errors in thrust and moments, crucial for real-world performance.
Performance Comparison (Simulated & Real-World):
The E2E method was directly compared against a SOTA network that commands thrust and body rates to an INDI inner-loop controller.
In simulation, E2E demonstrated a significant 1.39-second time advantage.
In real-world flight, E2E maintained a 0.17-second advantage.
Key Takeaway & Future Potential: The results underscore the substantial potential of end-to-end reinforcement learning for quadcopter control. The observed performance drop from simulation to reality indicates areas for further improvement, such as refining strategies to bridge the reality gap or exploring offline RL with real flight data.