This video introduces LELO-bench, a new benchmark dataset and evaluation framework designed to assess the performance of long-term robot localization algorithms in dynamic indoor environments. The work addresses the common failure of traditional localization methods that assume a static world, which is often not the case in real-world settings like warehouses or hospitals. The dataset was collected using a mobile robot equipped with LiDAR, RGB-D cameras, and wheel encoders in a multi-room office. A key feature is its highly accurate, millimeter-range ground truth pose, obtained via a novel method using ceiling-mounted AprilTags and an upward-facing camera, which overcomes the line-of-sight limitations of typical motion-capture systems. The benchmark includes sequences with four distinct conditions: static, dynamic (moving people), long-term changes (rearranged furniture), and a combination of both. To foster research, the creators have established a public challenge with a server for automated evaluation and a leaderboard to rank the performance of submitted algorithms.
LELO-bench: A Benchmark for Long-Term Robot Localization
0:00:03 The Core Problem: Standard robot localization assumes a static environment. However, real-world indoor spaces like warehouses and hospitals change over time due to moving people (dynamics) or rearranged furniture (long-term changes), which can make existing maps obsolete and cause localization to fail.
0:01:26 Introducing LELO-bench: The video presents LELO-bench, a benchmark specifically created to evaluate how well localization algorithms perform in these challenging, changing environments over extended periods.
0:01:47 Data Collection Platform: The data was gathered with a mobile robot equipped with three RGB-D cameras, two LiDAR scanners, and wheel encoders, providing rich multi-modal sensor data.
0:02:00 Novel Ground Truth System: To ensure high accuracy, the researchers established ground truth using ceiling-mounted AprilTags. An upward-facing camera on the robot detects these tags to calculate its pose with millimeter-level precision continuously across multiple rooms, a method superior to traditional systems that require a constant line of sight.
0:02:58 Diverse and Challenging Scenarios: The dataset is composed of 21 sequences (totaling 2.5 hours) divided into four distinct categories for comprehensive evaluation:
Static: The environment is unchanged from the initial mapping run.
Dynamics: People are moving within the robot's sensor view.
Long-term changes: Furniture and other objects have been moved or added.
Combined: A mix of both dynamics and long-term changes.
0:03:32 Baseline Algorithms Struggle: Initial tests with standard Monte Carlo Localization (MCL) approaches showed that while they handle dynamics reasonably well, they are particularly challenged by sequences with long-term changes, highlighting the need for more robust algorithms.
0:03:59 A Public Evaluation Challenge: To encourage innovation, the test dataset has been released as part of a public challenge on the Codabench platform. Researchers can submit their algorithm's estimated poses for automated evaluation against the private ground truth.
0:04:14 Standardized Metrics and Leaderboard: The evaluation server automatically calculates pose errors and ranks submissions on a public leaderboard using a single, comprehensive score. This standardized approach allows for a direct and fair comparison of different localization methods.