This video is a tutorial on using Rerun's DataFusion-based query interface to programmatically access and align multimodal robotics data. It demonstrates setting up a Rerun server, connecting to it from a Jupyter notebook, and executing SQL-like queries on RRD files. The core concept shown is temporal alignment of asynchronous sensor streams (IMU, GPS, obstacle detection) by "pulling forward" the latest available value from slower streams onto the timeline of a higher-frequency stream. The ultimate goal of this process is to prepare a synchronized, tabular dataset suitable for ingestion into machine learning pipelines. The tutorial also covers converting query results into Pandas DataFrames and performing basic data manipulation like filtering and unnesting columns.
A Guide to Querying and Aligning Robotics Data in Rerun
1:03 Server and Client Architecture: The tutorial starts by demonstrating how to run a Rerun server that points to a directory of data files (RRDs). A separate rerun connect command launches a viewer, decoupling the data source from the visualization client. This setup allows multiple clients to connect to the same data server.
2:31 Programmatic Data Access: The video shows how to connect to the Rerun server from a Jupyter notebook using a catalog_client. This client is the entry point for querying the data programmatically.
3:13 Schema and Timeline Inspection: You can use the client to inspect the contents of a recording, listing its schema, component columns, and available timelines (e.g., imu_time, gps_time).
5:16 Querying Aligned Data: The core of the tutorial demonstrates how to query the dataset from the perspective of a single timeline. This action programmatically simulates what happens when a user scrubs through a specific timeline in the Rerun viewer.
6:43 Temporal Synchronization: The key takeaway is the ability to align asynchronous sensor data. The query interface can pull forward the last known value from a low-frequency stream (like GPS) and align it with each timestamp of a high-frequency stream (like IMU), creating a fully populated, synchronized data row.
9:55 Preparing Data for Machine Learning: This synchronization is presented as a critical step for preparing data for machine learning models, which typically require a consistent set of feature values for each time step.
10:50 DataFusion and Pandas Integration: The system uses Apache DataFusion for its query engine, providing SQL-like capabilities for filtering, selecting, and manipulating data (e.g., unnesting columns). The final query results can be converted directly into a Pandas DataFrame.
12:09 Active Development: The presenter notes that this querying functionality is under active development and encourages users to provide feedback on their use cases.