This video captures a live coding and debugging session focused on fine-tuning NVIDIA Groot, a robotic foundation model (specifically the N1.5 version), to a custom "Tatbot" embodiment. The presenter meticulously walks through the process of setting up the development environment, managing code with Git, configuring the la_robot data pipeline, and integrating Docker for consistent execution. Key challenges encountered include resolving data configuration mismatches, handling Python dependency issues (like flash_attention), and correctly setting up GPU access within Docker containers. Despite successfully debugging numerous software and configuration problems, the session culminates in an "out of memory" error on the local Nvidia 3090 GPU, highlighting the significant computational demands of the Groot model and necessitating a move to cloud infrastructure for training. The video offers a transparent look into the practical, often iterative, nature of robotics and AI development.
Fine-tuning NVIDIA Groot: A Live Debugging Session
0:00 Stream Introduction: The stream, titled "Finetuning at Home," features a live coding session aimed at fine-tuning a robotic foundation model, accompanied by AI-generated lofi background music.
4:01 System & Project Overview: The presenter uses an Ubuntu PC with an Nvidia 3090 GPU for the fine-tuning. The primary focus is adapting Nvidia Groot (N1.5), a recently updated robotic foundation model, to a custom bi-manual "Tatbot" (tattooing robot) embodiment.
8:48 Fine-tuning Strategy: The fine-tuning concentrates on embodiment-specific modules (state encoder, action encoder, action decoder) to tailor the model to Tatbot's unique physical characteristics, rather than retraining the larger vision or language model components.
10:49 Development Workflow: The importance of maintaining a detailed markdown file for project instructions and using a disciplined Git workflow (pulling from upstream) to manage code changes and avoid conflicts is demonstrated.
15:07 Human-in-the-Loop Robotics: A discussion on HILS (Human-in-the-Loop Reinforcement Learning) and its application in robotics as a business model, where human operators initially intervene to guide robot behavior, gradually reducing their involvement as policies improve ("ramped autonomy").
20:19 Dataset & Modality Configuration: A small, "throwaway" dataset of Tatbot tracing calibration patterns is used for initial pipeline debugging. A key mismatch ("Skipping left arm") in the modality.json file, which maps robot data to Groot's expected format, is identified and corrected.
29:40 NVIDIA's Open-Source Commitment: The presenter highlights Nvidia's strategic decision to leverage Hugging Face's la_robot tooling for dataset management and model publication, emphasizing its positive impact on the open-source robotics community.
34:08 Debugging Initial Training Errors: The first attempt to run the training script fails due to an invalid data_config choice. This is resolved by manually adding a TatbotDataConfig class and updating the data transformation pipeline within the Groot fine-tuning script.
52:21 Dependency & Docker Issues: Subsequent errors arise from missing flash_attention modules. This leads to the decision to use Docker containers for a more controlled environment. Building the Docker image and crafting the docker run command for GPU access (--gpus all vs. --gpus 0) introduces further debugging.
1:37:37 Overfitting Explanation: In response to a user question, the concept of overfitting is explained using training and validation loss curves, where divergence indicates the model is memorizing training data rather than generalizing. The presenter briefly discusses niche scenarios where a degree of overfitting might be acceptable for highly specialized applications.
1:43:53 Recommended Robotics Kit: The SO101 is strongly recommended as the best open-source robotic kit for 3D printing, citing its growing adoption by Nvidia, integration with la_robot, and support in simulation frameworks.
2:13:28 Final Training Hurdles & Hardware Limitations: A final shape mismatch in the action horizon (14 vs. 16) is debugged and corrected. However, the training repeatedly encounters "out of memory" errors, even with a minimal batch size, indicating that the Nvidia 3090 GPU is insufficient for training the large Groot model.
2:26:28 Conclusion: The session concludes with the acknowledgment that further training will require a cloud computing environment due to local hardware limitations, providing a realistic perspective on the debugging-heavy nature of robotics and AI development.