Automating ML Research: Evolutionary Code Generation for the ARC AGI Challenge
0:00 Introduction: The streamer outlines his plan to tackle the ARC AGI challenge, a reasoning benchmark, using an automated ML research approach.
4:11 The Challenge: The ARC AGI challenge, described as the MNIST of reasoning, presents visual reasoning tasks that are easy for humans but challenging for AI.
6:22 The Hypothesis: The streamer hypothesizes that within the vast space of simple deep learning programs, there exists a program capable of solving the ARC AGI challenge. His approach is inspired by examples of surprisingly effective small models in chess and the Lottery Ticket Hypothesis.
10:28 Evolutionary Code Generation: The streamer introduces EvoArc, a repository containing code for his approach. It involves an evolutionary loop of mutation and selection, applied to simple deep learning training scripts.
11:00 Tools: The streamer utilizes GPT-40 as the mutation agent, generating variants of the base training script. Jax is chosen as the deep learning framework for its flexibility in representing a wide range of programs. The process runs on multiple devices, including a Jetson AGX Orin, a 3090, and a cloud instance with an H100.
13:00 Mutation Prompts: The streamer shows examples of mutation prompts given to GPT-40, including instructions to modify hyperparameters, models, or training loops. He also incorporates random archive paper abstracts and "glazing prompts" that encourage the model to produce high-quality code.
20:11 Challenges: The streamer highlights the challenge of GPT-40 frequently generating invalid code, especially due to dimensionality mismatch errors. He emphasizes the importance of a quick, iterative workflow with minimal debugging, emphasizing code generation over manual refinement.
38:00 RNN Implementation: The streamer asks both GPT-40 and Claude to implement a simple RNN solution. Both models struggle with dimensionality errors, requiring multiple iterations and back-and-forth fixes.
46:00 Experimentation: The streamer tests various models (MLP, CNN, RNN) generated by the different LLMs, discarding those that consistently fail. He starts an evolutionary search process on the working models, allowing the process to run on a powerful cloud instance.
55:00 Stream Conclusion: The stream ends with a summary of the project and the successful generation of working RNN and MLP models. The streamer reflects on the potential of this approach to discover simple yet surprisingly effective solutions to the ARC AGI challenge.