Automated ML Research: Exploring Evolutionary Code Generation for the ARC AGI Challenge
4:10 The ARC AGI Challenge: A benchmark for reasoning, similar to MNIST but for more complex cognitive tasks.
4:26 Hypothesis: The streamer proposes that a small, simple deep learning program could be discovered that effectively solves the ARC AGI challenge through supervised learning. This is inspired by research showing:
Grandmaster level chess achieved with a small transformer model using supervised learning.
Distillation's effectiveness in replicating larger model performance with smaller models.
The lottery ticket hypothesis suggesting the existence of smaller, highly effective subnetworks within larger networks.
10:42 The EvoArc Project: An open-source project implementing evolutionary code generation for the ARC AGI Challenge.
10:55 The Morph: A single Jupyter notebook containing a simplified deep learning pipeline for tackling the ARC AGI challenge. Morphs are the evolving “organisms” in this process.
13:44 Mutation and Selection: The core of the evolutionary algorithm, mutating candidate solutions and selecting those that perform best.
13:44 Mutation via GPT-4: The GPT-4 language model is used to mutate the code within morphs, introducing variations.
13:55 Selection Based on Performance: The selection process favors morphs that achieve higher accuracy scores on the ARC AGI task.
20:46 Choice of JAX: JAX is chosen as the deep learning framework for its flexibility and wider space of representable programs, increasing the chances of finding a winning solution through evolution.
38:09 Challenges with Mutation: A major obstacle is the high failure rate of generated mutations, often resulting in non-functional code due to dimensionality mismatch errors and other issues.
54:39 RNN Implementation with GPT-4: The stream explores using GPT-4 to generate both a simple RNN and a multilayer perceptron (MLP) implementation to tackle the ARC AGI task, demonstrating the model's ability to generate code for different architectures.
1:21:44 Fixing Bugs with GPT-4: Sonnet's RNN implementation generated errors, showcasing the process of using GPT-4 to debug and fix the code by simply providing the error message and the problematic notebook.
1:46:18 Successful RNN and MLP: By the end, the stream demonstrates two working models: RNN Sonnet 01 and MLP 01, adding to the pool of functional morphs.
1:48:08 Starting the Evolutionary Process: The stream concludes with initiating the evolutionary process, setting the stage for future runs of mutation, selection, and further exploration of the solution space.
2:00:04 Exploring Neural Cellular Automata (NCA): Discusses NCAs as a potential model for the ARC AGI task due to their combination of CNN and RNN features and ability to learn spatial patterns. A paper showcasing NCA's outperformance of GPT-4 on a 1D version of the ARC AGI dataset is presented as a motivation.
2:09:58 Conclusion: The stream highlights the unique approach of evolutionary code generation and its potential to unlock surprisingly simple yet effective solutions for challenging reasoning tasks like the ARC AGI, suggesting a future where humans guide AI exploration by introducing unconventional ideas.