Okay, here is the abstract and summary for the provided transcript.
Abstract:
This presentation by Ian Ozsvald at Pi Data Global 2024 details his six-month personal project exploring the capabilities of Large Language Models (LLMs), specifically Llama 3, in solving novel reasoning problems from Kaggle's ARC AGI (Abstraction and Reasoning Corpus Artificial General Intelligence) challenge. The challenge involves transforming grid-based patterns according to hidden rules, presented via JSON. Ian outlines his methodology using local GPU hardware (RTX 3090), detailed prompt engineering inspired by Program-of-Thoughts prompting to guide the LLM to generate Python code solutions, and an isolated execution environment to evaluate the code. Key findings highlight the critical impact of input data representation, showing a significant performance increase (from 12% to 54% success) by switching from dense number blocks to a quoted, comma-separated format alongside JSON. Furthermore, implementing iterative feedback loops, particularly providing specific information about which examples the generated code failed or succeeded on, dramatically improved the success rate to around 75% without explicitly correcting the logic. Future directions include addressing the LLM's tendency to fixate on incorrect approaches and developing a 'techniques library' by summarizing successful code patterns generated by the LLM itself. The work suggests that treating LLM interaction akin to classical machine learning, focusing on representation (feature engineering) and iterative refinement with evaluation, is a powerful approach.
LLM Code Generation for Novel Reasoning Problems: Insights from the ARC AGI Challenge
0:00:03 Introduction: Ian Ozsvald, an interim Chief Data Scientist and active community member (Pi Data London co-founder, author), presents his work at Pi Data Global 2024.
0:02:47 The ARC AGI Challenge: The talk focuses on Kaggle's Abstraction and Reasoning Corpus (ARC) Artificial General Intelligence challenge, designed by François Chollet to test if LLMs can reason about novel problems they've never encountered.
0:03:23 Problem Format: The challenge uses grid-based visual puzzles (e.g., finding dominant colors, copying/resizing shapes) presented as JSON data. The goal is to infer the transformation rule from examples and apply it correctly to unseen test cases.
0:06:55 Experimental Setup: Ian uses a local setup with an RTX 3090 GPU running the Llama 3 (8 billion parameter) model via llama.cpp, allowing for relatively fast (~20 seconds) prompt evaluations.
0:07:55 Prompt Engineering: A detailed prompt structure is used:
Sets the LLM's role (problem-solver).
Provides 3 examples (input/output pairs) using specific data representations.
Asks the LLM to generate a Python transform function to solve the task, inspired by Program-of-Thoughts prompting.
0:10:52 Execution & Evaluation: The LLM-generated Python code is executed in an isolated subprocess (using joblib) to prevent environment contamination and evaluate its correctness against known solutions and hidden test cases.
0:12:11 Initial Success & Challenges: The LLM successfully generates correct Python/NumPy code for a specific problem about 42% of the time (using an early representation), but often produces non-runnable code, overfits, or identifies the wrong pattern.
0:13:48 Rationale for Multiple Runs: Inspired by recent research ("More Agents is All You Need", "Self-Consistency"), running the process multiple times and potentially ensembling results can significantly boost accuracy.
0:15:01 Representation Matters: Experimenting with different ways to represent the grid data in the prompt dramatically affects performance:
Takeaway: How data is presented to the LLM is crucial, similar to feature engineering.
0:18:18 Basic Iterative Feedback: Simply telling the LLM its previous code attempt "didn't work" and asking it to "try again" (while providing the prior code as context) improved the success rate from 54% to ~65% over several iterations.
0:20:01 Specific Iterative Feedback: Providing more detailed feedback – telling the LLM which specific input examples its code succeeded or failed on (without explaining why) – significantly boosted performance further, reaching ~75% success after about 5 iterations.
Takeaway: Targeted feedback, even without explicit correction, helps the LLM refine its approach effectively.
0:21:26 Code Execution as Grounding: Using code generation and evaluation provides a way to counteract LLM hallucinations, as the code's correctness can be objectively verified.
0:22:23 Future Work - Addressing Fixation: LLMs can get stuck on incorrect reasoning patterns (e.g., fixating on 'mode' when position matters). Ian is exploring ways to record history and guide the LLM away from repeatedly failed approaches.
0:23:40 Future Work - Techniques Library: A promising idea is to have the LLM summarize successful code snippets it generated, creating a "seed bank" of useful patterns (e.g., "majority wins logic") that could potentially be injected as hints for solving new, related problems.
Input representation is critical for LLM problem-solving.
Iterative refinement with scoring/evaluation is a powerful technique.
Generating and executing code provides grounding against incorrect LLM outputs.
Significant improvement (12% to 75% on one task) was achieved through representation changes and feedback loops alone.
0:25:54 Call for Collaboration: Ian is interested in collaborating with others working on similar problems, particularly the ARC AGI challenge. Contact details and slides are shared.