Summary of Semantic Text: Improving Exploratory Programming in Squeak with Generative AI
0:00:16 Introduction: The project, "Semantic Text," aims to enhance exploratory programming within the Squeak Smalltalk environment using generative AI (Large Language Models and Embedding Models), developed over approximately 1.5 years.
0:01:23 Core AI Concepts: Utilizes Large Language Models (LLMs) for text generation/conversation and Autonomous Agents for interacting with the system via function calls.
0:02:29 Embeddings for Semantics: Employs Text Embedding Models to convert text/code into vectors, representing semantic meaning and allowing for similarity comparisons and search.
0:04:23 Project Architecture: Built in layers, including connections to AI model providers (like OpenAI), domain models for conversations/agents within Squeak, and integrations into Squeak's standard tools.
0:05:15 Basic Chat Interface: A ChatGPT-like window integrated into Squeak allows users to converse with an LLM. Voice input/output is also supported (though microphone access was blocked during the demo).
0:07:11 Cost Awareness: The system includes an expense tracker to monitor the cost associated with using external AI APIs, as they are not free.
0:08:13 Agent Prototyping: An "Advanced Mode" in the chat UI allows developers to define system prompts and tools (Smalltalk functions) for AI agents, enabling rapid prototyping and debugging, even mocking tool results before implementation.
0:19:06 Class-Based Agent Development: Provides a framework for building more robust agents by subclassing SemanticAgent, defining prompts, and implementing tool methods directly in Smalltalk.
0:21:45 Agent Capabilities & Limitations: Demonstrates an agent executing Smalltalk code (e.g., World submorphs size) to answer questions, including automatic error correction and retries. Highlights the current limitations of models like GPT-4 in accurately understanding and writing complex Smalltalk syntax without guidance.
0:25:03 Live Code Generation: Shows the agent generating and compiling a new Smalltalk method directly into a class (String) based on a natural language request.
0:38:02 Semantic Search (Embeddings): Introduces a "Corpus" (simple vector database in Squeak) to store embeddings for code (e.g., methods).
0:40:01 Semantic Search Demo: Demonstrates finding semantically similar methods based on an existing method or a natural language query.
0:41:23 Retrieval-Augmented Generation (RAG): Combines semantic search with an agent to answer questions based on retrieved documentation or code, demonstrated by querying the Semantic Text package itself.
0:45:48 AI-Assisted Code Commenting & Refactoring: Shows using the AI to generate meaningful comments for existing code and refactor code based on natural language instructions (e.g., "use ifNil ifNotNil", "move returns out").
0:53:22 Tool Integration - Help Browser: Semantic search and summarization are integrated into the Squeak Help Browser, allowing natural language queries and displaying related topics.
0:56:06 Tool Integration - Squeak Inbox Talk: AI features added to the mailing list client include finding similar past conversations, summarizing long discussion threads, and chatting with the entire mailing list history using RAG.
1:01:28 Discussion on Knowledge & Training: Explores the challenges of teaching AI deep semantics vs. syntax, the stateless nature of current models, the potential and cost of fine-tuning vs. building custom models, and leveraging existing knowledge within models.
1:13:35 Exploratory Programming Agent: Introduces a core concept where the user can "talk" to Smalltalk objects directly in natural language.
1:14:03 Inspector Integration: Demonstrates asking questions about an object (e.g., "When was this order created?") directly within the Squeak inspector, with the agent interacting with the object to find the answer.
1:16:13 Object Manipulation via Language: Shows instructing objects to change their state or appearance using natural language commands within the inspector (e.g., "make all digits red", "rotate them all").
1:17:15 Direct Language Invocation: The agent can be invoked directly from Smalltalk code (self ask: '...').
1:22:41 Debugger Integration: An experimental feature allows chatting with the debugger context to understand the call stack or variable states.
1:23:48 Time-Travel Debugger Integration: Combines the agent with a trace debugger, enabling natural language questions about the past or future state of variables during program execution.
1:26:31 Availability: The "Semantic Text" framework and the "Exploratory Programming Agent" are available on GitHub, compatible with recent Squeak versions (6.0, 6.1 Alpha/Trunk). Further details are in papers, a thesis, and mailing list posts.