Building and Improving AI Agents: An Inside Look at Arize CoPilot's Architecture and Iterative Development Process
0:00 Introduction: Arize's CoPilot is an AI assistant embedded in their platform to help AI engineers troubleshoot and improve their AI applications.
1:46 CoPilot Skills: CoPilot offers a range of skills, including prompt optimization, data curation with AI search, evaluation building, and root cause analysis.
3:51 CoPilot Architecture: The architecture involves multiple specialized "routers" (LLM-based) directing user questions to the most relevant skills, rather than a single overarching router. This approach improves efficiency and skill selection accuracy.
5:13 Specialized Routers: Routers are categorized by use case (e.g., tabular data, LLMs, prompt optimization) to streamline skill selection.
11:05 Router Selection: The choice of which router to use is currently determined by the user's location within the Arize platform (e.g., search bar, evaluation workflow). Future iterations may involve a “mega router” to handle routing more dynamically based on user input.
17:42 Deep Dive: AI Search Skill: The AI Search skill allows users to use natural language to filter and analyze data within the Arize platform.
18:27 AI Search Skill Initial Implementation (Column Search): The first version of AI Search could only perform semantic searches on a single column of data at a time.
20:29 AI Search Skill Iteration Cycle: The team uses a continuous iteration cycle for developing and improving skills, where production data informs development decisions and experiments.
28:57 Iterative Development Workflow: The workflow emphasizes a constant back-and-forth between production data analysis, evaluation, and development experiments to identify areas for improvement.
33:18 AI Search Skill Production Insights: Production data revealed that users desired broader search functionalities, prompting the development of AI Search V2.
34:04 AI Search Skill V2 Expansion: AI Search V2 expanded to include Table Search (searching across multiple columns), Text to Filter (translating natural language into structured filters), and Analysis (categorizing and segmenting data).
36:53 Rationale for Multiple Skills: The team opted to create separate skills rather than expanding existing ones to improve performance and focus for the LLM.
42:13 AI Search V2 Implementation: The team updated the Search Router to include the new skills and developed datasets for training and evaluating each skill.
42:57 Data Set Creation: Specialized data sets were created for each new skill, including a “mega dataset” encompassing all search types to evaluate the router’s skill selection capability.
44:19 Data Set Curation: The team utilized data editing tools to add ground truth labels to existing production data, streamlining the dataset creation process.
45:20 Experimentation and Evaluation: Experiments were conducted to evaluate the Search Router’s ability to select the correct skill based on user input and the performance of each individual skill.
46:48 Evaluation Metrics: Evaluators were used to assess the correctness of function calls (for the router) and the accuracy of the filters generated by the Text to Filter skill.
52:28 Deployment and Production Monitoring: After successful experimentation, AI Search V2 was deployed, and its performance is continuously monitored and evaluated in production.
57:22 Key Takeaway: The presentation highlights the importance of an iterative development process, leveraging production data for continuous improvement of AI agent skills, and using specialized routers and skills to optimize performance and user experience.