This tutorial demonstrates how to perform session-level evaluations on conversations with an AI tutor chatbot using Arize AI's Python SDK. Unlike traditional span-level evaluations that assess isolated interactions, session-level evaluations provide a holistic view of entire multi-turn dialogues, enabling the assessment of broader patterns such as user goal achievement, frustration, and overall conversational correctness. The video outlines the process of tracing agent conversations, aggregating individual spans into complete sessions using unique session or user IDs, and then evaluating these sessions across multiple dimensions. It also covers how to format and log these evaluation results back to the Arize platform for comprehensive monitoring and analysis of AI system performance and user experience.
Summarizing Session Evaluation On An AI Tutor Chatbot
0:00 Introduction to Session-Level Evaluations: The video introduces session-level evaluations, contrasting them with span-level evaluations. While span-level focuses on isolated elements (e.g., tool call correctness, LLM hallucination), session-level provides a holistic view of multi-turn conversations, crucial for assessing user experience.
0:44 Defining a Session: A session is defined as a group of related traces, connected by a session ID, encompassing an entire user-app dialogue. This allows for a comprehensive understanding beyond individual messages, illustrated with a customer service chatbot example.
1:20 Session Visualization and Benefits: An example diagram of a human-chatbot interaction for purchasing a truck clarifies how multiple "traces" (human-chatbot exchanges, including internal LLM/tool calls) are grouped under a single session ID. This holistic view unlocks the ability to measure overall user goal achievement, context retention, and user frustration—insights not possible with span-level data.
3:08 Viewing Sessions in Arize UI: The Arize UI provides a dedicated "Sessions" tab where entire conversations, with all traces and messages, can be visually reviewed for better understanding and debugging.
4:08 Grouping Traces for Sessions: The tutorial explains how to group traces into sessions using either a session ID, a user ID, or both via Arize's using_attributes context manager from openinference.instrumentation. This allows for flexible tracking of user journeys.
5:27 Building and Running the AI Tutor: A Python function run_session is demonstrated, which simulates an AI tutor conversation. It uses the using_attributes context manager to tie all generated spans to a specific session and user ID, effectively creating conversation sessions.
7:13 Interactive AI Tutor Example: A live demonstration shows a student interacting with the AI math tutor. The tutor asks questions, provides hints, and the conversation continues until the student types "DONE," signaling the session's end and illustrating the multi-turn nature.
8:47 Preparing Spans for Evaluation: The first step for evaluation is to export all relevant spans from the Arize platform into a Pandas DataFrame using the ArizeExportClient.
9:39 Aggregating Spans into Sessions: A prepare_sessions function is used to group these raw spans by their session_id, separating user inputs and AI outputs into ordered lists. This function also includes logic for truncating long sessions to prevent context window issues for evaluation models.
11:01 Session Correctness Evaluation: The process of evaluating session correctness is shown using llm_classify (LLM as a judge) with a specific prompt and an Anthropic model. This evaluates the factual accuracy and educational quality of the AI tutor's responses throughout the entire session.
12:27 Session Frustration Evaluation: Another evaluation dimension, student frustration, is demonstrated. The prompt defines signs of frustration, and the LLM classifies sessions as "frustrated" or "not_frustrated," highlighting a key metric for user experience in conversational AI.
13:10 Session Goal Achievement Evaluation: The final evaluation focuses on whether the AI tutor successfully helped the student achieve their learning goals. The prompt outlines criteria like directness, problem resolution, and logical progression, classifying sessions as "achieved" or "not_achieved."
14:03 Logging Evaluations Back to Arize: After running evaluations in code, the results (labels and explanations) are formatted and merged back into the session DataFrame. They are then logged to Arize using arize_client.log_evaluations_sync, attached to the root span of each session for centralized monitoring.
15:30 Viewing Results in Arize UI: The final step shows how the logged evaluation results appear directly within the Arize UI's "Sessions" tab, providing clear visibility into session performance metrics. Users can filter, analyze, and iterate on their AI tutor based on these holistic evaluation insights.