This tutorial provides a comprehensive walkthrough on performing session-level evaluations for an AI tutor chatbot using the Arize platform's Python SDK. The video begins by defining a "session" as a complete, multi-turn conversation between a user and an AI, distinguishing it from isolated "spans" (individual interactions). It emphasizes that session-level analysis is crucial for evaluating holistic metrics like user frustration, goal completion, and overall contextual correctness, which are not visible at the span level. The presenter demonstrates a practical workflow, starting with instrumenting an AI tutor to generate conversational sessions with unique session and user IDs. The tutorial then covers exporting this data from Arize, preparing it by aggregating spans into structured session dataframes, and running a series of LLM-as-a-judge evaluations (using Phoenix evals) to assess correctness, frustration, and goal achievement. Finally, it shows how to format and log these evaluation results back to Arize, attaching them to the root span of each session for powerful monitoring and analysis within the Arize UI.
Evaluating AI Tutor Performance: A Guide to Session-Level Analysis with Arize
0:00 Introduction to Session-Level Evaluations: A session is a complete user conversation (a group of related traces), unlike a span, which is a single interaction. Evaluating at the session level provides a holistic view, enabling assessment of high-level metrics like user frustration and goal completion that cannot be captured from isolated spans.
4:08 Grouping Traces into Sessions: The tutorial uses the using_attributes context manager from openinference to group traces. By passing a consistent session_id and user_id, all spans generated within a single conversation are linked together, forming a coherent session.
5:51 Generating Sample Conversations: An AI tutor chatbot is built and run to generate several multi-turn conversations. This process creates the raw session data that will be used for evaluation later in the tutorial.
8:47 Preparing Spans for Evaluation: The generated span data is exported from Arize into a pandas DataFrame. A script is then used to process this data by grouping spans by their session_id, separating user messages from the AI's responses, and sorting them chronologically to reconstruct the conversation.
10:55 Implementing LLM-as-a-Judge Evaluations: The tutorial uses Phoenix evals to run three distinct session-level evaluations, where an LLM acts as a judge based on custom prompt templates:
11:01 Session Correctness: Evaluates if the AI tutor provided factually and educationally sound information throughout the entire conversation.
12:27 Session Frustration: Assesses the user's messages for signs of frustration, such as repeated questions or expressions of confusion.
13:10 Session Goal Achievement: Determines whether the AI tutor successfully helped the student achieve their learning objective by the end of the conversation.
14:03 Logging Evaluations Back to Arize: After the evaluations are run, the results (labels and explanations) are formatted and merged with the session data. They are then logged back to the Arize platform by attaching them to the root span of each corresponding session.
15:30 Visualizing Results in the Arize UI: The tutorial concludes by showing the "Sessions" tab in the Arize UI, where the newly logged evaluations appear. This centralized view allows for powerful monitoring, filtering, and analysis of chatbot performance at a conversational level.