This tutorial demonstrates how to perform session-level evaluations on conversations with an AI tutor chatbot using the Arize AI platform and its Python SDK. It differentiates session-level evaluations from typical span-level evaluations, highlighting their importance for gaining a holistic view of user interactions to assess broader patterns like user frustration, context retention, and goal completion across multi-turn dialogues. The video walks through collecting traces and spans into structured sessions using session and user IDs, building a sample AI tutor, running conversational sessions, and then evaluating these sessions across multiple dimensions such as correctness, frustration, and goal achievement using LLM-as-a-judge techniques. Finally, it covers logging these evaluation results back to Arize for monitoring and analysis within the UI.
Summarizing Session-Level Evaluations for AI Chatbots with Arize AI
0:00 Introduction to Session-Level Evaluations: The video introduces session-level evaluations in Arize using the Python SDK, contrasting them with standard span-level evaluations which focus on isolated actions like tool calls or LLM responses.
0:44 Defining a Session: A session is defined as a holistic, multi-turn conversation or related interaction, grouping individual traces and spans together using a session_id. This allows for a broader view of user experience beyond single actions.
1:20 Illustrative Example of a Session: An example of a customer service chatbot conversation (e.g., inquiring about a truck purchase and comparison) is used to explain how multiple human-chatbot exchanges form distinct traces, all linked by a shared session ID to create a complete session.
2:34 Sessions in Arize UI: Arize automatically collects and concatenates traces and spans into sessions, which are then visually represented in the Arize UI under the sessions tab, displaying the full conversation flow.
3:35 Benefits of Session-Level Evaluations: Evaluating at the session level unlocks the ability to answer higher-level questions such as "Did the user reach their goal?", "Was the chatbot consistently accurate throughout the conversation?", or "Was the user frustrated by the end?", which are not possible with isolated span evaluations.
4:08 Grouping Traces with Session/User IDs: Traces can be grouped into sessions using either a session_id or a user_id. The tutorial demonstrates the using_attributes context manager from Open Inference instrumentation, which allows for linking spans to sessions using both IDs (e.g., student name as user_id).
5:25 Building and Running the AI Tutor: A Python code walkthrough shows how to set up tracing, configure an AI tutor (using Anthropic models), and generate conversational sessions by iterating through user inputs until a "done" command, ensuring each turn is logged as part of the ongoing session.
8:47 Preparing Spans for Evaluation: Spans generated by the AI tutor are exported from Arize into a Pandas DataFrame. The data is then processed to collapse spans into single rows per session, separating user inputs from AI outputs and sorting them chronologically for proper evaluation.
10:51 Session Correctness Evaluation: The first evaluation type, "session correctness," assesses the factual accuracy and educational quality of the AI tutor's responses throughout the conversation. It uses an LLM-as-a-judge (LM classify) with defined criteria and "correct"/"incorrect" rails.
12:25 Session Frustration Evaluation: The "session frustration" evaluation determines if the student expressed frustration during the interaction, based on predefined criteria (e.g., "I don't get it," disagreements). This is a generalizable evaluation for user experience.
13:09 Session Goal Achievement Evaluation: The "session goal achievement" evaluation assesses whether the AI tutor successfully answered the student's questions and if the student gained the proper knowledge by the end of the session, classifying outcomes as "achieved" or "not achieved."
14:03 Logging Evaluations Back to Arize: All computed evaluation results (labels and explanations) are parsed and merged, then attached to the "root span" (the first span of the first trace) of each session. These are then logged back to the Arize platform for centralized monitoring and analysis.
15:27 Viewing Results in Arize UI: The video concludes by demonstrating how the logged session-level evaluations appear directly within the Arize UI's sessions tab, enabling users to click into sessions, filter by evaluation results, and use the data for iterative improvements and troubleshooting of their AI applications.