Okay, here is the abstract and summary for the provided transcript about Priceline's Penny AI assistant.
Abstract:
This presentation features Brooks Stevens (Director of Software Engineering) and Brio (Staff Software Engineer) from Priceline detailing the development journey of real-time voice capabilities for their AI travel assistant, Penny. Initially a text-based chatbot and later enhanced with non-real-time voice (record-transcribe-generate-synthesize), Priceline sought a more natural, faster, and interactive user experience. They implemented real-time voice using OpenAI's endpoint, building an event-driven system with websockets connecting the client-side chatbot, a backend orchestration service, and the OpenAI API. Key technical aspects discussed include managing dual websocket connections, handling audio input/output events, implementing function calling for business actions (like hotel searches with UI components), context management, voice activity detection for interruptions, and crucially, output-based content moderation. The team shares challenges related to latency, websocket complexity, audio processing for noise reduction, and ensuring correct event timing. They highlight the significant role of the Arise platform for observability, debugging, and performance evaluation, detailing their evolution from session-based to transaction-based telemetry for more accurate insights. The goal is a seamless voice interface that enhances the travel booking and servicing experience.
Building Penny's Voice: Priceline's Journey to Real-Time AI Interaction
0:01:05 Who is Penny: Priceline's customer-facing AI travel assistant, evolving from answering itinerary questions to having broader skills, aiming to make travel deals accessible.
0:02:26 Why Add Voice: Voice offers a natural, intuitive, hands-free interface, potentially faster and more enjoyable, aligning with the goal of seamless travel booking and servicing.
0:03:38 Initial Non-Real-time Voice: The first voice implementation involved recording user audio, sending it to OpenAI Whisper for transcription, getting a text response via the completions API, converting the text back to speech, and then playing it. This worked but suffered from noticeable latency.
0:05:04 Transition to Real-time Voice: Motivated by the desire for less friction (key in e-commerce), a better user experience, and potentially higher conversions, the team moved to implement real-time voice.
0:05:43 Real-time Demo: A demonstration showcases Penny responding quickly to a hotel search request and handling user interruption smoothly.
0:07:29 Real-time System Architecture: Uses websockets connecting the client (React-based chatbot) to a backend orchestration service, which in turn connects to the OpenAI real-time endpoint. The backend manages context, event handling, and communication flow.
0:09:29 Development Timeline: Penny's initial version launched in Q2/Q3 2023, with non-real-time voice added shortly after OpenAI's Whisper/Speech endpoints were available. Real-time capabilities followed later.
0:11:45 Core Real-time Functionality: Key goals included supporting function calls (for UI display and business logic), natural conversation flow, interruption handling (Voice Activity Detection), context awareness within the booking funnel, content moderation, and conversation persistence.
0:14:27 Function Calling for UI & Actions: The system uses function calling not just for backend actions but also to return descriptors for UI elements (like hotel cards) to be displayed in the chat interface, even during voice interactions.
0:16:17 How Real-time Events Work: The system is event-driven over websockets. The client streams audio; OpenAI sends back various events (session created, speech started/stopped, response generated, etc.) that the backend service reacts to.
0:20:48 Handling Audio Challenges: Dealing with incorrect speech recognition or background noise required adding frontend audio processing (high/low pass filters). Device variability also presented challenges.
0:22:46 Websocket Management: Managing two separate websocket connections (Client <-> Backend, Backend <-> OpenAI) was a learning curve, requiring retry logic and careful handling. DevOps involvement is recommended for teams new to websockets.
0:25:30 Content Output Moderation: Implemented on the output from the LLM due to the speed requirements of real-time interaction. If potentially problematic content is detected, a separate LLM call assesses compliance before it reaches the user.
0:29:40 Importance of Observability (Arise): The Arise platform proved crucial for development, debugging, and understanding system behavior, being much easier than parsing logs, especially for complex event-driven flows.
0:30:13 Telemetry Evolution: Their initial Arise implementation used long-lived, session-based spans, leading to inaccurate latency metrics.
0:31:43 Improved Transaction-Based Telemetry: They are moving to a transaction-based approach (tying user turn + AI response together via session ID), recommended by Arise, which provides much clearer insights and facilitates better evaluation.
0:33:00 Evaluation and Future Plans: Currently using LLM-based evals for moderation. Future plans include leveraging Arise's ability to link directly to audio recordings for more sophisticated audio-based evaluations.