This presentation by McKenzie Leroy, Lead Data Scientist at S&P Global, details a comprehensive framework for evaluating Generative AI and Large Language Model (LLM) solutions, particularly Retrieval Augmented Generation (RAG) models. Using their "MI Sales Assistant" as a case study – a tool providing sales teams instant answers from federated internal documentation – he highlights the critical need for accurate evaluation due to high business impact and the inherent challenges of language. The framework leverages LLMs as "judges" to measure metrics like correctness, relevance, similarity, and hallucination detection. It emphasizes a hybrid approach to generating ground truth data, combining LLM-generated questions with subject matter expert (SME) verification and targeted SME-generated questions. The technical implementation utilizes MLFlow for experimentation and AWS CloudWatch for daily monitoring, significantly accelerating document deployment and improving internal knowledge base accuracy by identifying conflicting information.
Summarizing the Transcript: Evaluating Generative AI Solutions
0:05 Introduction to Speaker and Topic: McKenzie Leroy, Lead Data Scientist at S&P Global, discusses the evaluation of AI-powered productivity solutions, specifically Generative AI and Large Language Model (LLM) solutions like Retrieval Augmented Generation (RAG).
1:22 The MI Sales Assistant Tool: The primary example is S&P Global's MI Sales Assistant, a RAG tool for non-technical teams (HR, finance, sales). It centralizes previously federated internal documentation to provide instant answers on topics like client engagement, Salesforce data entry, and commission structures.
3:56 High Stakes and Risks: The tool deals with critical information (legal, policy documents), meaning answers must be highly accurate. Even minor errors carry significant business impact, leading to a "zero tolerance for risk" evaluation approach.
Business Impact: Errors can have significant consequences.
Language Complexity: Evaluating language is difficult due to infinite ways to convey the same meaning.
RAG Specifics: Evaluation must consider both the retriever (did it find the correct documents?) and the generator (did the model produce a good answer from those documents?).
Relative Truth: Correctness is context-dependent (e.g., S&P brand colors vs. general advice).
Dynamic Truth: Policies change over time, requiring continuous updates and re-evaluation.
Expertise Gap: The development team often lacks the subject matter expertise to verify complex outputs.
Retrieval Step (Traditional ML): Uses standard classification metrics like precision, recall, F1 score, and accuracy.
Generation Step (LLM as a Judge): Employs an LLM to rate outputs based on:
09:07 Accuracy/Correctness: Factual accuracy against source material.
10:03 Relevance: Whether the response directly addresses the question.
10:53 Similarity: How closely the response's meaning matches reference answers, including structural alignment (e.g., maintaining discrete steps in a process).
11:52 Hallucination Detection: Identifying information generated that is not present in the source materials (often subtle and plausible-sounding).
Hybrid Approach: Combines LLM-generated questions (for volume) with validation and specific question generation by Subject Matter Experts (SMEs) (for quality and complexity).
This reduces the burden on SMEs by over 70%, as they primarily check rather than generate.
Important Lessons: Crucial to validate LLM-generated questions, vary models/prompts (to counteract LLMs favoring their own output), and mix simple with complex questions.
Uses MLFlow, an open-source package, for running evaluation experiments.
Chosen for its open-source nature, AWS partnership (SageMaker integration), cost-efficiency, scalability, and customization capabilities (e.g., custom metrics like "did the model cite its document?").
Daily evaluations are run, with retrieval steps taking seconds and generation steps involving LLM calls taking longer.
Metrics are pushed to AWS CloudWatch for tracking performance over time.
Comprehensive Testing: Performed when fundamental changes occur (model, prompt, large document additions/subtractions) to establish a baseline.
Daily Testing: 20-25 stratified samples are run daily to ensure the system maintains expected output and detect early warning signs of performance drift.
Faster Problem Detection: Issues are identified proactively, often during user acceptance testing, boosting stakeholder confidence.
Accelerated Deployment: Onboarding new document sets has been reduced from over a month to less than two weeks.
Reduced SME Burden: SMEs spend over 70% less time on question/answer generation.
Knowledge Base Improvement: Unexpectedly, the framework identified conflicting or outdated documents across different teams, leading to significant improvements in the company's knowledge base.