Dasha Metropolitansky, a research data scientist at Microsoft, introduces Claimify, an advanced system designed for claim extraction. Claimify defines a "claim" as a simple, factual statement that can be verified as true or false, and its core function is to break down complex texts into these atomic units. This process is presented as crucial for evaluating the quality of long-form content generated by large language models, particularly for applications like hallucination detection, assessing relevance, and measuring comprehensiveness. Claimify employs a sophisticated methodology: it first processes text sentence by sentence, incorporating surrounding contextual information. Then, it applies a three-stage pipeline consisting of selection (filtering out non-factual or opinion-based sentences), disambiguation (identifying and attempting to resolve ambiguities using context, or flagging them if unresolvable), and decomposition (breaking down the clarified sentences into individual, verifiable claims). The presentation emphasizes Claimify's ability to handle nuanced language, resolve ambiguities, and provide more accurate and detailed claim extraction compared to simpler baseline methods, illustrated with examples from various contexts.
Understanding Claimify: Microsoft's Approach to Claim Extraction for LLM Evaluation
00:00:05 Introduction to Claimify: Dasha Metropolitansky from Microsoft's special projects resilience team introduces Claimify, a system developed for claim extraction.
A "claim" is defined as a simple factual statement that can be verified as true or false.
"Extraction" is the process of breaking down a text into these individual claims.
Key properties of extracted claims include removing subjective language (e.g., "notable") and ensuring claims are simple, independent statements.
00:01:09 Core Function of Claimify: The system takes text of any length and decomposes it into high-quality, standalone factual claims.
00:01:20 Applications in Evaluating LLM-Generated Content:
Hallucination Detection (00:01:23): Claimify helps verify if answers from language models are based on source documents by breaking down potentially long answers into a set of easily checkable factual statements.
Relevance Assessment (00:01:56): By breaking answers into individual claims, their relevance to a specific question can be assessed independently and then aggregated.
Comprehensiveness Measurement (00:02:20): The number of extracted claims from an answer can serve as a proxy for its comprehensiveness.
00:02:34 Claimify's Methodology: The system avoids processing entire texts at once, instead:
It breaks down text into sentences and processes each independently, including a "window of text around the sentence" as context for accurate interpretation.
Claim extraction is a three-part process:
1. Selection (00:02:57): Filters out sentences that do not contain verifiable claims, such as opinions (e.g., "companies should embrace AI").
2. Disambiguation (00:03:14): Detects ambiguity (multiple plausible interpretations) within a sentence. It then determines if the ambiguity can be resolved using the provided context or flags it if it cannot. This is highlighted as a unique and powerful feature.
3. Decomposition (00:03:42): Takes the disambiguated sentence and breaks it down into simple, standalone factual statements (claims).
00:03:49 Claimify in Action - Comparative Examples:
Handling Ambiguity (00:04:14): In a sentence where the UN reported illness and "highlighting the need for improved water management," Claimify identified the ambiguity of whether the "highlighting" was by the UN or the author. As context didn't clarify, it flagged the sentence as "cannot be disambiguated" for that part, unlike a baseline model that might ignore it.
For statements about Argentina's inflation, Claimify extracted claims about economic hardship and specific rate predictions (>300%) that a baseline missed. It also specified "inflation has depreciated the currency" rather than a vaguer "currency value has plunged."
When a sentence referred to challenges in Afghanistan and Sudan as similar to "those of Libya" (00:05:23), Claimify used context (public health crises, flooding) to make specific claims about these issues, while a baseline left "those" unspecified.
Regarding Nigeria's wheat production (00:05:41), Claimify precisely captured that "Nigeria's efforts to become self-sufficient in wheat production" were hindered, not just "wheat production" itself.
00:05:58 Significance of Claimify: The system is presented as a powerful tool to facilitate the difficult task of evaluating the quality of long-form content generated by language models by breaking it into verifiable, atomic claims.