To review this material, the most appropriate group would be Senior AI Research Scientists and Principal Machine Learning Engineers specializing in Program Synthesis, Human-Computer Interaction (HCI), and Large Language Model (LLM) reliability.
As a Senior AI Research Scientist, I view this keynote as a seminal roadmap for moving beyond "black-box" generative AI toward robust, neuro-symbolic systems. The focus shifts from simple next-token prediction to a structured framework of Intent, Interaction, and Inspection to solve the fundamental problem of LLM hallucination and reasoning fragility.
Abstract:
In this KDD '25 keynote, Sumit Gulwani of Microsoft Research outlines a multi-pillar framework to enhance AI reasoning, particularly in structured domains like programming and data analysis. Moving beyond the limitations of Large Language Models (LLMs)—which often struggle with execution semantics and exact logic—Gulwani advocates for a "Neuro-Symbolic" approach.
The strategy is built on three core pillars: Intent, leveraging input-output examples and temporal context to define user goals precisely; Interaction, employing agentic patterns like "investigate-and-respond" and evaluating dialogue quality via Gricean maxims; and Inspection, using programmatic validators and symbolic checks to catch hallucinations and guide iterative refinement. The talk demonstrates how these techniques, integrated with a fourth emerging pillar—Memory—transform AI from a reactive tool into a collaborative, evolving partner capable of discovering new algorithms and empowering users through personalized, reliable automation.
Improving AI Reasoning: Intent, Interaction, and Inspection
0:00 Program Synthesis Origins: The speaker traces the evolution of AI reasoning from early symbolic program verification and matrix multiplication optimization to modern automated synthesis.
3:46 Input-Output Examples as Intent: Input-output examples are highlighted as the only form of specification that can be easily validated. This was the foundation for Flash Fill in Excel.
6:13 Domain-Specific Languages (DSL): To ensure efficiency and correctness, the system restricts search to a DSL rather than a general-purpose language, using symbolic back-propagation to prune the search space.
13:53 LLMs and Execution Semantics: LLMs often fail at exact logic (e.g., extracting specific character counts) because they lack deep understanding of execution. A "trick" is introduced to force the LLM to execute statements step-by-step to maintain exactness.
19:18 Distinguishing Inputs: When multiple programs satisfy an example, the system generates "distinguishing inputs" to ask the user for clarification on corner cases, significantly improving convergence to the correct logic.
20:01 Temporal Context (IntelliCode): Beyond explicit examples, AI can infer intent from a user’s recent actions. IntelliCode in Visual Studio uses this "temporal context" to suggest repetitive code edits across a codebase.
28:37 The "Investigate-and-Respond" Pattern: A major breakthrough in AI debugging success (from 25% to 88%) was achieved by programming agents to investigate the root cause and ask clarifying questions rather than immediately leaping to a (potentially incorrect) fix.
38:15 Gricean Maxims for Interaction: Effective AI communication is measured against Paul Grice’s maxims: Quantity, Quality, Relevance, and Manner. Domain-specific rubrics are necessary to evaluate whether a conversation is actually helpful.
44:48 Persona-Specific User Proxies: To evaluate agents at scale, researchers use "user proxies" grounded in specific personas (e.g., a student vs. a business analyst) to simulate diverse interaction styles and needs.
46:52 Handling Ambiguity with Placeholders: Rather than hallucinating a value, models can be trained using a modified loss function to "abstain" and insert placeholders (e.g., "?") where user input is required.
51:46 Neuro-Symbolic Inspection: Robustness is improved by a feedback loop where an LLM generates an output, a symbolic checker identifies syntactic or semantic errors, and a critique LLM provides actionable feedback for a retry.
59:47 The Memory Pillar: The final frontier for AI reasoning is "Memory." Without it, every interaction is a "first date." Memory allows for personalization, learning user preferences, and evolving the AI-human relationship over time.