Alena, a data streaming expert at Confluent, presents a fun project that combines her professional expertise with her favorite game, Stardew Valley. The talk details the creation of a simulated village where game characters, powered by Large Language Models (LLMs), move, interact, and converse autonomously. Inspired by recent research in generative agents and LLM-driven simulations, the project leverages Apache Kafka for real-time data streaming, Apache Flink for real-time data transformation (specifically, proximity detection), OpenAI for generating character dialogue, and Retrieval Augmented Generation (RAG) with vector search (using OpenSearch) for long-term memory management and contextual conversations. Alena demonstrates how character identities are formed and how goals, like organizing a conference, can propagate through the simulated community, highlighting challenges encountered and solutions implemented in building complex, interactive AI agents.
Simulating Stardew Valley Characters with LLM Agents and Streaming Data
0:00 Introduction & Motivation: Alena, working with streaming data at Confluent, combines her professional skills with her favorite game, Stardew Valley, to create a fun demo simulating game characters interacting.
1:18 Core Technologies: The project utilizes Apache Kafka for streaming data, Apache Flink for data processing, OpenAI for LLM interactions, and RAG with vector search for memory.
1:37 Inspiration - Generative Agents Paper: Alena's work is heavily inspired by the "Generative Agents" paper (Stanford/Google), which created a village of AI agents that form identities, remember events, reflect, plan, and act autonomously. A key example is an agent successfully organizing a party by spreading the idea.
4:44 Thinking Process of Agents: Agents perceive their environment, store memories in a memory stream, reflect on past events, plan their actions (which can adapt to interactions), and then act. Memory retrieval prioritizes recency, importance, and relevance.
10:17 Early Experiments: Alena applied these concepts to a mafia game simulation, observing how initial statements by "mafia" agents could strongly influence the game's outcome due to LLM influence. Another demo involved alien characters using pub/sub (Kafka) and OpenSearch for long-term memory.
12:44 Addressing Context Window Limitations: To overcome limited LLM context windows, Alena used semantic search (vector search) to retrieve only the most relevant past conversations for future interactions.
14:08 Human Emulation: Another influential paper demonstrated the ability to create LLM agents based on human interviews, with their responses closely matching those of real humans.
15:27 Other LLM Agent Implementations: Mentions Voyager (self-teaching LLMs in Minecraft), Open City (scaling to thousands of agents using distilled prompting), and Tiny World (Microsoft's library for agent collaboration and brainstorming).
17:43 Latest Research: Highlights recent papers on different types of memory (long-term, short-term, forgetting) and reasoning models that use chain-of-thought processes and external searches to gather information.
19:28 Stardew Valley Implementation - Characters & Map: Used existing Stardew Valley characters, their backstories, and sprites. The browser's animation frame was used to trigger character movement.
21:39 Kafka for Location Streaming: Each villager constantly sends their real-time location (coordinates and timestamp) to a Kafka topic.
24:21 Flink for Proximity Calculation: Apache Flink processes these real-time location streams, quickly calculating distances between villagers and emitting "close" events when they are near enough to each other.
26:56 LLM-Powered Conversations: When villagers are close, the system initiates conversations using a large language model (OpenAI). Messages are exchanged via Kafka.
27:34 Conversation Management Challenges: Initially, characters talked too much; solutions included adding cooldowns (10-second wait) and aborting conversations if characters moved away. Also, restricted conversations to only two villagers at a time to avoid messy circular dialogues.
28:54 Prompt Engineering: Prompts were refined to ensure characters replied concisely, in character, avoided small talk, and provided only the response without meta-comments.
30:04 Retrieval Augmented Generation (RAG): RAG is used to inject additional context (from external sources like databases or APIs) into LLM prompts.
31:07 Similarity Search: Conversation summaries are transformed into vectors (embeddings) and stored in OpenSearch. When a new conversation starts, relevant past memories are retrieved based on vector similarity and provided to the LLM.
33:22 Goal Propagation Demo: Judy was given a goal to organize a conference. Prompting encouraged her to spread the word. The demo successfully showed the idea spreading through the village, with other characters remembering and discussing it, even if the "conference" topic slightly shifted (e.g., to "farming conference").
36:18 Conclusion & Tools: The demo showcases the combined power of Kafka, Flink, OpenAI, and vector search for creating dynamic and interactive AI agent simulations.
36:35 Resources & Certification: Links to project materials are provided, along with information about a free data streaming engineer certification for Kafka and Flink.