Key Takeaways from the Discussion on Shifting from Models to Compound AI Systems
0:00:11 The Future is Compound Systems: The speaker emphasizes that while large language models (LLMs) get a lot of attention, the present and future of AI lie in compound systems.
0:00:40 Raise Consciousness: The purpose of the talk is to make listeners more aware of the critical nature of "systems".
0:01:04 Scaling is Key: The speaker acknowledges that scaling of models, which really kicked off with the gpt-3, has played a key part to where AI currently stands.
0:02:52 Core Lesson: Focus on what you DO with models, not the models, themselves.
0:02:57 LLMs on Disk are Useless: A downloaded LLM is inert until prompted and a sampling method is defined.
0:03:51 Choice of System Components Matters: The selection of prompt and sampling method are non-trivial choices that define the system's behavior. This is why many are giving LLMs web APIs, Databases, etc. The model is just one component to larger systems.
0:04:55 Blogging Post for More Info: The speaker references a blog post from February 18, 2024, titled "The shift from Models to compound AI systems". Sam Alman, CEO of openAI, was noted to echo it later on.
0:06:11 Analogy: Formula 1 Race Car: Designing an AI system is like designing an F1 race car - more than just the engine (the LLM) matters, it's also aerodynamics, friction, the driver, etc.
0:07:34 Prioritize System Components: A smaller model in a smart system is often better than a larger model in a simplistic system, especially considering cost, latency, safety, and privacy.
0:08:23 Safety and Regulations: Regulations should focus on entire systems, not just models, to avoid dangerous oversights and prevent stifling innovation.
0:09:14 Sampling for Generation: The method used for sampling is an under-considered component for which innovation abounds, and is absolutely critical to any "minimal AI system".
0:13:03 No One True Sampling Method: Choice of sampling method is highly consequential to an AI system, including but not limited to top P sampling, beam search, diversity methods, grammar adherence, etc.
0:13:38 Prompt Engineering is Key: Prompt engineering is the core of modern AI system development, but can be heartbreaking if not addressed correctly.
0:14:20 Zero-Shot Setting: Origins traced back to gpt2 which states that it can solve downstream tasks in a zero shot setting. The speaker admitted to overlooking this fact when this paper initially released.
0:16:38 In Context Learning: The paper uses instruction, followed by demonstrations, then finally, a target. In context, the model can learn to imitate that behavior and generate outputs in response.
0:17:31 Model Sensitivity to Prompts: Llama 27B's performance varied by 80 points based on the presence of colons in prompts, highlighting the dramatic impact of prompt formatting.
0:19:01 What is Optimal Prompt Model: Prompt the model for an optimal prompt model combination.
0:19:17 Chain of Thought: How Chain of Thought is framed effects outcome.
0:20:25 The Model and Prompt are Inextricably Linked: Tweets reveal model and prompt are intertwined and bettered through a series of tests and updates.
0:20:39 Model as Alien Creature: Language Model act as a sort of "alien creature". Apple's intelligence prompts show the amount of special pleading with the model for intended behavior.
0:22:37 The essence of AI seems to have been forgotten.
0:22:55 Important of Mod System Design/Datadriven Optimization/Generic Architecture: The importance of model system design, data-driven optimization and genetic architecture has been forgotten.
0:23:41 Dependence and the Tragic Story: Dependence is tragic since tested lessons has brought current AI as far as it is now.
0:24:03 Re-orient to language model programming and use programming libraries like the dspi.
0:24:34 Highlevel Tolls/DSP as Minimal System: DSP can have high level tools where it compiles a lot of what you are doing down to prompt language models.
0:26:28 The "Time-Tested" Lesson. the return that time test lesson of datadriven optimization.
0:30:08 Smaller Models are Gaining Attention: Small models are gaining popularity because they are faster.
0:30:43 You're going to also have to be as individuals in society just be on the look out for AI systems run.
0:30:55 Tool Access: Finally, we get to talking about Tool Access.
0:32:04 As thought, the question are what models will win vs what has the capabilities to access the most information?
0:34:05 Bill sb147 was vetoed by the governor to stop regulation with models of smaller size/what he thought was equally dangerous.
0:35:46 leaderboards around AI systems? That could be a re-oreintation around language models.
0:35:57 We aren't running races. we are running what can you do with all these complicated pieces.
0:37:07 You can think that over scaling for generation will lead to transformative and better models.
0:39:29 Think LMM then think about your system because it will make you more productive.
0:40:11 Recommendation for the listener's questions and agent's questions. Agent can become technique to access tools and bridge to help bridge language models for capabilities and design
0:42:34 I think open AI does not expose any of these traits in the current moment
0:45:57 It help with a guarantee that those uh assistance need to identify them as non-human when you are asking questions to the model.