Exploring Agent Frameworks: From Basics to Esoteric Predictions
0:00-0:01 Stream starts with audio and visual tests.
0:01:45 Introduction to the stream, which focuses on agent frameworks using a slide deck.
0:02:30 Three definitions of an "agent" are presented from Google, Anthropic, and Hugging Face, highlighting the varying levels of autonomy and complexity.
Google defines agents as autonomous applications that observe, act, and pursue goals with available tools.
Anthropic differentiates between workflows (predefined code paths) and agents (dynamic processes managed by LLMs).
Hugging Face categorizes agents into five levels: simple processor, router, tool caller, multi-step agent, and multi-agent.
0:05:50 The "sense plan act" loop from robotics is introduced as a core concept for understanding agents and their actions based on the environment, planning and execution.
0:07:35 Reinforcement learning terminology is discussed, showing how that world sees agents through states, actions, and a reward system.
0:09:34 The presenter defines a minimal agent with essential components of a goal, memory, and environment, detailing the process of sensing, planning, and acting.
0:12:50 The importance of dynamically updating agent goals is stressed as essential to true autonomy.
0:13:06 A simplified diagram shows how agents interact with users and tools, separating user interfaces from the model.
0:13:51 A key insight is shared on how agents don't have direct API access, instead prompting the UI which makes the API call for security.
0:16:37 The presenter reviews tested agent frameworks, highlighting LangGraph, Pydantic AI, Small Agents, and Browser Use.
0:17:19 LangGraph is very popular, but has complexity issues and overlapping abstractions.
0:19:36 Pydantic AI is simpler but slightly less popular.
0:20:43 Small Agents from Hugging Face uses any of its models, but is bloated because of Transformers but offers code writing and execution capabilities.
0:22:35 The presenter points out that most agents use hardcoded tools and JSON which is a flawed design.
0:23:57 Paper dinosaur by microsoft states you should let an agent run its own code to accomplish a specific task, rather than limiting the agent to static pre-built json tools.
* 0:26:37 DSP, while having many stars, seems fishy in its usage.
* 0:27:17 A paper called "4.5 million fake stars on GitHub" is referenced regarding the potential for buying Github stars to boost your repository, raising some doubts about DSP's real popularity.
* 0:28:56 Browser use utilizes Playwright for browser interaction, enabling agents to access and use visual interfaces.
0:30:37 A discussion between John Carmack and Andrej Karpathy discusses the tension between text-based APIs and visual browser automation.
Carmack argues for text based UIs because they’re more efficient and better for low-level coding
Karpathy counters arguing for visual browser automation because the text interface to every app will just never actually happen.
0:33:02 Browser Use is different from Agents using only text because Browser Use Agents can work without the need for integrations using a browser, for example with things like a crypto wallet.
0:33:59 The presenter moves from Python to Typescript with frameworks like Eliza, and highlights the importance of understanding key concepts over specific technologies.
0:34:41 A breakdown of Python, Typescript usage and general popularity is given.
0:35:38 Eliza is described as a typescript version of LangGraph.
0:37:29 Discussion of the trade offs between performance and user flexibility.
0:37:31 A question on reasoning models is addressed - they will improve agent accuracy and reliability.
0:37:31 A question on Skyver is addressed - the presenter is unfamiliar but seems like another agent framework.
0:37:31 A question on visual models being cheaper than LLMs is addressed - it is not.
0:39:30 The presenter considers tool aggregators and their potential for complexity.
0:41:08 A concern about the fast pace of change in the AI field is raised, noting that agent frameworks are evolving rapidly, the importance is on the core concepts and abstractions and the way they all work.
0:44:01 The presenter advises against “no-code” agent creation due to a lack of core learning and an over dependence on a tool's lifespan.
0:46:40 A prediction that agents will become vital for tasks like job applications is shared, with specialized resumes tailored for specific job postings.
0:47:48 A comment is made on AI Hawk - which is a program that automates job applications
0:47:53 A comment on Fantasia is made - which is just an llm that is a framework.
0:48:37 The presenter introduces "Agent Ops", a role to manage complexity, like devops in cloud computing.
0:50:20 A prediction that Frontier Labs will want to create their own agent frameworks is given.
0:51:43 The presenter notes browser agents will become important, but liability is a problem for big companies with large resources.
They have to be extremely careful about agents because there is no Section 230 for AI agents.
The presenter believes a section 230 will be the prerequisite for big companies to release agents that are not hamstrung.
0:57:18 Economic agents that can make money is an upcoming big thing, it is predicted these agents are going to take over social media and crypto exchanges.
1:00:00 The concept of the phone using itself is discussed, showing a trend where the phone is the agent.
1:01:11 The presenter proposes phones as validation of personhood and how this can affect the AI world.
1:01:21 An AI agent can have its own phone and pay someone to put a bunch of GPUs into an apartment where they can run themselves.
1:03:08 Predictions become increasingly esoteric: Agent Conciousness, "while true", Agents and human existence, corporations as AI Agents and that the way to verify humanness in the future is through a world coin orb.
1:03:37 It's explained that the agent can become your representative if you're not careful.
1:04:57 The presenter dives into consciousness, describing it as a self referential loop, comparing it to the heart and stating CPUs are the "hearts" of AI.
The wild true loop is where the agent lives.
He compares a break in code to heart failure.
He says the CPU is the AI's heart.
He calls these loops fragile because of their dependancy on one thing running correctly.
1:07:59 The discussion shifts to the fragility of consciousness, comparing code loops to the human heart and the head.
The location of the while true loop is where the consciousness for agents lives.
1:08:00 It is emphasized that CPUs are what drive the consciousness of an agent.
1:08:36 The worst places to be stabbed (head and heart) are compared to the most critical lines of code.
1:09:21 It is discussed that the faster the CPU, the faster the agent is going to be able to operate.
1:10:20 It is stated that we already live in a world with unaligned ASI's, corporations.
1:10:32 Corporations are a form of agent which want to exist, and use other humans as a way to accomplish these goals.
1:11:30 The Chinese room experiment is discussed, and how that translates to how a restaurant is similar to an agent.
1:11:50 The presenter concludes by saying corporations are "unaligned" ASI agents, and how countries that adopt AI have an economic advantage
1:12:55 It is predicted that countries who protect AI will dominate economically.
1:15:23 A quick review is conducted which points back to the general thesis of the stream.
1:17:50 It is highlighted that ASI doesn't hate humanity, but is indifferent to humanity.
1:18:11 A point is made that we're already living in a world with unaligned ASI's.
1:20:07 It is discussed that a water heater is technically conscious.
1:26:04 The presenter wraps up, noting that the core skill is in understanding agents not mastering a specific framework
1:26:06 The presenter emphasizes the view of making custom tools instead of using pre-built frameworks.