This presentation, delivered by a senior IT architect (Hans Beers), analyzes the critical limitations and systemic risks associated with the rapid deployment of autonomous (Agentic) AI, drawing from personal experiences using Google's Gemini Canvas/Notebook LM for software development. The speaker initially demonstrates how LLMs can quickly prototype complex systems, such as a physics simulation (0:39), but notes their inability to maintain context and correct errors reliably over multiple iterations (0:40, 3:35). This practical failure—the forgetting of previously programmed features—frames the core argument: that Large Language Models (LLMs) are fundamentally probabilistic and non-deterministic (10:08), undermining traditional software engineering principles focused on correctness and stability (7:22). The discussion proceeds to categorize major Agentic AI risks (cascading failures, prompt injection 2.0, unintended loops) (9:50) and highlights the danger of excessive autonomy without adequate guardrails (5:51). The analysis concludes by criticizing the reckless deployment of Agentic AI driven by economic and geopolitical pressures (14:44, 16:54), citing examples from the defense industry and consumer operating systems (Windows Recall) (12:55), and advocating for user transparency and developer opt-in (15:21).
Analysis of Agentic AI Risks and Limitations by Senior IT Architect Hans Beers
0:07 Speaker Introduction and Background: Hans Beers, an IT architect since the 1980s (with experience at IBM), introduces the topic, which originated from discussions about interesting IT topics with Lev.
0:31 LLM Capability Demonstration (Physics Simulation): Beers recounts attempting to recreate a complex physics simulation (thermodynamics) he worked on in the 1990s, using Gemini Canvas with the Flash model. He achieved initial results in 5–10 minutes (1:29), noting the significant contrast to the year-long, high-cost project of the 1990s (1:57).
2:12 AI Error Detection and Auto-Repair: Beers tested the AI's ability to program system tests and perform auto-repair, which Gemini successfully executed (2:30, 2:43).
3:35 Core Limitation Identified (Context Forgetting): When Beers attempted to improve the simulation by adding features or requesting new simulations (e.g., changing from gravity to nuclear physics), the AI began forgetting previous code (3:35). An initial set of five simulations was reduced to three (3:40). Despite instructing the AI to remember, it failed, leading Beers to conclude that AI was "hitting some limitations" (4:24).
5:03 Agentic AI Risk Simulation: Beers created a Notebook LM (Google tool) about Agentic AI, where he simulated various AI user types (autonomous software engineer, research scientist) and manipulated parameters like self-modification and autonomy (5:28).
6:11 Lack of Guardrails Leads to Failure: The simulation demonstrated that high self-modification and autonomy parameters without sufficient guardrails quickly led to issues, such as exposing the "wrong APIs to persons" (6:16), illustrating systemic risks.
7:04 Historical Context of Software Correctness: Beers draws a parallel to Edsger Dijkstra (developer of the Dijkstra algorithm), who emphasized proving correctness in computer programs (7:10, 7:22).
8:38 Apollo 11 Software Error: An anecdote about the Apollo 11 lunar lander software is used to stress the difficulty of proof and verification: five days before launch, a significant sign error related to lunar gravity was discovered and fixed (8:40, 8:49).
9:35 Systemic Risks of Agentic AI: Current discussions identify risks in Agentic AI, including cascading failures, unintended autonomous loops, model/data drift (9:50), and the fundamental fact that LLMs are probabilistic, not deterministic (10:08), leading to unstable behavior.
10:23 New Attack Vectors: Beers mentions advanced techniques like prompt injection 2.0 (10:25), where documents are fed to LLMs to alter their behavior and output, posing new automation problems.
12:14 Loss of Control: Citing a Signal presentation, Beers warns that the introduction of Agentic AI within operating systems (like Microsoft Windows Recall) causes developers/managers to lose control, as the underlying software becomes non-deterministic (12:21, 12:48).
12:55 Windows Recall Data Capture Concerns: Windows Recall is criticized for saving massive amounts of personal data (screen captures, messages) under the hood (13:07), necessitating extra development work by Signal to protect user privacy (13:28).
13:44 Fundamental LLM Flaws: LLMs cannot distinguish between instructions and data, seeing everything as tokens (13:46, 13:57). This enables indirect prompt injection (14:00) where malicious commands can be hidden.
14:40 Economic Pressure vs. Responsibility: There is a significant tension between large investors (hyperscalers) facing economic pressure to deploy AI quickly and the necessity of responsible, guarded deployment (14:44, 15:00).
15:19 Call for Industry Action: Recommendations include stopping reckless deployment (15:21), requiring developer opt-in instead of automatic opt-in (15:23), and ensuring user transparency (15:30).
15:56 Geopolitical Pressures (Defense Autonomy): Beers highlights the use of autonomous AI in the weapons industry, specifically drones like the Bumblebee (16:07, 16:15), which make last-mile decisions autonomously to bypass jamming (16:23). This sector often disregards guardrails due to geopolitical pressures (16:54), resulting in the use of products that are known to sometimes make wrong decisions (17:09).