This stream explores the thesis that large-scale generative video models are on a trajectory to become unified, general-purpose vision foundation models, similar to how LLMs transformed natural language processing. The discussion is anchored by a recent DeepMind paper demonstrating that Google's Veo 3 model can perform a wide range of computer vision tasks—such as edge detection, segmentation, and object extraction—in a zero-shot manner by generating a video that transforms an input image into the desired output. This process is termed "Chain of Frames," an emergent visual reasoning analogous to "Chain of Thought" in LLMs.
To understand the underlying technology of these closed-source models, the stream delves into open-source papers from ByteDance (OmniInsert, Lynx). These papers reveal the complex, multi-stage training curriculums and sophisticated synthetic data generation pipelines that power modern video models. Key techniques discussed include using other AI models (VLMs, SAM) as filters and quality judges in "pipeline engineering," and advanced architectural adaptations like LoRAs and ControlNet-style modules. The speaker concludes that the future of vision models lies in a self-improvement loop, where they will increasingly be trained on their own AI-generated data, eventually replacing all specialized vision models and becoming a core technology for robotics and content creation.
Video Models as the Next General-Purpose Foundation Models
0:01:53 The Rise of Zero-Shot Video Models: The stream's central topic is the DeepMind paper "Video Models are Zero Shot Learners and Reasoners," which argues that generative video models like Google's Veo 3 are evolving into general-purpose vision systems.
0:03:56 The Generalist Vision Trajectory: The core argument is that video models are following the path of LLMs, developing the ability to perform a wide variety of vision tasks (segmentation, edge detection, image editing) without being explicitly trained for them.
0:05:58 A Major Research Caveat: A significant limitation noted is that the DeepMind researchers accessed Veo 3 via an external API that includes an LLM-based prompt rewriter. This makes it difficult to distinguish the video model's inherent intelligence from that of the prompt-rewriting LLM.
0:12:15 A New Paradigm for Vision Tasks: Veo 3 performs tasks like edge detection by generating a video that transitions from the input image to the desired output (e.g., an edge map). This novel approach often produces results superior to traditional algorithms and "ground truth" datasets.
0:17:43 The Dawn of Visual Prompt Engineering: Just as with LLMs, the emergence of generalist vision models is creating a new field of prompt engineering. Crafting the right text prompt and selecting the perfect starting frame is becoming crucial to eliciting specific, high-quality results.
0:25:30 "Chain of Frames" as Emergent Reasoning: The intermediate frames generated by the video model are framed as a visual analogue to "Chain of Thought." This "Chain of Frames" acts as an emergent reasoning process, allowing the model to visually "think" through a problem, even though this behavior was not explicitly trained.
0:28:57 The Complexity of Data Generation: An analysis of ByteDance's OmniInsert paper reveals the intricate "pipeline engineering" required to train these models. This involves using a combination of other AI models, like VLMs and Segment Anything (SAM), to synthetically create and filter massive training datasets.
0:45:28 Increasingly Complex Training Schedules: Modern model training now involves multi-stage progressive strategies that go far beyond simple pre-training and fine-tuning. This concept of a multi-phase curriculum is also seen in Meta's CWM paper, which introduces the term "mid-training."
0:49:50 Generalization Through Data and Distillation: The speaker asserts that the models' general capabilities come from the massive scale and diversity of their training data (i.e., the internet of video). This has created a recursive self-improvement loop where models are used to generate or filter data to train other, more capable models.
0:57:30 The Future is AI-Generated Video Data: The next major leap in capabilities will likely come from video models training on their own synthetically generated data. This will create an "infinite data flywheel," which will eventually dwarf the amount of human-generated video data, even with the advent of AR glasses.