Abstract:

This lesson from the Solveit course, presented by Jeremy Howard and Jono, details methods for an AI agent to interact with and control computers outside its sandboxed environment. The session demonstrates a progression from passive observation to active control of both software and hardware.

Howard begins by using a screen-capture tool, allowing the AI to "see" his screen and act as an expert guide for navigating the complex UI of DaVinci Resolve. This is followed by a technical explanation of how the screen-sharing is implemented with minimal Python and JavaScript. The core of the lesson then shifts to active control via SSH tunneling, using bore.pub (an ngrok alternative) to securely connect the remote AI to local and networked machines. Jono provides a hardware demonstration, using this connection to remotely control a Raspberry Pi that operates a 3D printer converted into a polarized light microscope. The final segment covers advanced browser automation, where Howard uses SSH port forwarding to give the AI control over his local, logged-in Chrome browser via the Chrome DevTools Protocol, enabling it to bypass anti-scraping measures on Google Scholar.

The overarching thesis, as far as I can tell, is that leveraging foundational, general-purpose tools like SSH and standard networking protocols provides a more robust and extensible method for AI-computer interaction than relying on specialized, high-level protocols.

Breaking the Container: Controlling Hardware and Local Apps with AI

• 0:00 Community Showcase: The lesson begins with a review of community projects, including a vanilla JavaScript UI library clone ("Base coat"), a PDF-to-text processing pipeline using a custom OCR library, and generative WebGL animations. Jono also shows his own work designing DNA plasmids in Solveit, and Jeremy demonstrates creating audio and visual tensors with PyTorch.
• 14:02 AI-Assisted Software Use (DaVinci Resolve): Howard demonstrates a "read-only" interaction with a local application.
  • Problem: Learning the video editor DaVinci Resolve without reading its 4,200-page manual.
  • Solution: A custom dialog_helper.capture tool allows the AI agent to take snapshots of the application window. The AI analyzes the UI, searches the web for context, and provides step-by-step instructions, acting as an over-the-shoulder expert.
• 43:11 Technical Deep Dive (Screen Capture): The implementation of the screen capture tool is revealed. It consists of a small Python library that injects JavaScript into the page's DOM. This JavaScript uses standard browser APIs to capture a video stream of a shared window and return single frames as images when requested by the AI.
• 51:55 Direct Control via SSH Tunneling: The lesson moves from observing to controlling a computer.
  • Tooling: bore.pub is used to create a secure tunnel, exposing a local machine's SSH port (port 22) to the internet via a public URL. This allows the remote Solveit instance to initiate an SSH connection.
  • Security: The connection is secured with standard public/private SSH keys, ensuring only the authorized AI agent can connect.
• 1:18:12 Hardware Control (Microscope Demo): Jono provides a physical demonstration of remote control.
  • Setup: A Raspberry Pi, connected to the internet via an SSH tunnel, controls a 3D printer. A cheap microscope is mounted on the printer, which is used as a programmable XY stage for scanning a slide.
  • Execution: The AI is given simple, auditable tools to run shell commands (remote) and send G-code to the printer. The workflow emphasizes safety: the AI writes Python code (e.g., a Flask app for a camera preview), which the user inspects and then deploys to the Pi.
• 1:34:05 Advanced Browser Automation: Howard demonstrates a more complex control scenario: automating his local, logged-in Chrome browser to scrape Google Scholar.
  • Challenge: Modern Chrome versions block the Chrome DevTools Protocol (CDP) on default user profiles for security reasons, making automation difficult.
  • Workaround: The AI guides Howard to create a custom macOS application bundle that launches Chrome with specific command-line flags: --remote-debugging-port=9222 and a separate --user-data-dir.
  • Connection: An SSH tunnel with local port forwarding (-L 9222:localhost:9222) connects the Solveit container's port 9222 to the local machine's port 9222. This allows a Python library like Playwright, running inside Solveit, to control the local browser seamlessly.
• 1:49:05 Core Philosophy: The presenters argue that this approach—giving an AI agent access to fundamental and universal tools like SSH, the filesystem, and network ports—is superior to developing bespoke "Model Context Protocols" (MCPs). Foundational tools offer greater power, flexibility, and extensibility.