This lesson focuses on overcoming the inherent limitations of the Solveit sandbox environment by demonstrating how to securely and effectively command local and remote external computing resources. The core mechanisms involve injecting JavaScript for local screen vision and utilizing SSH tunneling and port forwarding to establish secure command-line (CLI) and network access to external machines.
Key practical demonstrations include leveraging Solveit's screen capture to act as an expert tutor for a complex local application (Da Vinci Resolve), establishing full SSH access to a local computer using bore.pub for remote command execution, automating a physical device (a Raspberry Pi controlled 3D printer/microscope stage) via G-code commands, and performing advanced browser automation (e.g., Google Scholar scraping) by tunneling the Chrome Debugging Protocol (CDP) via SSH. The lesson advocates for using foundational tools like SSH rather than relying on numerous specialized APIs for agentic workflows.
Controlling Other Computers: Breaking the Solveit Sandbox
0:02 Lesson Context & Demos: The session opens with a showcase of community projects, including creating UI clones (Basecamp UI), structured knowledge extraction from PDFs (using a tool based on mDDEV), generative WebGL animations, and complex synthetic biology/DNA design workflows within Solveit.
13:02 The Motivation: Solveit instances offer limited computing resources (small shared CPU), making them unsuitable for large tasks like model training. The lesson aims to extend Solveit’s capabilities to external computers.
14:02 Local Application Control (Da Vinci Resolve): Solveit is demonstrated guiding the user through complex video editing tasks. This relies on injecting custom, open-source Python/JavaScript functions (dialog_helper.capture) to take non-streaming screenshots of the user's local Da Vinci Resolve window upon request.
25:28 AI Knowledge and Vision: The AI uses web searches and screen capture, proving sufficient to navigate Da Vinci Resolve's obscure interface and workflows, negating the need for the tool's 4,200-page manual. As far as I can tell, this shows that targeted vision combined with expert knowledge is highly effective.
35:07 Context Management Insight: The AI avoids filling its context window with massive images by writing descriptive notes based on the current screenshot, which is a surprisingly effective method for maintaining focus.
43:04 Technical Implementation of Capture: The screen capture mechanism is purely client-side, using standard browser JavaScript APIs (e.g., htmx.t trigger) to capture the shared video track and return the data URL. This framework allows for expanding control using other browser capabilities (audio, Bluetooth, local filesystem access).
52:26 Secure Remote CLI Access (SSH): To gain secure command-line control of a local machine (which typically lacks a public IP), the user employs SSH tunneling via a service like bore.pub (or ngrok). This connects the remote Solveit server to the local SSH port (22).
1:06:36 Authentication: Access is secured using public key authentication; the public key must be placed in the local machine's authorized_keys. This setup ensures no one else can exploit the tunnel, as far as I can tell.
1:08:30 Remote Execution Functions: Custom Python functions (remote) are defined to wrap standard CLI commands (subprocess.run) and execute them via the established SSH tunnel, effectively giving Solveit full command execution rights over the local machine.
1:18:12 Physical World Automation (Microscope): The SSH access is used to control a Raspberry Pi connected to a 3D printer acting as a microscope stage. Solveit uses custom remote tools (Rview, Rrg) to explore the Pi’s filesystem and then writes a small Flask application (in a Python string) to the remote machine to provide a camera preview and accept G-code commands for stage movement.
1:33:50 Advanced Browser Automation (CDP): To automate a challenging site like Google Scholar using a logged-in browser session, the user tunnels the Chrome Debugging Protocol (CDP) port (9222) via SSH local port forwarding (-L 9222:localhost:9222).
1:39:31 Chrome Configuration Required: Due to recent security measures, Chrome must be launched with specific flags (including a non-default user data directory) to enable remote debugging. The AI is used to write the Mac .app shell script necessary for this specialized launch.
1:43:51 Playwright Execution: The Python automation library Playwright connects to the local Chrome instance through the SSH tunnel, allowing it to perform actions (search, type, take screenshots) on the user's fully configured, logged-in browser.
1:48:46 Key Takeaway: The combination of SSH (for CLI and network tunneling) and visual capture grants Solveit control over the entire local computer and network, rejecting the premise that specialized APIs (like MCPs) are necessary for complex agentic workflows. SSH is the foundational tool.