Abstract:

This technical session, part of an ongoing curriculum on the "Solvet" AI-integrated development environment, explores advanced methodologies for extending AI capabilities beyond sandboxed containers into local and remote hardware. The presenters demonstrate the implementation of a Python-based screen capture utility (dialoghelper.capture) that allows Large Language Models (LLMs) to provide context-aware technical support for complex software like Da Vinci Resolve.

The core of the presentation focuses on bridging the gap between cloud-based AI instances and local workstations using SSH tunneling and reverse proxies (e.g., bore.pub and ngrok). Technical workflows detailed include the remote execution of G-code to control 3D printer-mounted microscopes, the creation of custom macOS application wrappers to enable Chrome DevTools Protocol (CDP) for browser automation, and the use of the Playwright library for navigating anti-scraping measures on platforms like Google Scholar. The session emphasizes utilizing foundational networking protocols and terminal abstractions over specialized, high-level APIs to achieve universal computer control.

Technical Summary: Remote Execution, Hardware Integration, and Browser Automation

• 0:01:37 Showcase of Student Implementations: Discussion of recent projects using Solvet, including a "vanilla" JavaScript/CSS UI library (Base Code UI), a PDF-to-structured-text OCR pipeline built with nbdev, and generative WebGL animation tutorials.
• 0:04:50 Synthetic Biology Design: A demonstration of using Solvet for DNA plasmid design, highlighting the AI’s ability to verify amino acid sequences and optimize codon usage compared to manual "cut-and-paste" methods in word processors.
• 0:08:52 Multi-Dimensional Data Visualization: Use of PyTorch tensors to generate images and sine-wave audio chords from scratch to educate users on vector mathematics and signal processing.
• 0:17:18 Integrated Screen Capture: Introduction of the dialoghelper.capture module. It allows users to share specific windows or tabs with the AI via a JavaScript-Python bridge, enabling the model to "see" and troubleshoot third-party software interfaces.
• 0:18:41 Contextual Learning (Da Vinci Resolve): A case study on learning complex video editing software by providing the AI with real-time screen captures and manuals. The model identifies UI elements and suggests specific keyboard shortcuts based on the current state of the application.
• 0:44:05 Under the Hood (Capture API): Technical breakdown of the capture.py utility. It utilizes htmx-triggered events and a JavaScript listener to grab video tracks from the browser API, scale them to a canvas, and return them as base64-encoded PNG data for LLM consumption.
• 0:52:02 SSH Tunneling and Reverse Proxies: Procedure for connecting a sandboxed cloud instance to a local machine behind a NAT. This involves using bore.pub (a Rust-based tunneling service) to map local port 22 to a public endpoint, secured via SSH public-key authentication.
• 1:08:42 Remote Shell Execution: Implementation of a Python remote() function that wraps subprocess.run and SSH commands. This allows Solvet to execute bash scripts, manage file systems, and debug local environment variables remotely.
• 1:19:00 Hardware Control (Microscopy & 3D Printers): A demonstration of remote-controlling a Raspberry Pi-powered microscope mounted on a 3D printer. The system uses SSH to send G-code commands to the printer’s serial port for XY-axis movement while streaming a camera preview via a Flask web server.
• 1:34:05 Browser Automation via CDP: Overview of automating local browsers using the Playwright library and Chrome Debugging Protocol (CDP). This requires tunneling port 9222 and launching Chrome with a non-default data directory to bypass modern security restrictions.
• 1:41:07 Custom macOS App Wrappers: Detailed steps for creating a .app bundle on macOS that initializes a Chrome instance with remote debugging enabled, allowing the AI to automate sessions while maintaining user login cookies.
• 1:51:32 Agentic Workflows vs. Manual Execution: Analysis of the trade-off between fully autonomous agents and "human-in-the-loop" coding. The presenters argue for a hybrid approach where the AI proposes code blocks that the user reviews and executes via remote shell utilities.