Abstract:

This lesson covers three main topics: packaging a Python library with nbdev, web scraping, and automating cloud infrastructure.

First, Jeremy Howard demonstrates the full nbdev workflow, turning the DaisyUI helper functions from previous lessons into a complete, pip-installable library called fhj-daisy. This process involves creating a GitHub repository, initializing the project with nbdev new, writing code and documentation within solveit dialogues, and using nbdev commands to export the library, build documentation, and publish to PyPI. He also discusses a pragmatic philosophy on software testing.

Next, Johno Whitaker introduces web scraping using httpx and BeautifulSoup, emphasizing the utility of CSS selectors. He presents two examples: first, extracting a structured bus schedule from a messy HTML table, and second, using an LLM to perform structured data extraction from an unstructured prose description of perfume ingredients.

Finally, Howard tackles scraping JavaScript-heavy websites that require browser rendering, using the Zyte service as an example. He then applies this to a practical problem: automating Google Cloud. By scraping the latest Google Cloud documentation and feeding it to the AI, he guides it to generate correct, up-to-date Python code to programmatically start, monitor, and terminate a GPU virtual machine, bypassing the complex web UI.

Solveit Lesson 6: From Library Creation to Advanced Web Scraping

• 0:07 Course Philosophy: Embracing Generalism and AI-Assisted Learning The lesson begins with a discussion on the course's "choose your own adventure" format. Howard advocates for becoming a generalist and using AI to handle implementation details, allowing learners to focus on high-level concepts and customize lessons to their interests.

• 8:05 Creating a Python Library with nbdev Howard walks through the process of packaging Python code into a distributable library. The workflow is demonstrated from scratch:

  • Create a GitHub repository (gh repo create).
  • Initialize the project structure (nbdev new).
  • Write code and documentation within solveit dialogues (notebooks), using #| export to mark code for inclusion in the library.
  • Use nbdev export to generate the .py source files.
  • Install an editable version for local development (pip install -e .).
  • nbdev automatically sets up GitHub Actions to build and deploy a documentation website on every push.

• 43:45 A Pragmatic Approach to Testing Howard argues against writing excessive, brittle tests that couple the tests to implementation details. For a UI library like this, he favors visual inspection and ensuring the code runs without exceptions, as the maintenance cost of overly specific tests often outweighs their benefit.

• 59:56 Publishing the Library The process is completed by publishing the package to the Python Package Index (PyPI) using a single command, nbdev_pypi, making fhj-daisy publicly available via pip install.

• 1:11:30 Fundamentals of Web Scraping Johno Whitaker introduces web scraping with httpx for fetching web pages and BeautifulSoup for parsing HTML. He strongly recommends using CSS selectors (.select()) as a foundational and versatile tool for navigating the document structure. The process is shown to be iterative and investigative, especially when dealing with poorly structured real-world HTML.

• 1:30:32 Extracting Unstructured Data with LLMs A second scraping example demonstrates a more advanced technique. When the desired information (perfume ingredients) is embedded in free-form text, the strategy is to first scrape the relevant text block and then use an LLM with tool-calling to perform structured data extraction, converting the prose into a JSON object.

• 1:48:44 Handling Dynamic (JavaScript-Heavy) Websites Howard addresses the challenge of scraping sites where content is loaded by JavaScript after the initial page load. The recommended solution is to use a service that renders the page in a full browser, such as Zyte, which can return the final, rendered HTML.

• 1:50:29 Automating Google Cloud with Scraped Docs This advanced scraping technique is applied to automate creating a Google Cloud VM. By scraping the current documentation, Howard provides the AI with up-to-date context, enabling it to generate correct Python API calls to start, stop, and delete a VM instance. This workflow effectively turns solveit into a specialized, state-of-the-art assistant for complex, rapidly changing APIs.

• 2:14:33 Preview of Next Lesson The next lesson will focus on system administration, demonstrating how to use solveit to interact with remote machines via SSH.