This video features Isaac Flath, an expert on the Python web framework FastHTML, demonstrating how to build custom AI evaluation and data annotation applications. Using a real-world commercial app built for AnkiHub (an AI-powered medical flashcard platform) as a case study, the presentation highlights the advantages of bespoke tooling for domain-specific challenges.
The talk covers the AnkiHub app's features, including a multi-annotator workflow, a disagreement resolution step, and the ability to rate not just retrieval results but also the initial user queries. Isaac then provides a code walkthrough of a simplified version of the application, explaining FastHTML's core principles of minimal boilerplate and developer productivity. He demonstrates how to define routes, interact with a SQLite database, and use HTMX for dynamic UI updates, all within Python. The discussion also touches on deployment options, comparisons with stacks like FastAPI/React, and the rapid development cycle enabled by the framework.
Building Custom AI Evaluation Tools with FastHTML
0:00 Isaac & FastHTML Intro: The session begins by emphasizing the high value of building custom data annotation apps for AI projects. Isaac Flath introduces FastHTML as a Python framework that allows data scientists to build full-fidelity web applications without extensive JavaScript knowledge.
1:40 AnkiHub Use Case: The core example is an annotation tool for AnkiHub, an AI platform for medical flashcards. The key challenge is the wide variability of user queries—from single keywords to entire documents—and the need for high domain-specific accuracy in retrieving very short results (flashcards).
4:49 AnkiHub App Demo: A walkthrough of the live application showcases its domain-specific features.
Workflow: The app supports a multi-annotator process. Two annotators independently rate results for a query.
Disagreement Resolution: If the annotators disagree, the query enters a "finalize" stage where both initial ratings are displayed side-by-side, allowing for collaborative resolution and comments.
Key Feature: Annotators can rate not only the relevance of the retrieved flashcards but also the quality of the user's query itself (e.g., marking it as "too general").
User Experience: The UI is designed for speed, with extensive keyboard shortcuts to efficiently process hundreds of results per query.
14:10 FastHTML Explained & Code: Isaac explains FastHTML's philosophy and demonstrates its implementation with a simplified version of the app.
Core Principles: FastHTML aims for minimal boilerplate and developer productivity by allowing developers to remain almost entirely within Python.
Implementation: The code shows how to define web routes, fetch data from a SQLite database using fastlite, and generate HTML components directly in Python.
Dynamic Updates: The app uses HTMX to handle user interactions like saving notes or updating ratings without requiring a full page reload. Python routes return small HTML snippets to update specific parts of the page.
17:31 Deployment & Resources: Deployment options like Railway and Plash are mentioned. Isaac points to a companion GitHub repository containing the presentation slides, source code, and a FastAPI/React version of the app for comparison.
19:40 FastHTML vs. FastAPI/React & Effort: The discussion covers the trade-offs of using FastHTML versus more mainstream frameworks. While AI code generators are less familiar with FastHTML, its simplicity allows for extremely rapid development. The entire simplified app is only ~80 lines of Python code, and the full-featured AnkiHub app was built in approximately three days.