This presentation introduces Docling, an open-source Python library designed to convert complex documents (PDFs, images, etc.) into AI-ready formats like Markdown, HTML, and JSON. The core problem Docling addresses is the loss of structural and semantic information—such as columns, tables, headers, and image captions—that occurs with simple text parsers. The talk details Docling's three conversion pipelines: a simple one for easily parsable files, a PDF pipeline with OCR for standard and scanned documents, and an efficient Visual Language Model (VLM) pipeline for image-based conversion.
The presentation functions as a workshop, demonstrating key features through three labs. It covers structure-aware chunking, comparing a strict hierarchical chunker with a hybrid model that balances structure and token size for optimal Retrieval-Augmented Generation (RAG) performance. It also showcases advanced capabilities like enriching chunks with contextual headers and using integrated VLMs to generate detailed descriptions for images. The final lab culminates in building a complete multimodal RAG pipeline that can query both text and images, featuring "visual grounding" to highlight the precise source of retrieved information on the original document page for verification.
Docling: Structurally-Aware Document Conversion for AI
0:00:19 Introduction to Docling: Docling is an open-source Python library that converts various document formats (PDF, PowerPoint, scanned images) into AI-ready formats (Markdown, HTML, JSON) while preserving critical structural elements like tables, headers, and columns.
0:05:12 Three Conversion Pipelines: The library offers three distinct methods for conversion:
Simple Pipeline: For straightforward, easily parsable documents like Word files.
PDF Pipeline: Processes both standard and scanned PDFs, with options to enable OCR, layout analysis, and table extraction models.
VLM Pipeline: Uses a small, efficient 256M parameter Visual Language Model (Granite Docling) to convert any scanned document or image directly into the structured Docling format.
0:07:18 Basic Conversion and Extraction: The core conversion process is simple, requiring only three lines of code. After conversion, individual elements like text, tables (preserving merged cells in HTML), images, and captions can be extracted and handled separately.
0:14:11 Visualizing Structure and Enrichment: Docling can use the stored coordinate metadata to draw bounding boxes on the original document, visualizing how it parsed elements like headers, text, and footnotes. It can also enrich documents by using integrated VLMs to automatically generate detailed text descriptions for every image.
0:21:31 The Importance of Semantic Chunking: For RAG applications, naive fixed-size chunking splits text arbitrarily, destroying semantic context. Docling enables structure-aware chunking to create semantically coherent chunks based on paragraphs, sections, or sentences.
0:26:20 Hierarchical vs. Hybrid Chunking: Docling provides two main chunkers:
Hierarchical Chunker: Strictly follows the document's structure, which can result in chunks that are too large or small for embedding models.
Hybrid Chunker: The recommended method, as it balances respecting the document's structure with adhering to token size limits, making it ideal for RAG.
0:27:52 Chunk Contextualization: To improve retrieval quality, Docling can automatically prepend section headers to all chunks that fall under them, ensuring each chunk retains its broader context.
0:30:41 Building a Multimodal RAG Pipeline: The workshop demonstrates building a RAG system that ingests a complex, image-heavy document. The pipeline processes and embeds text, tables, and AI-generated image descriptions into a vector database.
0:38:48 Visual Grounding for Verification: A key feature is visual grounding, which allows the RAG system to not only provide an answer but also display the original page with a bounding box drawn around the exact text or image that was used as the source. This is critical for verifying the accuracy of the retrieved information.
0:46:41 Key Use Case and Value Proposition: Docling was created to provide a free, open-source, and locally-executable tool for preparing real-world organizational documents for AI applications. This approach provides significant cost savings and enhances data privacy compared to relying on expensive, closed-source cloud APIs.