This video is a technical tutorial demonstrating the creation of a complete video processing pipeline on a Xilinx Zynq UltraScale+ FPGA. The presenter uses a ZC102 board, a custom MIPI CSI-2 to FMC adapter, and an IMX477 camera. The guide covers the hardware IP chain in Vivado (MIPI RX, Demosaic, DMA) and, more importantly, the software configuration in Linux using the V4L2 framework. A substantial focus is placed on constructing the device tree overlay, which the presenter claims is a poorly documented but critical step for defining the pipeline's structure for the kernel. The process culminates in successfully capturing raw image frames and viewing them, validating the entire hardware and software stack.
Building an FPGA Video Pipeline: From Camera to Linux Framebuffer
0:00 Hardware Setup: The system consists of a Xilinx ZC102 FPGA board, an IMX477 camera, and a custom MIPI CSI-2 to FMC adapter. The adapter is the key piece of custom hardware, allowing up to four MIPI cameras to connect to the FPGA's FMC port.
2:58 Vivado IP Pipeline: The FPGA design is straightforward. It uses a MIPI CSI-2 Receiver Subsystem to get data from the camera, a Demosaic IP to convert the sensor's raw Bayer format to RGB, and a Video Frame Buffer Write IP to DMA the final image into system memory.
5:50 Critical Design Constraints: The MIPI PHY requires a specific 200 MHz clock to function correctly. Additionally, the V4L2 driver model requires that individual IP blocks have their resets controlled by GPIO pins rather than a global reset signal, which is a common point of failure.
10:52 Linux Device Tree Overlay: As far as I can tell, this is the most critical part of the tutorial. The presenter walks through the device tree overlay file, which describes the entire hardware pipeline to the Linux kernel. It defines each IP and explicitly links them together using port and endpoint nodes, which is mandatory for the V4L2 subsystem to understand the data flow.
16:47 Command-Line Configuration: Once the drivers are loaded, the entire pipeline must be configured using V4L2 utilities. The user must manually set the correct resolution and pixel format for each link in the chain (sensor output, CSI receiver output, demosaic output, etc.) to ensure they match.
20:42 Verifying the Output: Raw frames are captured from the /dev/video0 device. The presenter uses a web-based raw pixel viewer to display the captured data, confirming the pipeline works. The resulting image has obvious color errors, which is expected since the minimal pipeline lacks any color correction or image signal processing (ISP).