This video introduces Vision Transformer (ViT) models enhanced with registers, a concept that earned the ICLR 2024 Best Paper award. The presentation first identifies a significant problem in traditional ViT models: the presence of "artifacts" β tokens exhibiting peaky, outlier attention values and high L2 norms, particularly in background areas of attention maps. These artifacts, more prevalent in larger models after substantial training, appear to store global rather than useful local information. To address this, the paper proposes a simple architectural tweak: appending a set of trainable, randomly initialized "register" tokens to the input sequence alongside the CLS token and image patches. This modification effectively eliminates the attention artifacts, leading to smoother and more interpretable attention heatmaps. Crucially, the introduction of registers maintains or slightly improves ImageNet classification accuracy while significantly boosting performance in dense prediction tasks like object detection for models such as DeiT and DinoV2. Analysis also suggests that individual registers learn to attend to diverse and specific regions or objects within an image.
Summarizing Vision Transformer Models with Registers
0:00 Introduction to ViT Models with Registers: The video introduces a new approach to Vision Transformer (ViT) models using "registers," highlighting its recognition as an ICLR 2024 best paper.
0:18 Problem: Artifacts in Traditional ViT Attention Maps: Traditional ViT models (e.g., DeiT, OpenCLIP, DinoV2, but not Dino B) exhibit "peaky outlier values" or "artifacts" in their attention heatmaps, particularly in background regions, which is counter-intuitive for visual understanding.
0:54 Artifact Characteristics: High Norms: These artifacts are tokens with approximately 10 times higher L2 norm output compared to normal patches, often exceeding 150 for local feature norms. Dinov2, for instance, shows a bimodal distribution of norms, with about 2.37% of patches exhibiting these high norms.
0:36 Artifact Occurrence Patterns: Artifacts typically appear in the middle layers (around layer 15 and beyond) of the ViT model, emerge only after sufficient training (approximately one-third of pre-training iterations), and are primarily found in larger transformer models (Large, Huge, but not Tiny, Small, or Base).
0:10 Information Content of Artifacts: Despite their high norms, these artifact patches are visually similar to their neighbors (background) and convey little useful local information. They show poor position prediction accuracy and higher pixel reconstruction error, indicating a loss of local spatial data.
0:06 Artifacts Store Global Information: Conversely, when used for image classification, these outlier patches achieve high accuracy, comparable to the CLS token, suggesting they aggregate and store global image information.
0:20 Hypothesis: Token Recycling: The authors hypothesize that ViT models learn to identify low-informative background patches and "recycle" these tokens to aggregate global information, discarding their local spatial details, as an optimal strategy for the model.
0:39 Solution: Vision Transformer Models with Registers: To counteract these artifacts, the paper proposes a simple architectural tweak: appending randomly initialized, trainable "register" vectors to the input sequence, similar to the CLS token, before processing by the transformer layers.
0:29 Registers Eliminate Artifacts: The introduction of registers successfully removes the artifacts and high-norm patches, resulting in significantly smoother and more semantically sensible attention heatmaps.
0:21 Impact on Classification and Detection: Using registers maintains or slightly improves ImageNet classification accuracy (e.g., +0.5% for DinoV2) and does not harm zero-shot classification. Crucially, it leads to significant improvements in object detection accuracy, nearly doubling performance for models like DeiT and DinoV2.
0:42 Registers Learn Diverse Attention: Analysis shows that individual registers learn to attend to diverse parts of an image, often focusing on specific objects or regions, suggesting a specialized role in feature extraction.
0:38 Key Takeaway: The addition of simple register tokens effectively addresses artifact issues in ViT attention maps, enhancing interpretability and leading to substantial performance gains in dense prediction and object discovery tasks without sacrificing classification accuracy.