This document summarizes a discussion originating from a post by Andrej Karpathy regarding the DeepSeek-OCR paper. Karpathy proposes that Large Language Models (LLMs) might be fundamentally better off using rendered images (pixels) as their primary input instead of traditional text tokens. He argues that pixels offer superior information compression, create a more universal input stream capable of handling text formatting and images, enable more powerful bidirectional attention, and, most importantly, eliminate the problematic text tokenizer. The subsequent Hacker News discussion largely concurs with the critiques of tokenizers, exploring the implications of Karpathy's idea while also raising counterpoints about output generation and suggesting alternative data representations like perceptual units ("percels") or tokens derived from image compression techniques.
Are Pixels Better Than Text for LLMs? A Summary of Karpathy's Proposal
The Core Idea: Sparked by the DeepSeek-OCR paper, Andrej Karpathy questions if using rendered images (pixels) as the universal input for LLMs is more effective than using text tokens. This would involve rendering even plain text into an image before feeding it to the model.
"Kill the Tokenizer": A central argument is the elimination of the text tokenizer. Karpathy criticizes tokenizers as a flawed, non-end-to-end component that inherits the "ugliness" of Unicode and byte encodings, creates security risks, and fails to represent the visual and semantic nature of characters, such as emojis or visually similar letters.
Information Compression & Efficiency: Representing text as an image could be a more compressed format for the model, potentially leading to shorter context windows and greater efficiency.
A More General Input: Pixels provide a universal information stream. This approach would natively handle not just text, but also formatting (bold, color), tables, and arbitrary images within a single input modality.
Superior Processing: Image-based data can be processed with powerful bidirectional attention by default, which is more powerful than the standard autoregressive (one-direction) attention used for text generation.
Input vs. Output: Karpathy clarifies that this pixel-first approach would primarily apply to the model's input (the user's message). The model's output (the assistant's response) would likely remain text, as generating realistic pixel-based responses is a much harder challenge.
Community Discussion: In the Hacker News thread, there is broad agreement that tokenizers are a "hack." Commenters draw parallels to how humans consume text visually and explore alternatives like "percels" (units of perceptual information) or using image compression algorithms like DCT (used in JPEGs) to create more meaningful "tokens" from visual data.
Linguistic Perspective: The discussion highlights how a visual approach could benefit logographic languages like Chinese. For example, the visual connection between characters like 木 (tree) and 林 (forest) is lost in text-token systems but would be preserved in a pixel-based model.