This presentation introduces a novel framework for quantifying "unintended memorization" in Large Language Models (LLMs) by distinguishing it from generalization through the lens of information theory. Moving beyond binary "extraction-based" definitions—which the speaker argues fail to separate a model's ability to generalize (e.g., solving novel math problems) from its retention of specific training sequences—the research proposes a continuous metric based on Shannon compression. By measuring the bits required to represent a data point relative to a reference model, the researchers can calculate "model capacity."
Experimental results using GPT-style transformers demonstrate that these models possess a stable capacity of approximately 3.6 to 3.8 bits per parameter. A critical finding is the alignment between this capacity limit and the onset of "double descent," where models transition from memorizing individual points to learning reusable distributional patterns once data volume exceeds model storage capacity. Furthermore, the study reveals that while memorization remains constant for uniform random data, it actually decreases per-example in natural text as datasets scale, suggesting a shift toward generalization. Finally, the research highlights that word rarity (measured via TF-IDF) is a primary driver of high-memorization scores and that traditional membership inference is a more sensitive, though distinct, signal than sequence extraction.
Technical Review: LLM Memorization, Model Capacity, and Scaling Dynamics
0:01:45 New Definition of Memorization: Proposes a metric for measuring memorization at the individual data point level, allowing for the summation of total "unintended memorization" across a corpus to define a model's information capacity.
0:05:40 Compression Disparity: Analysis of Llama 3 (8B) illustrates that the model weights (~16GB) are nearly 1,000 times smaller than the estimated Shannon-compressed training data (~9TB), framing the fundamental constraint of lossy compression in LLMs.
0:08:59 Limits of Extraction Tests: Argues that "completion tests" and "adversarial compression" are flawed proxies for memorization because they cannot distinguish between learning a specific string and generalizing a rule (e.g., arithmetic).
0:13:51 Generalization vs. Memorization: Defines generalization as learning the underlying world distribution, whereas unintended memorization is the learning of specific noise or unique features of training samples.
0:20:42 Calculating Model Capacity: Utilizes uniform random data (IID bits) to establish a baseline for maximum storage capacity. Experimental math suggests that when data exceeds capacity, the model memorizes exactly half as many bits per example when the dataset size doubles.
0:27:18 Capacity and Double Descent: Demonstrates that the "double descent" phenomenon in test loss begins precisely when the training data volume reaches the model's total bit-storage capacity (the saturation point).
0:30:16 Bits per Parameter Constant: Empirically identifies that GPT-style transformers converge to a capacity of ~3.6 bits per parameter (using FP32 weights), regardless of sequence length or vocabulary size.
0:32:37 Text Data Scaling Divergence: Unlike random data, natural language shows a decrease in per-example memorization as the dataset grows, indicating that generalizable patterns (rules/facts) eventually supplant rote memorization in the model's limited storage.
0:41:01 Drivers of Memorization: High-memorization outliers correlate strongly with high Term Frequency (TF) scores; rare tokens and non-target language characters (e.g., Japanese characters in an English corpus) are disproportionately stored.
0:45:12 Extraction vs. Membership Inference: Functional testing shows that extraction rates for test and train data are often indistinguishable in large-scale regimes, whereas Membership Inference (identifying if a point was in the training set via loss analysis) remains a much easier task for smaller data-to-model ratios.