Reviewer Persona: Senior Research Scientist, Privacy-Preserving Machine Learning (PPML)
Abstract:
This presentation introduces Differentially Private Prototypical Learning (DPPL), a transfer learning framework designed to mitigate the systematic utility degradation observed in minority classes when training under Differential Privacy (DP). The research identifies a fundamental bias in DP-SGD: gradient clipping disproportionately affects underrepresented classes, rotating the global gradient towards majority class directions.
To address this, the speaker proposes two primary methods for prototype estimation in a latent embedding space provided by a public encoder: DPPL-Mean, which utilizes a Gaussian mechanism for private mean estimation, and DPPL-Public, which employs the exponential mechanism to select prototypes from a public dataset. The latter leverages a monotonic utility function and bounded range analysis to achieve superior utility-privacy trade-offs in Zero-Concentrated DP (zCDP). Empirical results on imbalanced versions of CIFAR-100 and Food-101 demonstrate that while standard transfer learning baselines fail on minority classes, DPPL maintains high balanced accuracy, particularly in high-imbalance regimes (1:100 ratio), by utilizing parallel composition across classes and distance-based classification.
Technical Summary: Improving Private Imbalance Transfer Learning via Public Data Reuse
00:03:00 The Gradient Bias Problem: DP-SGD introduces systematic bias against minority classes. Because minority samples often produce higher-magnitude gradients (containing more "novel" information), they are clipped more aggressively. This rotates the aggregate gradient toward the majority classes, leading to "disparate impact" where accuracy on underrepresented groups suffers.
00:05:10 Limitations of Conventional Mitigation: Standard techniques like SMOTE (synthetic oversampling) incur prohibitive privacy costs under DP because the privacy budget must be expended on additional synthetic samples derived from private data.
00:07:34 DPPL Framework: The proposed method uses a fixed public encoder (e.g., Vision Transformer) to map private data into an embedding space. Classification is performed by calculating the minimum distance between a new sample's embedding and class-specific "prototypes."
00:09:12 DPPL-Mean (Private Mean Estimation): This variant calculates class prototypes using a private mean estimation with a Gaussian mechanism. The speaker notes that for normalized embeddings centered around a known origin, "naive" mean estimation often outperforms complex algorithms like CoinPress.
00:11:00 DPPL-Public (Exponential Mechanism): This method selects prototypes from a public data pool. By using cosine similarity as a utility function and ensuring monotonicity (adding a sample only increases utility), the researchers apply "bounded range" analysis. This provides a factor-of-four improvement when converting to zCDP compared to standard report-noisy-max approaches.
00:14:30 Multi-Prototype Sampling: To increase robustness, the method can select a set of $k$ prototypes. To manage the combinatorial explosion of choices (up to $10^{30}$), the algorithm samples the utility value first in log-space before selecting set members, ensuring numerical stability.
00:18:21 Benchmarking Imbalance: The authors tested imbalance ratios up to 1:100. In balanced settings, DPPL is competitive with SOTA (e.g., Mehl et al.), but in imbalanced settings, DPPL significantly outperforms all baselines.
00:21:05 Minority Class Performance: On the smallest 25% of classes in CIFAR-100, DPPL maintains high utility where baseline methods collapse. DPPL-Public achieves satisfying utility at an epsilon an order of magnitude lower than mean-based estimation.
00:22:14 Encoder Dependency: The choice of the public encoder is critical. High-capacity models like ViT-H/14 provide significantly better latent separation than ResNet-50, which is essential for distance-based prototype classification to succeed.
Parallel Composition: Treating classes independently and assigning each an equal privacy budget prevents majority classes from "consuming" the budget.
Inherent Balancing: Using a distance-based decision function (minimum distance to prototype) removes the possibility of the model learning a frequency-based bias toward majority classes.
Public Data Synergy: Public data is most effective when the task is "in-distribution" relative to the pre-training data; out-of-distribution tasks (like Food-101) show higher sensitivity to public data size reductions.