This presentation by Benjamin Stal details a summer internship project focused on making large, high-performance speech quality assessment models more compact and practical. The work begins by evaluating state-of-the-art models, identifying a large, self-supervised learning model (XLSR-based) as the most accurate and generalizable for predicting Mean Opinion Scores (MOS). However, its prohibitive size (1.8 GB) limits its use in real-world applications. The core of the project is to reduce this model's size using two primary techniques: knowledge distillation and pruning. For distillation, the large model acts as a "teacher" to train a smaller "student" model (CNN+Transformer architecture) on a vast, artificially generated dataset with pseudo-labels. For pruning, both importance-based and magnitude-based methods are explored to remove redundant weights. The results show that distillation successfully creates compact models that close over half of the performance gap to the large teacher model. Importance-based pruning also proves to be an effective reduction strategy. A key remaining challenge is the accurate prediction of quality for generated speech (Text-to-Speech), where distillation tended to amplify the teacher model's existing errors.
Shrinking AI: Distilling and Pruning Large Models for Speech Quality Assessment
0:00:40 The Problem: The talk addresses the need for computational models to automatically estimate speech quality (Mean Opinion Score - MOS) as an alternative to expensive and time-consuming human listening tests.
0:02:41 Applications: These models are crucial for various tasks, including selecting the best speech processing systems, evaluating dataset quality, personalizing on-device speech enhancement, and monitoring the quality of commercial products.
0:05:03 Baseline Models: The project begins by evaluating several state-of-the-art speech quality models, including DNSMOS, NISQA, and XLSR-SQA, which use different architectures and are trained on different types of speech data (e.g., telephone networks, VoIP, text-to-speech).
0:09:39 Key Finding - XLSR Generalization: Evaluation shows that the XLSR model, based on a large self-supervised learning framework, provides the best generalization and performance across a wide range of speech degradation tasks, even those it was not explicitly trained on.
0:15:18 The Challenge - Model Size: While the XLSR-based "teacher" model is highly accurate, its massive size of 1.8 GB makes it impractical for on-device deployment or use as a real-time cost function in training.
0:16:12 The Solution: Knowledge Distillation: The primary strategy to reduce model size is knowledge distillation. This involves using the large, accurate teacher model to generate "pseudo-labels" for a massive, unlabeled, and artificially generated dataset. A much smaller "student" model is then trained on this data, effectively learning the essential knowledge from the teacher.
0:22:06 Alternative Solution: Pruning: As an alternative size-reduction method, the project explores model pruning, which systematically removes unimportant weights from the large teacher model. Importance-based pruning (which identifies weights that have the least impact on the loss function) proved much more effective than simple magnitude-based pruning.
0:24:23 Distillation Results: The distillation process is highly successful. The resulting compact student models manage to close more than half of the performance gap between a small model trained from scratch and the very large teacher model, offering a great balance of size and accuracy.
0:27:39 A Remaining Challenge: Text-to-Speech (TTS): Predicting the quality of generated speech remains a significant hurdle. The distillation process was found to sometimes amplify the teacher model's errors on unseen TTS data, making the student model perform even worse on these specific tasks.
0:40:21 Final Recommendation: The choice of model depends entirely on the use case. For offline analysis where computational resources are not a constraint, the largest teacher model is best. For practical applications like on-device deployment or frequent validation during training, a smaller, distilled model provides the optimal trade-off between performance and efficiency.