This presentation introduces a novel approach to extend the capabilities of Large Language Models (LLMs) to the audio domain, enabling both comprehension and generation by teaching them the "language of audio." The core of the project is a two-step process. First, a causal audio tokenizer is developed to convert continuous audio waveforms into a sequence of discrete tokens. This is achieved by using a pre-trained VAE from Stable Audio to create a compressed latent representation, which is then quantized using a vector quantization model featuring a causal transformer and a conditional flow matching decoder. This process results in an ultra-low bitrate audio representation. Second, a Llama 7B model is fine-tuned on these audio tokens alongside text tokens using Low-Rank Adaptation (LoRA). The fine-tuned model demonstrates strong conversational and descriptive capabilities, accurately identifying and explaining audio content. However, audio generation remains a significant challenge, with the model producing repetitive sounds, indicating it is severely undertrained. The work shows promise in teaching LLMs audio comprehension, while highlighting the difficulty of high-fidelity audio generation with this method.
Teaching an LLM the Language of Audio: A Detailed Look at Using Sound Tokens with Llama
0:00:38 Motivation: The central question of the project is whether the powerful emergent abilities of Large Language Models (LLMs) can be extended to understand and generate audio by treating it as a language of discrete tokens.
0:08:24 Two-Step Approach: The project is broken down into two main stages: (1) developing a novel audio tokenizer to convert sound into discrete tokens, and (2) fine-tuning a Llama model to predict these tokens for both comprehension and generation tasks.
0:12:31 Creating Causal Audio Tokens: To create tokens suitable for an LLM, a vector quantization (VQ) model with a causal transformer is trained on the compressed output of a VAE. An improved version uses a powerful conditional flow matching (CFM) decoder, which results in better reconstruction quality and an ultra-low bitrate representation of 0.29 kbps.
0:18:56 Fine-Tuning the LLM: A Llama 7B model is fine-tuned using LoRA. The process starts with a pre-training stage on audio-caption pairs to learn basic captioning, followed by an instruction-tuning stage on a larger, more conversational dataset.
0:24:32 Success in Audio Comprehension: The fine-tuned model demonstrates impressive "chitty-chatty" capabilities for understanding and describing audio. When given an audio clip of a piano, it can correctly identify the instrument and explain its mechanics.
0:25:46 Challenges in Audio Generation: Audio generation proves to be a significant hurdle. The model is severely undertrained and tends to produce simplistic, repetitive sounds. While some cherry-picked examples (like a siren or water dripping) show potential, the overall generation quality is low.
0:27:52 Key Training Details: To focus the model on learning the new modality, the training loss for audio tokens was weighted 10 times higher than the loss for text tokens. A z-loss was also used to ensure training stability and prevent exploding logits.
0:28:44 Competitive Captioning Performance: On quantitative benchmarks for audio captioning (FENSE and SPICE scores), the model performs competitively with the LTU baseline model, and its performance continues to improve with ongoing training.
0:30:19 Conclusion: The method successfully teaches an LLM a discrete language of audio for comprehension tasks, achieving strong descriptive results with an ultra-low bitrate tokenizer. However, high-fidelity generation remains a future goal that will require significantly more training.