Speech Language Model
Speech Language Models are the foundation models that process and generate speech directly as a native modality — either by tokenizing audio into discrete units that language models can process alongside text, or by operating on continuous audio representations, enabling unified models that can transcribe, translate, converse, and generate speech in a single architecture rather than cascading separate ASR → LLM → TTS systems.
Evolution of Speech AI
Era 1 (pre-2020): Separate ASR → NLU → TTS pipeline
[Audio] → [ASR: DeepSpeech/wav2vec] → [Text] → [NLU] → [Text] → [TTS] → [Audio]
Problem: Error propagation, high latency, loses prosody/emotion
Era 2 (2023+): Speech Language Models
[Audio] → [Speech LM] → [Audio + Text]
Unified model handles everything end-to-endKey Systems
| Model | Developer | Approach | Capability |
|---|---|---|---|
| Whisper | OpenAI | Encoder-decoder, continuous | Transcription, translation |
| AudioPaLM | Discrete audio tokens + LLM | Speech-to-speech translation | |
| VALL-E | Microsoft | Neural codec LM | Voice cloning from 3s sample |
| SpeechGPT | Fudan | Discrete speech tokens | Spoken dialogue |
| Moshi | Kyutai | Full-duplex streaming | Real-time spoken conversation |
| GPT-4o | OpenAI | Native audio modality | Multimodal conversation |
Audio Tokenization Approaches
| Approach | Method | Tokens/sec | Quality |
|---|---|---|---|
| Continuous (Whisper) | Mel spectrogram → encoder | N/A (continuous) | High |
| Semantic tokens (HuBERT) | Self-supervised clustering | 25-50 | Good meaning, poor quality |
| Acoustic tokens (EnCodec) | Neural audio codec (VQ-VAE) | 75-150 | High quality |
| Hybrid | Semantic + acoustic tokens | 100-200 | Best of both |
Whisper Architecture
[Audio waveform] → [Mel spectrogram] → [Transformer Encoder]
↓
[Transformer Decoder] → [Text tokens]- Trained on 680,000 hours of labeled audio from the internet.
- Multitask: Transcription, translation, language identification, timestamp prediction.
- Robust: Works across accents, background noise, technical terminology.
- Sizes: Tiny (39M) to Large-v3 (1.5B parameters).
Neural Codec Language Models (VALL-E)
- Step 1: Encode speech with neural codec (EnCodec) → 8 codebooks of discrete tokens.
- Step 2: Train autoregressive LM on first codebook (semantic content).
- Step 3: Train non-autoregressive model for remaining codebooks (acoustic detail).
- Result: Given 3 seconds of someone's voice → generate arbitrary speech in that voice.
- Implication: Zero-shot voice cloning with natural prosody and emotion.
Full-Duplex Speech AI
- Traditional: Half-duplex — system listens OR speaks, never both.
- GPT-4o / Moshi: Full-duplex — can listen while speaking, handle interruptions.
- Architecture: Streaming input + streaming output simultaneously.
- Enables: Natural conversation flow, backchanneling ("mmhmm"), interruption handling.
Training Data Scale
| Model | Training Data | Languages |
|---|---|---|
| Whisper | 680K hours | 99 languages |
| SeamlessM4T | 1M+ hours | 100+ languages |
| AudioPaLM | PaLM text + audio | Multilingual |
| VALL-E | 60K hours (LibriLight) | English |
Speech language models are the technology that will make AI conversational interfaces indistinguishable from human interaction — by processing speech as a native modality rather than converting to text as an intermediate step, these models preserve the full richness of spoken communication including tone, emotion, and timing, enabling real-time AI assistants that can truly converse rather than merely chat.