Text to Speech

Neural Text-to-Speech (TTS) is the synthesis of natural-sounding speech from text using deep learning — producing human-quality voice output that is indistinguishable from real speech for most applications, enabling voice assistants, audiobooks, accessibility tools, and synthetic media.

TTS Pipeline

1. Text Normalization: "2.5kg" → "two point five kilograms".
2. Text-to-Acoustic Features: Text → mel spectrogram (acoustic model).
3. Vocoder: Mel spectrogram → waveform.

Acoustic Models

Tacotron 2 (Google, 2018):
- Seq2seq with attention: Encoder processes text characters; decoder generates mel frames.
- First end-to-end TTS to achieve near-human quality.
- MOS (Mean Opinion Score): 4.53/5.0 vs. 4.58 for human speech.

FastSpeech 2 (Microsoft, 2020):
- Non-autoregressive: Parallel mel generation — 30x faster than Tacotron 2.
- Duration predictor: Explicitly predicts how many mel frames per phoneme.
- Variance adaptor: Controls pitch, energy, duration.

Vocoders

• WaveNet (DeepMind, 2016): Dilated causal convolution, 24 kHz audio. 0.5 RTF — too slow for production.
• HiFi-GAN: GAN-based vocoder. Real-time (RTF < 0.01), high quality. Standard in production.
• WaveGrad / DiffWave: Diffusion-based vocoders — highest quality but slower.

End-to-End TTS

• VITS (2021): Combines acoustic model + vocoder end-to-end with variational inference.
• Single model: Text → waveform. No two-stage pipeline.
• Naturalness competitive with two-stage at much simpler training.

Modern LLM-Based TTS

• VoiceBox (Meta, 2023): Flow Matching-based, in-context voice cloning.
• Tortoise TTS: DALL-E-like autoregressive + DDPM — ultra-high quality, slow.
• ElevenLabs, Bark: LLM-based voice synthesis with emotion and style control.

Neural TTS has effectively solved conversational-quality voice synthesis — the remaining challenges are real-time performance on edge devices, multilingual support without accent artifacts, and emotion expressiveness that matches the full range of human speech prosody.