Whisper

Speech-to-Text with Whisper

What is Whisper?
OpenAI Whisper is an automatic speech recognition (ASR) model trained on 680,000 hours of multilingual audio.

Model Sizes
| Model | Parameters | Speed | Quality |
|-------|------------|-------|---------|
| tiny | 39M | Fastest | Basic |
| base | 74M | Fast | Good |
| small | 244M | Medium | Better |
| medium | 769M | Slow | Great |
| large-v3 | 1.5B | Slowest | Best |

Basic Usage

import whisper

model = whisper.load_model("base")
result = model.transcribe("audio.mp3")

print(result["text"])

Advanced Options

result = model.transcribe(
    "audio.mp3",
    language="en",           # Specify language
    task="transcribe",       # or "translate"
    word_timestamps=True,    # Per-word timing
    fp16=True               # Faster on GPU
)

# Word-level timestamps
for segment in result["segments"]:
    for word in segment["words"]:
        print(f"{word['start']:.2f}s: {word['word']}")

Faster Whisper
Optimized implementation using CTranslate2:

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", compute_type="float16")
segments, info = model.transcribe("audio.mp3")

for segment in segments:
    print(f"[{segment.start:.2f}s] {segment.text}")

Real-Time Transcription

import sounddevice as sd
import queue

audio_queue = queue.Queue()

def callback(indata, frames, time, status):
    audio_queue.put(indata.copy())

# Stream and transcribe chunks
with sd.InputStream(callback=callback):
    while True:
        audio_chunk = audio_queue.get()
        text = model.transcribe(audio_chunk)["text"]
        print(text, end=" ", flush=True)

Use Cases
| Use Case | Features Needed |
|----------|-----------------|
| Meeting transcription | Timestamps, speaker diarization |
| Podcast processing | Chapter markers |
| Subtitles | Word timestamps, formatting |
| Voice search | Fast, streaming |
| Accessibility | High accuracy |

Speaker Diarization
Identify who is speaking:

from pyannote.audio import Pipeline

diarization = Pipeline.from_pretrained("pyannote/speaker-diarization")
result = diarization("audio.wav")

for turn, _, speaker in result.itertracks(yield_label=True):
    print(f"{turn.start:.1f}s - {turn.end:.1f}s: {speaker}")

Best Practices
- Use larger models for noisy audio
- Specify language when known
- Combine with diarization for meetings
- Consider Faster Whisper for speed
- Chunk long audio for memory efficiency