Voice Agent

Voice Agents and Real-Time Conversation

Voice Agent Architecture

[User Speech]
     |
     v
[Speech-to-Text] (Whisper, Deepgram)
     |
     v
[LLM Processing] (GPT-4, Claude)
     |
     v
[Text-to-Speech] (ElevenLabs, OpenAI)
     |
     v
[Audio Output]

Key Metrics
| Metric | Target | Description |
|--------|--------|-------------|
| Latency | <1s total | Time from speech end to audio start |
| TTFB | <500ms | Time to first audio byte |
| Interruption handling | Immediate | User can interrupt agent |

Real-Time API (OpenAI)

from openai import OpenAI

client = OpenAI()

# Real-time conversation API (conceptual)
session = client.realtime.create_session(
    voice="nova",
    model="gpt-4o-realtime"
)

# Bidirectional audio streaming
async for event in session:
    if event.type == "audio_chunk":
        play_audio(event.audio)
    elif event.type == "transcript":
        print(f"User: {event.text}")

Latency Optimization
| Strategy | Impact |
|----------|--------|
| Streaming STT | Start processing before speech ends |
| Streaming TTS | Start speaking before full response |
| Prefill responses | "Let me think..." while processing |
| Edge deployment | Reduce round-trip time |
| Speculative execution | Predict likely responses |

Turn-Taking

class VoiceAgent:
    def __init__(self):
        self.is_speaking = False
        self.vad = VoiceActivityDetector()

    async def handle_audio(self, audio_chunk):
        if self.is_speaking and self.vad.is_speech(audio_chunk):
            # User interruption
            await self.stop_speaking()
            await self.process_interruption(audio_chunk)

Use Cases
| Use Case | Requirements |
|----------|--------------|
| Customer support | Low latency, empathy |
| Voice assistants | Fast, accurate |
| Interviewing | Turn-taking, follow-ups |
| Language tutoring | Pronunciation feedback |

Platforms
| Platform | Features |
|----------|----------|
| Vapi | Full-stack voice AI |
| Retell | Enterprise voice agents |
| ElevenLabs Conversational | Low latency |
| Hume AI | Emotion-aware |

Challenges
- Managing interruptions gracefully
- Handling noise and multiple speakers
- Cross-talk in phone calls
- Maintaining conversation context
- Latency across components