Voice Agents

Voice is the oldest human interface and, for many tasks, still the fastest: customer support lines, appointment booking, in-car assistants, accessibility tools, and hands-busy work such as field service or clinical note-taking. Voice agents combine speech recognition, a language model with tools, and speech synthesis into a system that holds a spoken conversation and gets work done. The engineering is dominated by one constraint text chat never faces: time. In conversation, a pause of a second feels awkward and two seconds feels broken. This lesson explains the architectures, shows how to budget latency millisecond by millisecond, covers telephony basics, and sets out the evaluation and legal and ethical obligations that come with putting a synthetic voice on a phone line.

The intuition: a relay team versus a single runner

The classic voice agent is a relay: one runner (speech recognition) listens and writes down what was said, hands a note to the second runner (the language model), who writes a reply and hands it to the third (speech synthesis), who reads it aloud. Each handoff costs time, and the note loses information: tone, hesitation, sarcasm, background noise.

A speech-to-speech model is a single runner who hears and speaks directly, with no written notes in between. It is faster and catches tone, but you cannot read its notes to check them, and it is harder to coach.

Good conversation also needs turn-taking: knowing when the other person has finished (not just paused for breath), and stopping politely when interrupted. A comparison of conversation in ten languages (Stivers et al., 2009) found that every language studied minimizes both overlap and silence, with typical gaps between turns of a few hundred milliseconds or less. People begin planning their reply before the other speaker finishes. Machines that wait for silence and then start thinking will always feel slow; good voice systems overlap listening, thinking and speaking.

Architecture 1: the cascaded pipeline (ASR → LLM → TTS)

1. Voice activity detection (VAD) classifies short audio frames (typically 10 to 30 ms) as speech or non-speech.
2. Automatic speech recognition (ASR) converts speech to text. Streaming ASR emits partial hypotheses while the user talks and a final transcript at the end of the utterance.
3. Endpointing / turn detection decides the user has finished their turn.
4. The LLM (with tools, retrieval and state) generates a text reply, streamed token by token.
5. Text-to-speech (TTS) converts text to audio, ideally streaming: it starts speaking from the first phrase while the LLM is still generating.
6. Transport carries audio to and from the user (WebRTC in a browser or app, or a telephone network).

Advantages: every component is swappable and the best of its kind; you get a text transcript to log, inspect, moderate and evaluate; tool use and guardrails work exactly as in text agents. Disadvantages: latency stacks up across stages, and paralinguistic information (emotion, emphasis, who is speaking) is mostly lost at the text bottleneck.

Architecture 2: speech-to-speech models

Speech-to-speech (also called end-to-end or native audio) models take audio tokens in and produce audio tokens out, usually with an internal text stream as well. Research systems such as Kyutai's Moshi (Défossez et al., 2024) demonstrated full-duplex dialogue, where the model listens and speaks simultaneously, and commercial providers offer realtime speech-to-speech APIs over WebSocket or WebRTC.

Many production teams still use cascades for task-oriented agents (support, booking) because control, compliance and tool reliability matter more than prosody; speech-to-speech is attractive for open conversation and companionship-style products. Hybrids exist, such as a speech-to-speech front end that calls a text model for hard reasoning.

Turn-taking: endpointing and barge-in

Endpointing is the decision that the user's turn has ended. The simplest rule is "silence for $\tau$ milliseconds". The trade-off is sharp:

• Short $\tau$ (around 200 ms): fast, but the agent interrupts people who pause mid-sentence ("My account number is ... uh ... 4-7-2...").
• Long $\tau$ (around 800 ms or more): polite, but every turn feels sluggish.

Better systems use semantic endpointing: a small model looks at the partial transcript (and sometimes prosody) and predicts whether the utterance is complete. "I'd like to book a table for" is clearly unfinished; "Yes" is clearly finished. The silence threshold can then adapt: short when the sentence is complete, long when it is not.

Barge-in is the user interrupting the agent. The system must:

1. Detect user speech while the agent is talking, which requires acoustic echo cancellation so the agent does not hear its own voice through the user's speaker and interrupt itself.
2. Stop TTS playback within roughly 100 to 200 ms.
3. Cancel in-flight LLM generation and pending TTS.
4. Truncate the agent's last message in the conversation history to what the user actually heard. If the agent generated three sentences but only one was played, the model must not believe it said all three. This detail is easy to miss and causes strange follow-ups.
5. Distinguish real interruptions from backchannels ("mm-hm", "right") and noise, which should not stop the agent.

Worked example: a latency budget

Define voice-to-voice latency as the time from the moment the user stops speaking to the moment they hear the first audio of the reply. Here is a realistic first build of a cascaded agent on a phone line:

1.7 seconds feels slow. Now optimize stage by stage:

New total: $250 + 150 + 250 + 100 + 120 + 120 = 990$ ms, a saving of 710 ms. The two biggest wins came from not waiting: deciding the turn ended sooner and starting to speak before the sentence was complete. Two further techniques are common:

• Speculative generation: start the LLM on the partial transcript before the endpoint fires, and discard the result if the user keeps talking.
• Filler and acknowledgement: when a tool call will take seconds (a database lookup), have the agent say "Let me check that for you" immediately rather than going silent.

Always report latency as a distribution: medians hide the slow tail that users remember. A system with 900 ms median and 3,000 ms at the 95th percentile feels unreliable.

Streaming in code

Two building blocks: an endpointer over audio frames, and a chunker that releases text to TTS at clause boundaries.

import numpy as np

def frame_energy_db(frame: np.ndarray) -> float:
    rms = np.sqrt(np.mean(frame.astype(np.float64) ** 2)) + 1e-12
    return 20 * np.log10(rms)

def endpoint_index(frames, threshold_db=-40.0, silence_ms=500, frame_ms=20):
    # Return the frame index where the turn ends, or None if it has not ended.
    needed = silence_ms // frame_ms
    heard_speech, silent_run = False, 0
    for i, f in enumerate(frames):
        if frame_energy_db(f) > threshold_db:
            heard_speech, silent_run = True, 0
        elif heard_speech:
            silent_run += 1
            if silent_run >= needed:
                return i
    return None

sr = 16_000
t = np.arange(sr) / sr
speech = 0.1 * np.sin(2 * np.pi * 220 * t)               # 1 s of "speech"
audio = np.concatenate([speech, np.zeros(sr)])           # then 1 s of silence
frames = audio.reshape(-1, sr * 20 // 1000)              # 20 ms frames of 320 samples
print(endpoint_index(frames))   # 74: frame 50 is the first silent one, 25 silent frames needed

This energy-based endpointer is the textbook baseline; real systems use trained VAD models, which are far more robust to background noise, plus semantic turn detection. The example shows the mechanics: 20 ms frames at 16 kHz are 320 samples each; 500 ms of silence is 25 frames; speech ends after frame 49, so the endpoint fires at frame 74.

import re

def speakable_chunks(token_stream, min_chars=20):
    # Yield text pieces at clause or sentence boundaries for streaming TTS.
    buf = ""
    for tok in token_stream:
        buf += tok
        m = next((m for m in re.finditer(r"[.!?;:,]\s", buf) if m.end() >= min_chars), None)
        if m:
            yield buf[: m.end()].strip()
            buf = buf[m.end():]
    if buf.strip():
        yield buf.strip()

tokens = ["Sure", ",", " I", " can", " help", " with", " that", ".", " Your",
          " order", " ships", " on", " Monday", "."]
print(list(speakable_chunks(tokens)))
# ['Sure, I can help with that.', 'Your order ships on Monday.']

The first chunk is released as soon as a boundary appears after at least 20 characters, so TTS can start speaking while the model is still writing. Too small a minimum produces choppy prosody; too large adds latency. (The trailing space check means the final sentence is emitted by the flush at the end.)

Telephony integration basics

Most commercial voice agents answer phone numbers, so you need to know the plumbing.

• PSTN (public switched telephone network) is the traditional phone network. To reach it from software you go through a carrier or a communications platform that provides phone numbers.
• SIP (Session Initiation Protocol, RFC 3261) is the signalling protocol for setting up, modifying and ending calls (INVITE, ACK, BYE). A SIP trunk connects your system to a carrier.
• RTP (Real-time Transport Protocol) carries the actual audio packets, typically 20 ms per packet.
• Codecs: classic telephony uses G.711 at 8 kHz sampling and 8 bits per sample, giving $8{,}000 \times 8 = 64$ kbit/s. Each 20 ms frame is $8{,}000 \times 0.02 = 160$ samples, which is 160 bytes. Telephone audio is narrowband (roughly 300 to 3,400 Hz), which lowers ASR accuracy compared with wideband microphones; use ASR models tuned for telephony, and resample correctly (for example to 16 kHz) when a model expects it.
• DTMF tones (keypad presses) arrive in-band or as separate events; agents often need them for menus and PIN entry.
• WebRTC is the browser and mobile standard for real-time media. It handles NAT traversal (ICE with STUN and TURN servers), encryption (DTLS-SRTP), and adaptive jitter buffering, and typically uses the Opus codec, which is wideband and far better than G.711 for recognition.
• Call control: transfers to humans (warm transfer, where context is passed to the human agent), hold, recording, and hang-up detection. Design the human handoff from day one.

A typical architecture: carrier → SIP trunk → media server or communications platform → your agent service over a WebSocket or WebRTC stream of audio frames → ASR, LLM and TTS services, with tools calling your business systems.

Evaluating voice agents

Evaluate each layer and the whole.

ASR accuracy: word error rate,

where $S$, $D$ and $I$ are the numbers of substituted, deleted and inserted words in the best alignment and $N$ is the number of words in the reference transcript. Example: reference "please cancel my order for tomorrow morning" ($N = 7$); hypothesis "please cancel the order for tomorrow" has one substitution (my → the) and one deletion (morning), so WER $= 2/7 = 28.6\%$. WER treats all words equally, but "cancel" misheard as "council" matters more than "the" for "a"; also measure entity accuracy for names, numbers, dates and account IDs, the things that break tasks.

Latency: voice-to-voice latency at p50 and p95, measured end to end on real networks, not just component benchmarks.

Conversation quality: task completion rate, turns to completion, transfer-to-human rate, false barge-ins (agent stopped by noise or backchannels), missed barge-ins, cut-offs (agent interrupted the user), and repeated-question rate.

TTS quality: intelligibility, pronunciation of domain terms and numbers, and naturalness, often measured by human ratings such as mean opinion score (MOS) on a 1 to 5 scale.

Simulation at scale: an LLM-driven simulated caller speaks through TTS to your agent, with scripted personas, accents, background noise and interruptions, and an automated judge checks outcomes. Pair it with review of real call recordings (with consent) because simulated callers are more cooperative than real ones.

Safety, consent and the law

Voice adds risks text agents do not have.

Impersonation and voice cloning. Modern TTS can clone a voice from a short sample. Only clone voices with documented, informed consent from the speaker, restrict who can create cloned voices, and consider watermarking or provenance metadata for generated audio. Never build an agent that claims to be a specific real person without authorization.

Disclosure. Tell people they are talking to an AI, at the start of the call. Beyond being good practice, it is increasingly a legal requirement: for example, the EU AI Act includes transparency obligations for AI systems that interact directly with people, and several US states have enacted bot-disclosure rules in some contexts. Requirements differ by jurisdiction and change often, so get legal review for your markets.

Robocall rules (US). In February 2024 the US Federal Communications Commission issued a declaratory ruling (FCC 24-17) that AI-generated voices count as "artificial" voices under the Telephone Consumer Protection Act (TCPA). In practice this means outbound calls using AI voices fall under the same consent requirements as prerecorded-voice robocalls, generally requiring the called party's prior express consent, with stricter written consent for telemarketing. The ruling came shortly after AI-generated robocalls imitating President Biden's voice urged New Hampshire voters to skip the January 2024 primary. Inbound calls that a customer initiates are a different situation, but disclosure and recording rules still apply.

Call recording. Recording and transcription require consent under wiretapping laws. In the US some states require only one party's consent and others require all parties' consent; other countries have their own rules, and data-protection laws such as the GDPR govern storage and use of recordings and transcripts. Announce recording, honour opt-outs, set retention limits, and redact sensitive data (card numbers, health details) from transcripts.

Voice as an authentication factor. Voice cloning undermines "voiceprint" authentication. Your agent should not treat a familiar-sounding voice as proof of identity, and high-risk actions (payments, account changes) need out-of-band verification.

Vulnerable callers. Phone lines reach people in distress. Define escalation paths to humans for emergencies, complaints and anyone asking for a person.

Common pitfalls

• Fixed long silence thresholds. Every turn pays the full timeout. Use adaptive or semantic endpointing.
• Waiting for the full reply before TTS. Stream at clause boundaries.
• No echo cancellation. The agent hears itself and interrupts itself.
• Not truncating history on barge-in. The model believes it said things the user never heard.
• Measuring only median latency. The p95 tail drives user perception.
• Testing only on clean wideband audio. Phone calls are narrowband, noisy and accented.
• Silent tool calls. Several seconds of silence sounds like a dropped call; acknowledge and fill.
• Skipping disclosure and consent. Legal exposure, and a breach of user trust.

In practice

• Choose a cascade when you need control, auditability, strong tool use and compliance; consider speech-to-speech when natural prosody and minimal latency dominate. Prototype both against your latency and quality targets.
• Write a latency budget per stage and track p50 and p95 in production with per-stage timestamps.
• Stream everything: partial ASR, LLM tokens, and TTS audio; flush at clause boundaries; speculate where safe.
• Invest in turn-taking: trained VAD, semantic endpointing, echo cancellation, backchannel-aware barge-in, and history truncation.
• Design human handoff, disclosure, recording consent and data retention before launch, with legal review for each jurisdiction you call into.
• Build a simulation harness with noisy, accented, interrupting callers, and review real calls regularly.