Speech and Audio AI: Recognition, Synthesis and Real-Time Voice

Speech and audio AI covers every system whose input or output is sound: dictation and meeting transcription, voice assistants and phone agents, captioning, text-to-speech (TTS) for accessibility and media, speaker verification at a bank, and sound-event detection for machines, wildlife or security. Audio is also where real-time constraints bite hardest, because people notice a half-second delay in conversation immediately. This lesson takes you from the raw waveform to modern speech-to-speech models.

The picture in your head

A microphone records air pressure over time: a long list of numbers that wiggle up and down. That list is hard to reason about directly, in the same way a piano roll is easier to read than a recording of the piano. So almost every audio model first turns the waveform into a spectrogram: a picture with time on the horizontal axis, frequency (pitch) on the vertical axis, and brightness for loudness. Vowels show up as horizontal bands, consonants like "s" as bright hiss at the top, silences as dark columns. Once audio is an image-like grid, the CNNs and transformers from the vision and language lessons apply almost unchanged.

Audio as a signal

Sampling

Digital audio stores the pressure $x(t)$ at discrete times $t = n / f_s$, where $f_s$ is the sampling rate in samples per second (Hz). The Nyquist–Shannon theorem says a sampled signal can only represent frequencies below $f_s / 2$ (the Nyquist frequency); higher frequencies fold back as distortion called aliasing unless filtered out first. Common rates: 8 kHz for traditional telephony (speech up to about 4 kHz, hence the muffled phone sound), 16 kHz for most speech models, 44.1 kHz for CDs and 48 kHz for video. Each sample is usually a 16-bit integer or a 32-bit float in $[-1, 1]$.

Practical consequence: one second of 16 kHz mono audio is 16,000 numbers, so a one-hour meeting is about 57.6 million samples. Models do not consume this directly at that granularity; they consume frames.

From waveform to spectrogram

The short-time Fourier transform (STFT) cuts the signal into overlapping frames, multiplies each by a smooth window (Hann is typical) to avoid edge artifacts, and takes the discrete Fourier transform of each. Speech systems typically use 25 ms windows with a 10 ms hop, which at 16 kHz is 400 samples per window and 160 per hop, giving 100 frames per second. The squared magnitude of each frequency bin is the power spectrogram. Window length is a trade-off: longer windows give finer frequency resolution but blur fast events in time.

The mel scale

People hear pitch roughly logarithmically: the step from 100 to 200 Hz sounds much larger than from 5,000 to 5,100 Hz. The mel scale warps frequency to match perception; a common formula is

with $f$ in Hz and $m$ in mels. Under this formula 1,000 Hz maps to about 1,000 mel, 4,000 Hz to about 2,146 mel and 8,000 Hz to about 2,840 mel, so the top half of a 16 kHz recording's spectrum (4–8 kHz) is squeezed into about a quarter of the mel range. A mel filterbank is a set of triangular filters whose centres are evenly spaced in mels; summing the power spectrum through each filter and taking the logarithm gives the log-mel spectrogram, the standard input to modern speech models (Whisper, for example, uses 80 log-mel channels in its original release).

MFCCs (mel-frequency cepstral coefficients) go one step further: apply a discrete cosine transform to each log-mel frame and keep the first ~13 coefficients. This decorrelates the features and compactly describes the spectral envelope (the shape of the vocal tract). MFCCs were the workhorse features of the HMM/GMM era and are still used in small, classical systems; deep networks generally prefer log-mels because they can learn their own decorrelation.

The code below builds a log-mel spectrogram from scratch with NumPy; libraries such as torchaudio and librosa do the same thing with more options (filter normalization, padding, power-to-dB conversion).

import numpy as np

def hz_to_mel(f): return 2595.0 * np.log10(1.0 + f / 700.0)
def mel_to_hz(m): return 700.0 * (10 ** (m / 2595.0) - 1.0)

def mel_filterbank(sr, n_fft, n_mels):
    fft_freqs = np.linspace(0, sr / 2, n_fft // 2 + 1)            # Hz of each FFT bin
    edges = mel_to_hz(np.linspace(0, hz_to_mel(sr / 2), n_mels + 2))
    fb = np.zeros((n_mels, len(fft_freqs)))
    for m in range(n_mels):
        lo, mid, hi = edges[m], edges[m + 1], edges[m + 2]
        rising = (fft_freqs - lo) / (mid - lo)
        falling = (hi - fft_freqs) / (hi - mid)
        fb[m] = np.maximum(0, np.minimum(rising, falling))        # triangle filter
    return fb

def log_mel_spectrogram(x, sr=16000, n_fft=400, hop=160, n_mels=80):
    n_frames = 1 + (len(x) - n_fft) // hop                        # 25 ms windows, 10 ms hop
    frames = np.stack([x[i * hop: i * hop + n_fft] for i in range(n_frames)])
    power = np.abs(np.fft.rfft(frames * np.hanning(n_fft), axis=1)) ** 2
    return np.log(power @ mel_filterbank(sr, n_fft, n_mels).T + 1e-10)

sr = 16000
t = np.arange(sr) / sr                                            # one second
x = np.sin(2 * np.pi * 440 * t) + 0.5 * np.sin(2 * np.pi * 2000 * t)
S = log_mel_spectrogram(x, sr)
print(S.shape)          # (98, 80): 98 frames x 80 mel bands; energy peaks near 440 Hz and 2 kHz

Automatic speech recognition (ASR)

ASR maps a sequence of acoustic frames $X = (x_1, \dots, x_T)$ to a token sequence $Y = (y_1, \dots, y_U)$, where usually $T \gg U$ (100 frames per second versus a few words per second) and we do not know which frames belong to which token. Before deep learning this was solved with hidden Markov models, Gaussian mixture acoustic models, pronunciation dictionaries and n-gram language models glued together. Modern systems are end to end, built on one of three ideas.

CTC: Connectionist Temporal Classification

CTC (Graves, Fernández, Gomez and Schmidhuber, 2006) lets a network emit one symbol per frame from the vocabulary plus a special blank symbol _. A deterministic collapse function $\mathcal{B}$ first merges repeated symbols, then removes blanks. For example, the frame outputs h h _ e l l _ l o o collapse to h _ e l _ l o, then to "hello". The blank between the two l's is what allows a genuine double letter.

Many frame-level paths $a = (a_1, \dots, a_T)$ collapse to the same transcript, and CTC sums over all of them:

where $p_t(\cdot \mid X)$ is the network's softmax output at frame $t$. The sum over exponentially many paths is computed efficiently by a forward–backward dynamic program (similar to HMMs), and training minimizes $-\log P(Y \mid X)$. CTC assumes outputs at different frames are conditionally independent given the audio, so it has no built-in language model; accuracy improves when decoding with beam search plus an external LM. It is fast, naturally streaming and monotonic (output order follows time). wav2vec 2.0 (Baevski et al., 2020), which pretrains on unlabeled audio with a self-supervised objective and then fine-tunes with CTC, showed that good ASR can be built with very little transcribed speech.

Encoder–decoder with attention

An encoder (convolutions plus transformer or LSTM layers) turns the spectrogram into hidden states; an autoregressive decoder generates tokens one at a time, attending to the encoder states, exactly like a translation model. This removes the conditional-independence assumption and learns an internal language model, but the decoder can hallucinate or loop on silence and is harder to stream. RNN-Transducer (RNN-T) models are a middle ground widely used for on-device streaming: a CTC-like monotonic alignment combined with a prediction network that conditions on previous outputs.

Whisper-style weakly supervised models

OpenAI's Whisper (Radford et al., 2022, "Robust Speech Recognition via Large-Scale Weak Supervision") is an encoder–decoder transformer trained on about 680,000 hours of audio paired with transcripts collected from the web. Audio is processed in 30-second windows of log-mel features; the decoder is steered by special tokens that specify the language, the task (transcribe or translate to English) and whether to emit timestamps. The lesson of Whisper is the same as in vision and language: large, diverse, imperfectly labeled data produces models that are robust to accents, noise and domains without per-domain fine-tuning. Its typical failure modes are also instructive: on long silences or music it can hallucinate fluent text that was never said, and it repeats phrases in loops; production systems add voice-activity detection (VAD) and repetition checks.

from transformers import pipeline   # decoding audio files requires ffmpeg on the system

asr = pipeline("automatic-speech-recognition", model="openai/whisper-small",
               chunk_length_s=30)                     # chunk long files into 30 s windows
result = asr("meeting.wav", return_timestamps=True)
print(result["text"])
for chunk in result["chunks"][:5]:
    print(chunk["timestamp"], chunk["text"])

Measuring ASR: word error rate

Word error rate (WER) is the word-level edit distance between the hypothesis and the reference transcript, divided by the number of reference words:

where $S$ is substitutions, $D$ deletions, $I$ insertions (from the minimum-cost alignment) and $N$ the number of words in the reference. Because insertions are unbounded, WER can exceed 100%.

Worked example. Reference: "the cat sat on the mat" ($N = 6$). Hypothesis: "the cat sit on mat today". Best alignment:

$S = 1$, $D = 1$, $I = 1$, so WER $= 3/6 = 50\%$. The code computes it with the standard dynamic program (Levenshtein distance over words):

import numpy as np

def wer(ref, hyp):
    r, h = ref.split(), hyp.split()
    d = np.zeros((len(r) + 1, len(h) + 1), dtype=int)
    d[:, 0] = np.arange(len(r) + 1)          # delete every reference word
    d[0, :] = np.arange(len(h) + 1)          # insert every hypothesis word
    for i in range(1, len(r) + 1):
        for j in range(1, len(h) + 1):
            sub = d[i - 1, j - 1] + (r[i - 1] != h[j - 1])
            d[i, j] = min(sub, d[i - 1, j] + 1, d[i, j - 1] + 1)
    return d[-1, -1] / len(r)

print(wer("the cat sat on the mat", "the cat sit on mat today"))   # 0.5

WER's limits matter in practice. It weights every word equally, so "do not transfer the funds" misrecognized as "do transfer the funds" costs one deletion, the same as dropping "um." It is sensitive to normalization: "10%" versus "ten percent," casing and punctuation can swing scores by several points, so always normalize both sides the same way and report how. For languages without word boundaries, use character error rate (CER). For products, also track error rates on the words that matter (names, numbers, product terms).

Text-to-speech

A modern neural TTS pipeline has three stages:

1. Text front end: normalize ("Dr." to "doctor," "€5" to "five euros"), and often convert to phonemes.
2. Acoustic model: predict a mel spectrogram from the text. Tacotron 2 (Shen et al., 2018) did this with an attention-based sequence-to-sequence network; non-autoregressive models (FastSpeech-style) predict durations explicitly to be faster and avoid skipped or repeated words.
3. Neural vocoder: turn the mel spectrogram back into a waveform. The mel spectrogram discards phase, so this is a generative problem. WaveNet (van den Oord et al., 2016) generated audio one sample at a time with dilated causal convolutions: high quality, but slow because 24,000 sequential steps are needed per second at 24 kHz. GAN-based vocoders such as HiFi-GAN (Kong, Kim and Bae, 2020) generate all samples in parallel and run much faster than real time.

A newer family treats speech as language: a neural audio codec compresses audio into sequences of discrete tokens, and a transformer predicts those tokens like words, conditioned on text and a short sample of a target voice. This enables expressive, zero-shot voice cloning from a few seconds of audio, which is also why voice-cloning fraud, consent and watermarking are now core concerns (covered in the generative-AI and responsible-AI courses).

Who spoke, and when

• Speaker verification / identification. A network maps an utterance to a fixed-size speaker embedding (x-vectors and ECAPA-TDNN are well-known designs). Verification compares a new embedding to an enrolled one with cosine similarity and a threshold; quality is reported as the equal error rate (EER), the operating point where false accepts equal false rejects.
• Diarization answers "who spoke when" without knowing the speakers in advance. The classic pipeline: VAD to find speech, segmentation into short speaker-homogeneous chunks, embedding each chunk, then clustering. End-to-end neural diarization models handle overlapping speech better. The metric is diarization error rate (DER): missed speech plus false-alarm speech plus speaker confusion, divided by total reference speech time. Open-source toolkits such as pyannote.audio are a common starting point. Meeting transcription combines diarization with ASR by aligning word timestamps to speaker turns.

Audio classification

Sound-event and audio classification (gunshot, glass breaking, machine fault, bird species, music genre) usually applies an image-style model to log-mel spectrograms: CNNs, or the Audio Spectrogram Transformer, which patches the spectrogram like a ViT. Google's AudioSet (2017), roughly two million human-labeled 10-second YouTube clips drawn from an ontology of more than 600 sound classes (527 of them appear in the released labels), is the standard pretraining and benchmark set. Because clips often contain several sounds, it is a multi-label problem (sigmoid per class, not softmax) evaluated with mean average precision.

from transformers import pipeline

clf = pipeline("audio-classification", model="MIT/ast-finetuned-audioset-10-10-0.4593")
for r in clf("street.wav", top_k=5):
    print(f"{r['label']:>25s}  {r['score']:.3f}")

Real-time constraints

• Real-time factor (RTF) = processing time ÷ audio duration. RTF below 1 is necessary for live use but not sufficient; latency (time from speech to output) matters separately.
• Streaming models emit partial results as audio arrives. CTC and RNN-T stream naturally; Whisper-style models were designed for 30-second windows and need chunking tricks or streaming variants.
• Endpointing (deciding the user has finished) is often the largest hidden delay in voice agents: wait too little and you interrupt; wait too long and the agent feels sluggish.
• Barge-in: users interrupt. The system must detect speech during its own playback (with echo cancellation) and stop talking.
• Budget the chain. In a cascaded agent (ASR → LLM → TTS), latencies add: final transcript, the LLM's time to first token, and TTS time to first audio. Streaming every stage and starting TTS on the first sentence are standard mitigations.

Speech-to-speech models

The cascade throws away information: the transcript loses tone, emphasis, hesitation and laughter, and every hand-off adds delay. Speech-to-speech (or "native audio") models take audio tokens in and produce audio tokens out within one model, often while also producing text internally. Kyutai's Moshi (2024) is a published example: a speech-text foundation model designed for full-duplex conversation, modeling the user's and its own audio streams in parallel so it can listen while speaking. As of November 2026, commercial APIs also offer this pattern: OpenAI released its gpt-realtime speech-to-speech model in August 2025 and announced GPT-Realtime-2 in May 2026, and Google offers native-audio models through its Gemini Live API. Cascades remain common because each stage can be swapped, inspected and logged as text, and because they are easier to control for compliance; native models win on latency and expressiveness. The voice-agents lesson covers building both.

Common pitfalls

• Sample-rate mismatch. Feeding 44.1 kHz audio to a 16 kHz model without resampling shifts every frequency and wrecks accuracy without raising an error.
• Stereo, clipping and gain. Mix to mono correctly, and check for clipped (saturated) recordings; models trained on clean levels degrade on very quiet or distorted input.
• Evaluating on clean read speech. Benchmarks of audiobooks overstate accuracy on far-field, accented, overlapping, noisy call-center audio. Build a test set from your real channel.
• Unnormalized WER. Comparing systems with different text normalization makes the numbers meaningless.
• Ignoring hallucinations. Encoder–decoder ASR can invent text during silence or music; gate with VAD and check output length against audio length.
• Consent and privacy. Voice is biometric data in several jurisdictions; recording, storing and cloning voices needs explicit consent and retention policies.

In practice

For transcription, teams usually start with a Whisper-family or commercial ASR model, add VAD, chunking and diarization, and measure WER and entity accuracy on a few hours of their own audio before considering fine-tuning. Custom vocabularies (product names, drug names) are handled with prompting/biasing features, post-correction, or fine-tuning on in-domain audio. For voice agents, they measure end-to-end latency percentiles (not averages), endpointing behavior and interruption handling with real users, and choose between a cascade (control, logging, swappable parts) and a native speech-to-speech model (lower latency, more natural prosody). For audio classification, they fine-tune an AudioSet-pretrained model on a few hundred labeled clips per class and pay close attention to background noise in the deployment environment.