Audio, Voice and Music Generation

Synthetic speech now reads audiobooks, voices navigation and customer-service agents, dubs video into other languages and gives people who have lost their voice a way to speak in something close to their own. Music generators produce full songs from a text prompt. The same technology also enables voice-clone fraud. This lesson explains how modern audio generation works, from codecs that turn sound into tokens to real-time conversational voice models, and how to evaluate and deploy it responsibly.

Why audio is hard

Audio is a long, dense signal. Speech is commonly sampled at 16 to 24 kHz and music at 44.1 or 48 kHz, so ten seconds of 24 kHz audio is 240,000 numbers. Generating each sample autoregressively, as WaveNet (van den Oord et al., 2016) did, produced excellent quality but was slow. Humans are also unforgiving listeners: small phase errors sound metallic, tiny timing glitches sound like clicks, and wrong prosody (rhythm, stress and intonation) makes speech sound robotic even when every word is right.

The field's answer has been to change the representation before generating. There are two main choices:

1. Mel spectrograms: a time-frequency picture of the sound with frequencies warped to the perceptual mel scale, $m = 2595 \log_{10}(1 + f/700)$, where $f$ is frequency in Hz (by construction, 1000 Hz maps to about 1000 mel). A vocoder converts the spectrogram back to a waveform.
2. Neural codec tokens: a learned compressor turns audio into a short sequence of discrete integers, which a language model can generate just like text.

Neural audio codecs and audio tokens

A neural audio codec is an autoencoder with a quantized bottleneck. SoundStream (Zeghidour et al., 2021) and EnCodec (Défossez et al., 2022) established the standard recipe:

• A convolutional encoder turns the waveform into a sequence of latent vectors at a low frame rate (EnCodec at 24 kHz uses 75 frames per second).
• A residual vector quantizer (RVQ) turns each latent vector into several integer codes.
• A decoder reconstructs the waveform, trained with reconstruction losses on waveform and spectrograms plus adversarial (GAN) discriminator losses, which make the output sound sharp rather than muffled.

Residual vector quantization

A single codebook of $K$ entries can only represent $K$ distinct vectors, too few for high fidelity. RVQ uses $N_q$ codebooks in sequence. The first quantizes the latent $z$; each later stage quantizes what the previous stages missed:

where $c^{(i)}_k$ is entry $k$ of codebook $i$, and $k_i$ is the integer token emitted by stage $i$. Coarse codebooks capture the gist (speaker, phonetic content), fine ones add detail. A useful property is bitrate scalability: drop the last codebooks and you still get intelligible, lower-quality audio.

Worked example: bitrate and token rate. EnCodec at 24 kHz uses codebooks of 1,024 entries, so each code carries $\log_2 1024 = 10$ bits. At 75 frames per second with 8 codebooks:

compared with $24{,}000 \times 16 = 384$ kbps for raw 16-bit audio, a 64× reduction. For a language model, the cost is the token count: $75 \times 8 = 600$ tokens per second, so 10 seconds is 6,000 tokens. That is still long, which is why models use tricks: predict the first (coarse) codebook autoregressively and the rest in parallel or with a smaller model; interleave codebooks with a delay pattern so one transformer step emits all codebooks with offsets (MusicGen, Copet et al., 2023); or use lower frame rates (newer speech codecs operate around 12.5 frames per second).

Semantic vs acoustic tokens. AudioLM (Borsos et al., 2022) combined two token types: semantic tokens from a self-supervised speech model (capturing content and long-range structure) and acoustic tokens from a codec (capturing voice and recording detail). Generating semantic tokens first and acoustic tokens second improved long-range coherence. Several later codecs distill semantic information directly into their first codebook.

Text-to-speech (TTS) architectures

Two-stage pipelines

Classic neural TTS splits the job:

1. Text front end: normalize text ("Dr." to "doctor", "£5" to "five pounds"), convert to phonemes (grapheme-to-phoneme), and sometimes predict prosody features.
2. Acoustic model: phonemes to mel spectrogram. Tacotron 2 (Shen et al., 2018) used an attention-based sequence-to-sequence model; FastSpeech 2 (Ren et al., 2020) is non-autoregressive, predicting durations, pitch and energy explicitly, which made it faster and more robust (no skipped or repeated words from attention failures).
3. Vocoder: mel to waveform. HiFi-GAN (Kong et al., 2020) is a GAN vocoder that runs much faster than real time.

VITS (Kim et al., 2021) merged these stages into one end-to-end model combining a VAE, a normalizing flow and adversarial training.

Codec language models

VALL-E (Wang et al., 2023) reframed TTS as language modeling: condition a transformer on phonemes and the codec tokens of a short acoustic prompt (about 3 seconds of the target speaker) and generate codec tokens that continue in that voice. This gave zero-shot voice cloning: no fine-tuning per speaker. It also inherits language-model failure modes: occasional repeated or skipped words, and sensitivity to sampling settings.

Diffusion and flow-matching TTS

Another family generates mel spectrograms or continuous codec latents with diffusion or flow matching, often non-autoregressively over the whole utterance with a separately predicted total duration. Voicebox (Le et al., 2023) used flow matching with speech infilling as its training task, which supports editing a word in the middle of a recording. Many current open TTS systems combine an autoregressive model for content and timing with a flow-matching stage for acoustic detail.

Comparison

Voice cloning: consent and safety

Zero-shot cloning from seconds of audio means anyone with a public video can have their voice imitated. Documented harms include impersonation scams targeting family members and businesses, bypassing voice-based authentication, harassment and non-consensual content, and political disinformation. Responsible practice:

• Consent and verification. Clone only voices with explicit, recorded consent. Some providers require the speaker to read a randomly generated verification sentence, which a pre-recorded clip cannot satisfy.
• Policy controls. Block cloning of public figures, keep audit logs, and rate-limit.
• Provenance. Embed inaudible watermarks in generated audio (for example Meta's AudioSeal or Google's SynthID for audio) and attach C2PA content credentials where formats allow. Watermarks help only for content from cooperating generators.
• Organizational defenses. Stop treating voice as proof of identity: use call-backs to known numbers, shared passphrases and multi-person approval for payments. The deepfakes lesson covers this in depth.
• Legal exposure. Laws on likeness rights, fraud and AI disclosure vary by jurisdiction and are changing; check current rules before shipping a voice product.

Music generation

Music is harder than speech: longer coherent structure (verses, choruses, key and tempo), polyphony, stereo, wide bandwidth and lyrics. Approaches mirror the rest of the field:

• Token language models over codecs. OpenAI's Jukebox (Dhariwal et al., 2020) used hierarchical VQ-VAE tokens with transformers. MusicLM (Agostinelli et al., 2023) extended AudioLM with text conditioning through a joint music-text embedding. MusicGen (Copet et al., 2023) is a single-stage transformer over EnCodec tokens using the delay pattern, conditioned on text and optionally a melody.
• Latent diffusion. Stable Audio (Evans et al., 2024) runs diffusion in a learned audio latent space and conditions on start time and total duration so outputs can be a requested length.
• Symbolic generation. Generating MIDI or scores with a sequence model, then rendering with instruments. It offers precise editability, but quality depends on the rendering.

Commercial song generators with vocals are widely used as of November 2026 (Google's Lyria is one example that carries SynthID watermarks), but training on copyrighted recordings is a contested legal issue, and some licensing arrangements between AI music companies and rights holders are evolving. For commercial use, check the provider's license terms, training-data disclosures and whether outputs can be claimed as your copyright.

Evaluation

Mean opinion score (MOS)

The reference method is listening tests. In a MOS test (ITU-T Recommendation P.800 describes the absolute category rating procedure), listeners rate each sample from 1 (bad) to 5 (excellent), and scores are averaged. Variants: CMOS (comparative, rate B relative to A from −3 to +3), SMOS (speaker similarity), and MUSHRA for finer quality distinctions with hidden references and anchors.

Worked example. Ten ratings for one system: 4, 5, 4, 3, 4, 5, 4, 4, 3, 4.

1. Mean: $40 / 10 = 4.0$.
2. Deviations from 4: 0, 1, 0, −1, 0, 1, 0, 0, −1, 0; squared sum $= 4$; sample variance $= 4/9 = 0.444$; standard deviation $s = 0.667$.
3. 95% confidence half-width with a $t$ critical value of 2.262 (9 degrees of freedom): $2.262 \times 0.667 / \sqrt{10} = 0.477$.
4. Report MOS $= 4.00 \pm 0.48$.

If a competing system scored $4.20 \pm 0.45$ from a similar test, the intervals overlap heavily and you cannot claim it is better. Real studies use many more listeners and utterances, randomize order, include hidden references, screen inattentive raters, and account for per-listener bias with mixed-effects models. MOS values are not comparable across different tests or listener pools.

Objective metrics

• Intelligibility: transcribe the output with a strong speech recognizer and compute word error rate against the input text.
• Speaker similarity: cosine similarity between speaker-verification embeddings of the prompt and the output.
• Predicted MOS: neural MOS predictors (UTMOS and similar) are useful for regression testing but can be gamed.
• Music: Fréchet Audio Distance (FAD), an FID analogue on audio embeddings, and CLAP score, a CLIP-score analogue for text-audio alignment.
• Latency: time to first audio and real-time factor (processing time divided by audio duration; below 1 means faster than real time).

Real-time voice models

Conversational voice agents need first audio within a few hundred milliseconds. There are two architectures.

Cascaded pipelines chain speech recognition, a text LLM and streaming TTS. They are modular, easy to debug, and let you swap components, but lose paralinguistic information (tone, emotion, laughter) at the ASR step, and latencies add up. A rough budget: end-of-speech detection 200 to 500 ms, ASR finalization 100 to 300 ms, LLM time to first token 200 to 500 ms, TTS time to first audio 100 to 300 ms, so 0.6 to 1.6 s in total unless each stage streams aggressively.

Speech-to-speech (native audio) models consume and emit audio tokens directly within one model. Moshi (Défossez et al., 2024, from Kyutai) is a published example of a full-duplex design: it models the user's and its own audio streams simultaneously, so it can listen while speaking and handle interruptions and back-channels ("mm-hmm"), and it predicts aligned text tokens as an "inner monologue" to keep content coherent. Several commercial providers offer speech-to-speech APIs as of November 2026; their names and capabilities change quickly, so check current documentation. Trade-offs: lower latency and richer prosody, but harder to inspect, constrain and evaluate than a text pipeline, and tool use and guardrails must operate on audio-native outputs.

Code: residual vector quantization from scratch

This builds a 4-stage RVQ with k-means codebooks on stand-in latent vectors (in a real codec, a neural encoder produces them and codebooks are learned jointly). It shows how each stage lowers the error and how tokens are produced.

import numpy as np
from sklearn.cluster import KMeans

rng = np.random.default_rng(0)
# Stand-in for encoder outputs: 5,000 frames of 8-dim latent vectors
latents = rng.normal(size=(5000, 8)) @ rng.normal(size=(8, 8))

def train_rvq(x, n_stages=4, codebook_size=64):
    codebooks, residual = [], x.copy()
    for _ in range(n_stages):
        km = KMeans(n_clusters=codebook_size, n_init=1, random_state=0).fit(residual)
        codebooks.append(km.cluster_centers_)
        residual = residual - km.cluster_centers_[km.labels_]
    return codebooks

def encode(x, codebooks):
    codes, residual = [], x.copy()
    for cb in codebooks:
        d = ((residual[:, None, :] - cb[None]) ** 2).sum(-1)
        idx = d.argmin(1)
        codes.append(idx)
        residual = residual - cb[idx]
    return np.stack(codes, 1)            # (frames, n_stages) integer tokens

def decode(codes, codebooks):
    return sum(cb[codes[:, i]] for i, cb in enumerate(codebooks))

cbs = train_rvq(latents)
codes = encode(latents, cbs)
for k in range(1, 5):
    err = ((latents - decode(codes[:, :k], cbs[:k])) ** 2).mean()
    print(f"{k} codebooks: MSE {err:.3f}, bits/frame {k * 6}")

With 64-entry codebooks each code is 6 bits. On one run the mean-squared error fell from about 1.64 with one codebook to about 0.10 with four, illustrating bitrate scalability. The codes array is exactly what an audio language model would be trained to predict.

Common pitfalls

• Judging by cherry-picked demos. TTS and music systems vary a lot by seed and prompt. Evaluate on a fixed, diverse set of texts including numbers, names, abbreviations, long sentences and other languages.
• Ignoring text normalization. Many "model errors" are front-end errors: dates, currencies, acronyms and homographs ("read", "lead").
• Single-number MOS claims. Without confidence intervals, listener counts and test design, a MOS is not evidence. Never compare MOS across separate studies.
• Latency measured end to end only once. Measure each stage, measure the 95th percentile not just the mean, and test under realistic network conditions.
• Treating consent as a checkbox. Store proof of consent for each cloned voice, allow revocation, and prevent use of uploaded voices outside the consented purpose.

In practice

For a product voice, teams typically license a TTS API or fine-tune an open model on a few hours of consented studio recordings, build a text-normalization layer with domain lexicons, and run regression tests that combine ASR word error rate, speaker similarity and periodic listening tests. For conversational agents, start with a cascaded pipeline when you need tools, logging and guardrails, and evaluate speech-to-speech models when natural turn-taking matters most. For music, check licensing first. In every case, watermark or label generated audio and design business processes so a convincing voice is never sufficient authorization.