Deepfakes and Content Provenance

The same generative models that make design and dubbing easier also make it cheap to fabricate a convincing video of a politician, a voice call from a "CEO" or sexual imagery of someone who never consented. Organizations now need two capabilities: deciding whether a piece of media can be trusted, and proving that their own media is authentic. This lesson covers how deepfakes are made, the harms they cause, why detection alone is not enough, how watermarking and C2PA content credentials work, the main laws as of November 2026, and a practical verification checklist.

Intuition: forensics, signatures and seals

Think of three ways to trust a document. A forensic examiner studies the paper and ink for signs of forgery (detection). The author signs the document so anyone can check who issued it and that it was not altered (cryptographic provenance, C2PA). The printer embeds an invisible pattern in the paper itself that survives photocopying (watermarking). Each method fails in different ways: forgers improve, signatures can be stripped off, and patterns exist only if the printer cooperated. Robust verification layers all three with ordinary process controls, such as calling the sender back.

How deepfakes are made

"Deepfake" refers to synthetic or manipulated media, usually of real people, produced with deep learning. Main techniques:

• Face swapping. The classic autoencoder method trains one shared encoder and two decoders, one per identity. Encode a frame of person A, decode with person B's decoder, and you get B's face with A's expression and pose, blended back into the frame. Modern tools use identity embeddings and GAN or diffusion refinement and need only a few photos of the target.
• Face reenactment and lip-sync. Drive a target face's expressions or mouth with another video or with audio (Wav2Lip, 2020, is a widely studied lip-sync model). Used for fake "statements" and for dubbing.
• Fully generated video. Text-to-video and image-to-video diffusion transformers (see the image and video lesson) can animate a single photo of a person or create entire scenes, sometimes with synchronized speech.
• Voice cloning. Zero-shot codec language models and flow-matching TTS clone a voice from seconds of audio; voice conversion changes one speaker's voice into another's in real time.
• Real-time pipelines. Combined face and voice conversion can run live on video calls, which removes the old assumption that a live, interactive call is genuine.
• Cheap fakes. Not all deceptive media is AI-made: slowed footage, misleading captions and out-of-context real photos remain common and are not caught by AI-artifact detectors.

Harms

• Fraud and social engineering. Cloned voices of relatives asking for emergency money; impersonated executives instructing staff to wire funds or share credentials, sometimes on live video calls; attacks on voice-based authentication; fake job candidates in remote interviews.
• Non-consensual intimate imagery (NCII). Sexual deepfakes and "nudification" of ordinary photos, overwhelmingly targeting women and girls, used for harassment, extortion ("sextortion") and abuse. Synthetic sexual imagery of minors is child sexual abuse material and illegal in many jurisdictions.
• Misinformation. Fabricated statements by candidates or officials, fake footage of disasters or conflicts, and synthetic "evidence" in disputes.
• The liar's dividend. Once fakes are common, real recordings can be dismissed as fake. Provenance for authentic media matters as much as detection of fakes.
• Reputational and legal harm to companies through fake endorsements, fake executive statements and market-moving hoaxes.

Detection approaches and their limits

Artifact-based classifiers are trained to separate real from fake media using visual cues (blending boundaries, inconsistent lighting, unnatural eye or teeth detail, frequency-domain fingerprints left by upsampling layers) or audio cues (spectral artifacts, unnatural breathing or phase). Datasets such as FaceForensics++ (Rössler et al., 2019) and the Deepfake Detection Challenge (Dolhansky et al., 2020) drove this work.

Physiological and semantic cues check blinking, pulse-induced skin color changes, lip shapes against phonemes, head-pose consistency, and whether shadows and reflections agree with scene geometry.

Behavioral and contextual checks ask whether the content is plausible, where it first appeared, and whether the account posting it has a history.

Why detection is not a solution on its own

1. Generalization. Detectors learn the fingerprints of the generators in their training data. New architectures, especially diffusion models, often evade them. Results in the DFDC illustrated this: top models scored much lower on a held-out "black-box" set than on the public data.
2. Laundering. Re-compression, resizing, screenshots, re-recording a screen, adding noise or filters destroy many artifacts. Social platforms do much of this automatically.
3. Adversarial evasion. An attacker with access to a detector can optimize fakes against it.
4. Base rates. Most media checked is real, so even an accurate detector produces many false alarms.

Worked example: base rates. A detector catches 95% of deepfakes (true-positive rate) and wrongly flags 2% of real videos (false-positive rate). On a platform where 1% of uploaded videos of real people are deepfakes, out of 10,000 videos:

• Deepfakes: 100; flagged: $0.95 \times 100 = 95$.
• Real: 9,900; flagged: $0.02 \times 9{,}900 = 198$.
• Precision $= 95 / (95 + 198) = 0.324$.

About two-thirds of flags are false. If prevalence rises to 20% (for instance, among videos already reported by users), precision becomes $0.95 \times 0.2 / (0.95 \times 0.2 + 0.02 \times 0.8) = 0.19/0.206 = 0.922$. The lesson: use detectors for triage with human review, report their scores as probabilities not verdicts, and expect the operating point to depend on where in your pipeline they run.

Watermarking

A watermark is a signal embedded in content at generation time that is imperceptible to people but detectable by an algorithm with the right key or model.

Image, video and audio watermarks are embedded by a trained encoder into pixels or samples and recovered by a decoder network; training includes simulated edits (compression, crops, color shifts) so the mark survives them. Research examples include Stable Signature, which fine-tunes the latent decoder of an image model so every output carries a mark, and Tree-Ring watermarks, which embed a pattern in the initial noise of a diffusion model. AudioSeal (Meta) localizes watermarked segments in audio.

Text watermarks bias token sampling. In the scheme of Kirchenbauer et al. (2023), a hash of the previous token seeds a pseudorandom split of the vocabulary into a "green list" (fraction $\gamma$) and a red list, and generation nudges probabilities toward green tokens. A detector who knows the key counts green tokens $|s|_G$ among $T$ tokens and computes

Under the null hypothesis (unwatermarked text), green tokens occur at rate $\gamma$, so $z$ is approximately standard normal. Example: $T = 200$, $\gamma = 0.25$, 82 green tokens. Expected 50, standard deviation $\sqrt{200 \times 0.25 \times 0.75} = 6.12$, so $z = 32 / 6.12 = 5.23$, far beyond a typical threshold of 4. Paraphrasing or translating the text, however, can wash the signal out, and short texts carry too few tokens for a reliable decision.

Google SynthID

SynthID is Google DeepMind's watermarking technology. It launched in August 2023 for images from Google's Imagen model and was extended to text (Gemini), audio (Lyria) and video (Veo). Google open-sourced its text watermarking method, which works by adjusting token selection as described above. In May 2025 Google said SynthID had marked over 10 billion pieces of content and launched SynthID Detector, a portal where users can upload images, video, audio or text to check for the watermark, including highlighting watermarked regions in edited images. Google reports the image watermark survives common changes such as JPEG compression, resizing, color changes and light cropping, but describes SynthID as not infallible: heavy edits can weaken it, it identifies only content from tools that embed it, and a determined adversary may remove it.

General limitations of watermarking: it requires generator cooperation (open-weight models can be run without it); there is no single universal detector across vendors; robustness trades off against imperceptibility and capacity; and false accusations are possible if detection thresholds are poorly calibrated. Absence of a watermark proves nothing.

C2PA content credentials

The Coalition for Content Provenance and Authenticity (C2PA), formed in 2021 by technology and media companies, publishes an open technical standard for attaching tamper-evident provenance to media. Consumer-facing implementations are branded Content Credentials.

How it works

1. Manifest. When a camera, editing tool or AI generator creates or edits a file, it builds a manifest containing assertions: who or what created it, which tool, actions taken (captured, cropped, generated with AI), ingredients (source files and their manifests) and optional metadata.
2. Hard binding. The manifest includes a cryptographic hash of the content, so any pixel-level change breaks the match.
3. Signature. The claim is digitally signed with the creator tool's private key, backed by an X.509 certificate. Validators check the signature, the certificate chain against a trust list, and the hash.
4. Storage. The manifest store is embedded in the file (in a JUMBF container for JPEG, PNG, MP4 and other formats) and/or held in the cloud and referenced by the file.
5. Chain of provenance. Each edit adds a new manifest that references the previous one as an ingredient, producing a verifiable history.
6. Soft binding and recovery. Because platforms often strip metadata, a manifest can be linked to an invisible watermark or a perceptual fingerprint, letting a verifier look up the credential from a repository even after stripping.

What C2PA proves and does not prove. A valid credential proves that a holder of a certain signing key made specific claims about this exact content and that it has not changed since. It does not prove the content is true, and an unsigned file is not evidence of fakery: most media today has no credentials. Trust depends on which signers you trust.

Adoption (as of November 2026)

C2PA released version 2.3 of the specification in February 2026, adding support for live video in broadcast and streaming and for more file types (including plain text documents, OGG Vorbis audio and large AVI video files). The coalition reports that more than 6,000 members and affiliates have live applications of Content Credentials, and it launched a Conformance Program in late 2025 so products displaying credentials handle them correctly. Support spans some cameras and phones, creative software, AI image generators and a number of social and news platforms, but end-to-end coverage remains uneven: many upload pipelines still strip or ignore manifests, and few consumers check them. Regulation (below) is a major driver of adoption.

Laws (general information as of November 2026, not legal advice)

• United States: TAKE IT DOWN Act. Signed into law on May 19, 2025. It makes it a federal crime to knowingly publish non-consensual intimate imagery, explicitly including AI-generated "digital forgeries," with heavier penalties where minors are involved. It also requires covered platforms (public websites and apps that primarily host user-generated content or regularly host intimate imagery) to provide a notice-and-removal process and remove reported NCII within 48 hours of a valid request. Platforms had one year to comply, so that obligation took effect on May 19, 2026. The Federal Trade Commission enforces the platform requirements; violations are treated like violations of an FTC rule, with civil penalties of up to 53,088 US dollars per violation, and the FTC runs a complaint site (TakeItDown.ftc.gov) for people whose removal requests are ignored.
• European Union: AI Act Article 50 transparency duties. These became applicable on 2 August 2026, and the Digital Omnibus amendments, in force since 27 July 2026, did not postpone them. Providers of AI systems that interact with people must disclose that users are dealing with AI unless obvious. Providers of systems generating synthetic audio, image, video or text must mark outputs in a machine-readable, detectable format. Deployers who publish deepfakes of real people, places or events must disclose that the content is artificially generated or manipulated (with lighter rules for evidently artistic or satirical work). Generative systems already on the market before 2 August 2026 have until 2 December 2026 to implement machine-readable marking. The Commission published non-binding guidelines on Article 50 on 20 July 2026, and in July 2026 it assessed a voluntary code of practice on transparency of AI-generated content as an adequate way to meet the marking and disclosure duties. Breaches can draw fines of up to €15 million or 3% of worldwide annual turnover. The Omnibus also adds a prohibition on "nudification" systems and systems capable of generating child sexual abuse material, applying from 2 December 2026.
• Elsewhere. Many US states have their own laws on sexual deepfakes, election deepfakes and digital replicas of voice and likeness, and other countries have enacted labelling or deep-synthesis rules. Requirements and effective dates change frequently; confirm current law with counsel for each market.

Code: detecting a green-list text watermark

This standard-library simulation implements the Kirchenbauer-style scheme: a keyed hash decides green tokens, a "watermarked" generator prefers them, and the detector computes the $z$-score.

import hashlib, math, random

GAMMA = 0.25                      # fraction of vocabulary on the green list
VOCAB = [f"w{i}" for i in range(1000)]

def is_green(prev_token, token, key="secret-key"):
    h = hashlib.sha256(f"{key}|{prev_token}|{token}".encode()).digest()
    return int.from_bytes(h[:8], "big") / 2**64 < GAMMA

def generate(n, watermark, bias=0.9, seed=0):
    rng, out = random.Random(seed), ["<s>"]
    for _ in range(n):
        cands = rng.sample(VOCAB, 20)              # stand-in for top-k candidates
        greens = [c for c in cands if is_green(out[-1], c)]
        if watermark and greens and rng.random() < bias:
            pick = rng.choice(greens)
        else:
            pick = rng.choice(cands)
        out.append(pick)
    return out[1:]

def z_score(tokens):
    prev, green = ["<s>"] + tokens[:-1], 0
    for p, t in zip(prev, tokens):
        green += is_green(p, t)
    T = len(tokens)
    return (green - GAMMA * T) / math.sqrt(T * GAMMA * (1 - GAMMA)), green

print("watermarked:", z_score(generate(200, True)))
print("plain:      ", z_score(generate(200, False)))

On one run the watermarked sequence had 193 green tokens of 200 ($z \approx 23$) and the plain one had 45 ($z \approx -0.8$). Real schemes nudge logits softly to preserve quality, so green fractions are lower and longer texts are needed. Try replacing every third token randomly to see how editing erodes $z$.

Practical verification checklist for organizations

Before an incident (prepare):

1. Remove media as an authorization factor. No payment, credential reset, access change or data release on the strength of a voice, video call or message alone. Require call-back to a number from your directory, a second approver, and out-of-band confirmation in a separate channel.
2. Agree on verification phrases or challenge questions for executives and finance teams, and rehearse them.
3. Sign what you publish. Attach C2PA credentials to official images, video and press materials; publish where the public can verify them; keep originals.
4. Mark what you generate. If you deploy generative AI, enable watermarking and machine-readable labels, and disclose synthetic content where the EU AI Act or other law requires.
5. Train staff with realistic examples of voice and video impersonation, especially finance, HR, help desk and executive assistants.
6. Prepare a takedown process that can meet a 48-hour removal deadline for NCII reports if you host user content, and an escalation path to platforms for impersonation of your executives or brand.

When suspicious media arrives (verify):

1. Pause and verify through a separate channel you already trust, before acting.
2. Check provenance: inspect C2PA credentials with a validator, check for known watermarks (for example SynthID Detector for Google-generated content), and run reverse image and video searches for earlier versions.
3. Check context: first appearance, uploader history, metadata, whether other reliable sources corroborate the event, weather and shadows against the claimed time and place.
4. Use detectors for triage only, with documented thresholds and human review; record scores, not verdicts.
5. Preserve evidence: keep original files, URLs, hashes and timestamps for investigators and legal action.
6. Respond: notify affected people, report to platforms and law enforcement where appropriate, and issue a signed correction if your organization was impersonated.

Common pitfalls

• Treating a detector score as proof in either direction. Scores are probabilistic and degrade on new generators and compressed media.
• Assuming no watermark or credential means fake (or that one means true). Most authentic media carries neither; a credential proves origin and integrity, not truthfulness.
• Stripping your own credentials. Many CMSs and image optimizers remove metadata; test your publishing pipeline end to end.
• Relying on training alone. People cannot reliably spot good fakes; process controls are what stop fraud.
• Ignoring legal deadlines. Platform removal duties and AI-labelling duties carry real penalties; assign owners and audit compliance.

In practice

Mature organizations treat synthetic media as a security and trust problem, not a curiosity. Finance and identity processes assume any voice or face can be faked. Communications teams sign official media with C2PA and monitor for impersonation. Trust-and-safety teams combine hash matching of known abusive content, provenance checks, detector triage and human review, with documented removal SLAs. Product teams that ship generative features build in watermarking, labelling and consent checks by design, and track the evolving legal landscape in each market they serve.