Advanced prompting techniques

Basic prompting gets you a reasonable answer. Professional prompting gets you the same good answer a thousand times in a row, in a format your code can parse, at a cost you can afford, and with evidence that it works. This lesson covers the techniques practitioners use for that: few-shot examples, reasoning prompts, sampling and voting, decomposition, structure, templates, and above all systematic testing. Provider-specific advice below is drawn from official documentation as of November 2026 and is labelled as such, because it changes as models change.

Intuition: briefing a brilliant contractor who has no memory

Imagine hiring an exceptionally capable contractor who has read most of the internet but knows nothing about your company and forgets everything after each job. Everything they need must be in the brief: the role, the goal, the materials (clearly separated from the instructions), examples of what good work looks like, how to format the deliverable, and what to do when something is unclear. Advanced prompting is the craft of writing that brief, and then checking the contractor's work on many jobs, not just one.

In-context learning and few-shot prompting

In-context learning is a model's ability to pick up a task from instructions and examples placed in the prompt, without any change to its weights. Brown et al. (2020) showed with GPT-3 that adding a handful of input-output examples ("few-shot"; none is "zero-shot") could substantially improve performance on many tasks.

What good examples do: they pin down format, tone, label set, edge-case handling and the level of detail far more precisely than prose instructions. Guidelines:

• Relevant: drawn from your real input distribution.
• Diverse: cover different classes and tricky edge cases, so the model does not copy surface features of one example. If every example's answer is "positive", the model learns a bias, not the task.
• Consistent: every example must follow your instructions exactly; models imitate examples more faithfully than they obey instructions.
• Clearly marked: wrap them, for example in <example> tags, so the model knows they are illustrations, not part of the current input.

Anthropic's prompting documentation recommends 3 to 5 examples for best results. OpenAI's reasoning-model guidance suggests trying zero-shot first and adding examples only if needed, keeping them closely aligned with the instructions. These are not contradictory: examples cost tokens and can over-constrain a strong reasoning model, so add them when evaluation shows they help.

Cost check. Five examples of 150 tokens each add 750 input tokens per call. At 1 million calls per month, that is 750 million extra input tokens; at a hypothetical price of 1 US dollar per million input tokens, that is 750 dollars per month. Prompt caching (covered in the next lesson) can cut this sharply because the examples form a static prefix.

Chain-of-thought, and how reasoning models changed it

Chain-of-thought (CoT) prompting asks the model to write intermediate reasoning steps before the answer. Wei et al. (2022) showed that few-shot examples containing worked reasoning greatly improved large models on arithmetic, commonsense and symbolic reasoning; Kojima et al. (2022) found that simply adding "Let's think step by step" helped in the zero-shot setting. Why it works, intuitively: a model produces each token with a fixed amount of computation, so writing out steps gives it more computation and lets later steps condition on earlier results.

It does not help everywhere. A large meta-analysis by Sprague et al. ("To CoT or not to CoT?", ICLR 2025) found the benefits concentrated mainly on math and symbolic or logical reasoning tasks, with much smaller gains elsewhere. Reasoning text also costs output tokens and latency.

Reasoning models changed the advice. Models trained to produce an internal reasoning process before answering (see the reasoning-models lesson) already do this work. As of November 2026:

• OpenAI's reasoning best-practices guide says prompting these models to "think step by step" or "explain your reasoning" is unnecessary, and recommends simple, direct prompts with clear constraints and success criteria.
• Anthropic's documentation says newer Claude models use adaptive thinking controlled by an effort setting, that general instructions such as "think thoroughly" often beat a hand-written step-by-step plan, and that manual chain-of-thought is a fallback for when thinking is off.

Rule of thumb: with a non-reasoning model on a multi-step problem, ask for reasoning (and put it in a separate tagged section so you can strip it before showing the answer). With a reasoning model, describe the goal, constraints and what a good answer looks like, set the reasoning effort, and let it plan.

Self-consistency: sample several, then vote

Self-consistency (Wang et al., 2022) samples several independent reasoning paths at a non-zero temperature and returns the most common final answer. It works because a correct answer can be reached by many routes, while errors tend to scatter.

Worked calculation. Suppose a single sample is correct with probability $p = 0.6$, samples are independent, and (pessimistically) all wrong samples agree on the same wrong answer, so we need a strict majority. With $n = 5$ samples, the vote is correct when at least 3 are correct. With $X \sim \text{Binomial}(5, 0.6)$:

$P(X \ge 3) = \binom{5}{3}0.6^3 0.4^2 + \binom{5}{4}0.6^4 0.4 + \binom{5}{5}0.6^5$

$= 10(0.216)(0.16) + 5(0.1296)(0.4) + 0.07776 = 0.3456 + 0.2592 + 0.07776 = 0.68256$

So accuracy rises from 60% to about 68%, at five times the cost. If wrong answers scatter across different values, plurality voting does even better. If $p < 0.5$ under the same pessimistic assumption, voting makes things worse, and if errors are correlated (the model has a systematic misconception) voting cannot fix them. Use self-consistency for high-value questions with short, comparable final answers (a number, a label), and measure the gain on your eval set.

Decomposition and prompt chaining

Complex tasks often fail because one prompt asks for too much. Decomposition splits a task into sub-tasks; prompt chaining runs them as a pipeline in which each step's output feeds the next. Least-to-most prompting (Zhou et al., 2022) applies the idea within reasoning: first list simpler sub-problems, then solve them in order. Example chain for analysing a contract:

1. Extract all clauses about termination, liability and payment, quoting them verbatim, as JSON.
2. For each clause, classify the risk level against a provided policy.
3. Write a one-page summary for a non-lawyer using only the outputs of step 2.

Benefits: each prompt is simpler and testable on its own; you can use a cheaper model for easy steps; you can validate between steps (does step 1's JSON parse? are the quotes really in the document?). Cost: more calls and more latency, plus errors that propagate down the chain if unchecked.

Role and system prompts

The system prompt (called a developer message in some APIs) holds standing instructions that apply to the whole conversation: role, audience, tone, rules, output format, available tools. A role ("You are a senior tax accountant reviewing small-business returns") focuses vocabulary, depth and tone; Anthropic's docs note that even one sentence of role makes a difference. Roles do not add knowledge the model lacks, and "You are the world's greatest expert" adds nothing measurable compared with a concrete description of the job and the audience.

Structure: delimiters, tags and output formats

Delimiters separate instructions from data. XML-style tags are widely used because they are unambiguous and nest naturally:

<instructions>
Classify the support ticket into exactly one category from <categories>.
If none fits, use "other". Respond with JSON only.
</instructions>
<categories>billing, bug, feature_request, account_access, other</categories>
<ticket>
{{ticket_text}}
</ticket>

Delimiters also reduce prompt injection risk slightly, by making clear which text is data, though they are not a security boundary (see ai-security-risks).

For structured output, specify the exact schema, give an example, and, where the API supports it, use native structured-output or JSON-schema modes, which constrain generation so the output parses. Always validate in code anyway. Put long documents near the top of the prompt and the question or instructions at the end: Anthropic's documentation reports this ordering can improve response quality by up to 30% in its tests on complex multi-document inputs.

Templates and variables

Production prompts are templates: fixed text with variables filled at run time. Keep templates in version control, give them IDs, and log which template version produced each output.

from string import Template

CLASSIFY_V3 = Template('''<instructions>
Classify the ticket into exactly one of: $categories.
Return JSON: {"category": "...", "confidence": "high|low"}.
</instructions>
<ticket>
$ticket
</ticket>''')

def render(ticket: str) -> str:
    cats = "billing, bug, feature_request, account_access, other"
    # Template.substitute does not re-scan inserted text, so a ticket
    # containing "$categories" stays literal.
    return CLASSIFY_V3.substitute(categories=cats, ticket=ticket.strip())

print(render("I was charged twice this month."))

Using string.Template instead of f-strings or str.format avoids clashes with the literal braces in the JSON example.

Negative instructions

"Don't do X" instructions are weaker than positive ones, because they name the unwanted behaviour without saying what to do instead. Prefer "Write in flowing prose paragraphs" over "Don't use bullet points", and explain why a rule exists ("this will be read aloud by a speech engine, so avoid symbols"), which helps the model generalise. Keep hard constraints, but pair each with the desired alternative.

Testing prompts systematically

A prompt that worked on the three inputs you tried is an anecdote. Professionals build a prompt eval set: typically 50 to 500 realistic inputs with expected outputs or grading criteria, including edge cases and past failures. Every prompt change is run against the whole set before it ships.

import json
from collections import Counter

def evaluate(prompt_fn, model_fn, cases):
    # prompt_fn: input -> prompt; model_fn: prompt -> raw text; cases: list of dicts
    results = Counter()
    failures = []
    for case in cases:
        raw = model_fn(prompt_fn(case["input"]))
        try:
            pred = json.loads(raw)["category"]
        except (json.JSONDecodeError, KeyError, TypeError):
            results["parse_error"] += 1
            failures.append((case["input"], raw))
            continue
        ok = pred == case["expected"]
        results["correct" if ok else "wrong"] += 1
        if not ok:
            failures.append((case["input"], pred))
    n = len(cases)
    return {k: v / n for k, v in results.items()}, failures

# Demo with a stand-in model (uses render() from the template example above);
# replace fake_model with a real API call.
def fake_model(prompt):
    return '{"category": "billing"}' if "charged" in prompt else '{"category": "bug"}'

cases = [{"input": "I was charged twice", "expected": "billing"},
         {"input": "App crashes on login", "expected": "bug"},
         {"input": "Please add dark mode", "expected": "feature_request"}]
print(evaluate(render, fake_model, cases))

Track accuracy, parse-error rate, cost and latency per prompt version. Because sampling is stochastic, run each case more than once (or at temperature 0 where supported) and compare variants on the same cases. Differences of a couple of percent on 50 cases are within noise; the evaluation lessons (evaluating-ai-systems) cover confidence intervals and model-graded rubrics.

Using models to help write prompts

Models are useful prompt-writing assistants, provided the eval set stays the judge. Common uses: asking a model to critique a draft prompt for ambiguity or contradictions; asking it to generate additional diverse few-shot examples from two or three hand-written seeds (Anthropic's documentation suggests this), which you then check by hand; asking it to produce adversarial and edge-case inputs for your eval set; and analysing a batch of failures to propose a hypothesis about what the prompt is missing. Automated prompt-optimisation tools take this further by proposing many variants and scoring them on a labelled set. All of these speed up iteration, but they can also overfit: a prompt tuned until it aces 50 cases may have memorised their quirks. Keep a held-out set of cases that you never look at while editing, and confirm the final prompt on it, exactly as you would hold out test data when training a model.

Common pitfalls

• Unrepresentative examples: few-shot examples that are all easy, all one class, or inconsistent with the instructions.
• Forcing chain-of-thought on reasoning models: adds tokens and can degrade results; set the effort level instead.
• Showing reasoning to users by accident: if you ask for reasoning, tag it and strip it before display.
• Unvalidated structured output: even schema-constrained output can be semantically wrong; validate values, not just syntax.
• Prompt edits without regression tests: fixing one failure often breaks two other cases.
• Copying provider tricks across models: assistant prefill, magic phrases and formatting switches differ between providers and model generations; for example, Anthropic's documentation states that prefilling the assistant's response is no longer supported on its newer models. Check the current docs.
• Instruction sprawl: a 3,000-word system prompt of accumulated patches often contradicts itself. Refactor periodically.

In practice

Teams treat prompts as code: templated, versioned, reviewed and tested against an eval set in CI. A typical workflow is: write a clear brief with role, goal, constraints and output schema; add tags to separate data; establish a baseline score on the eval set; then change one thing at a time (add examples, restructure, decompose into a chain, switch model or reasoning effort) and keep only changes that improve the score at acceptable cost. Use self-consistency or larger reasoning effort selectively for high-stakes items, and cheaper models for easy chain steps. Re-run the eval whenever the provider updates the model.