How Large Language Models Work

Why this matters

Chat assistants are the AI most people use every day. Knowing how they produce text explains their strengths, why the same question can get different answers, and why they sometimes invent facts with complete confidence.

A large language model (LLM) is a neural network, trained on a very large amount of text, whose core job is to predict what text comes next.

Step 1: text becomes tokens

Models don't read letters or whole words. They read tokens: chunks of text from a fixed vocabulary, typically tens of thousands to a few hundred thousand entries. Common words are usually one token; rare or long words are split into pieces; spaces and punctuation are often attached to or counted as tokens too. Each token is then turned into a number the network can work with.

An illustrative example (invented for teaching, not the output of any real tokenizer). The sentence:

The weather is unbelievably nice today.

might be split roughly like this:

The · weather · is · un · believ · ably · nice · today · .

That is 6 words but about 9 tokens, because the long word was split into three pieces and the full stop is its own token. Exact splits differ between models, so treat this as a sketch rather than the output of a specific tokenizer.

A useful rule of thumb for English, as published by OpenAI: one token is about four characters or about three-quarters of a word, so 100 tokens is roughly 75 words. Other languages often use more tokens per word. Tokens matter because models have limits on how many they can handle at once, and services charge by the token.

Step 2: predict the next token

Given a sequence of tokens, the model outputs a probability for every token in its vocabulary being the next one. An illustrative example with made-up numbers, for the text "The sky is":

One token is chosen, appended to the text, and the whole process repeats. A long answer is just this loop run hundreds or thousands of times. Nothing is written in advance; the reply is built one token at a time.

Step 3: the transformer and attention

Almost all modern LLMs use an architecture called the transformer, introduced in the 2017 research paper "Attention Is All You Need".

Its key idea is attention. When processing each token, the model looks back over all the earlier tokens and decides how much each one matters for the current step. In "The trophy didn't fit in the suitcase because it was too big", attention helps the model link "it" to "trophy". In plain words: attention lets every word consult every other relevant word, rather than reading strictly left to right and forgetting.

A transformer stacks many layers of attention plus ordinary neural-network layers. Earlier layers tend to capture simple patterns like grammar; later layers capture more abstract relationships. This is a simplification, but a fair one. Transformers also process all the tokens of a training passage in parallel, which made it practical to train on vast amounts of text using many graphics processors (GPUs).

Step 4: how a chatbot is trained, in stages

Pretraining. The model reads a huge collection of text, such as web pages, books and code, and learns to predict the next token. This is where it absorbs grammar, facts, styles and reasoning patterns. The result is a base model: knowledgeable, but it just continues text. Ask it a question and it might write three more questions.

Instruction tuning. The base model is trained further on many examples of instructions paired with good responses, written or curated by people. It learns to behave like an assistant that answers rather than rambles.

Preference training. People compare pairs of model responses and pick the better one. Those choices train the model further toward answers people prefer: more helpful, more honest, less harmful. The best-known version is reinforcement learning from human feedback (RLHF): OpenAI described using it for ChatGPT, training a separate reward model from human rankings and then optimising the chatbot against it. Developers also use variations, including feedback generated with the help of other AI models.

Why the same prompt gives different answers

At each step the model produces probabilities, not a single answer. The software then samples: it picks a token at random, weighted by those probabilities. In the "sky" example, "blue" wins about 62% of the time, but "clear" will sometimes be picked, and the rest of the sentence then follows a different path.

Temperature is a setting that controls how adventurous that sampling is. At low temperature the most likely tokens dominate, giving focused, repeatable answers. At higher temperature the probabilities are flattened, giving more varied and creative, but also more error-prone, text. At a temperature of zero, many systems always pick the top token, though small differences can still creep in for technical reasons.

Why LLMs hallucinate

A hallucination is a fluent, confident statement that is false, such as a made-up citation or a wrong date. It happens for understandable reasons:

• The model is trained to produce plausible text, and a plausible-sounding falsehood can score well.
• Facts are stored diffusely across billions of parameters, not in a lookup table, so rare facts are fuzzy.
• Unless it is connected to search or documents, the model has no way to check a source.
• Training has often rewarded giving an answer over admitting uncertainty, though developers now actively train against this.

Developers work hard to reduce hallucinations, but no current model is free of them.

Going deeper

This section puts numbers on three ideas from above: how scores become probabilities, what the model is trained to minimise, and what attention computes. All numbers are small and illustrative; real models do the same arithmetic with vocabularies of tens of thousands of tokens and vectors with thousands of entries.

From scores to probabilities: softmax and temperature

The network's last layer produces one raw score, called a logit, for every token in the vocabulary. A function called softmax turns logits into probabilities that are positive and add up to 1. With temperature $T$:

$p_i = \frac{e^{z_i / T}}{\sum_{j} e^{z_j / T}}$

Here $z_i$ is the logit of token $i$, $e \approx 2.718$, and the sum in the denominator runs over every token in the vocabulary.

Worked example. Suppose that after "The sky is" only three tokens matter, with logits blue $= 2.0$, clear $= 0.5$, falling $= -0.5$.

At $T = 1$: $e^{2.0} = 7.389$, $e^{0.5} = 1.649$, $e^{-0.5} = 0.607$. The total is $9.644$, so the probabilities are $7.389 / 9.644 = 0.766$, $0.171$ and $0.063$.

Changing the temperature divides every logit by $T$ before the softmax:

Low temperature sharpens the distribution toward the top token; high temperature flattens it. As $T$ approaches 0, sampling becomes greedy decoding: always pick the highest-scoring token.

Many systems also apply top-p (nucleus) sampling: keep only the smallest set of top tokens whose probabilities add up to at least $p$, then renormalise. With $p = 0.9$ at $T = 1$, "blue" alone ($0.766$) is not enough, so "clear" is added ($0.766 + 0.171 = 0.937$). "Falling" is dropped, and the survivors become $0.766 / 0.937 = 0.818$ and $0.182$. This cuts off the long tail of unlikely tokens that can derail a sentence.

The training objective: cross-entropy

During pretraining the correct next token is known, because it is simply the next token in the text. The loss for one position is the cross-entropy, the negative log of the probability the model gave the correct token:

$\mathcal{L} = -\ln p_{\text{correct}}$

If the correct token was "blue" and the model gave it 0.766, the loss is $-\ln 0.766 = 0.266$. Had the model given it only 0.01, the loss would be $-\ln 0.01 = 4.61$. Confident mistakes are punished hard. Pretraining minimises the average of this loss over an enormous number of positions, using exactly the gradient descent described in the previous lesson.

A related number you will see in papers is perplexity, $e$ raised to the average cross-entropy. An average loss of 1.2 gives a perplexity of $e^{1.2} \approx 3.3$: on average the model is about as uncertain as if it were choosing evenly among 3.3 tokens. Lower is better.

Attention in one formula

Each token's vector is projected into three new vectors: a query (what this position is looking for), a key (what this position offers) and a value (the information it passes on). Stacking these for all positions into matrices $Q$, $K$ and $V$, attention is

$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{Q K^{\top}}{\sqrt{d_k}}\right) V$

where $d_k$ is the length of the key vectors. Dividing by $\sqrt{d_k}$ keeps the scores from growing so large that the softmax becomes all-or-nothing.

Worked example (made-up two-number vectors). The query for "it" is $q = (1, 0)$. The keys are trophy $= (2, 0)$ and suitcase $= (0.5, 1)$, so $d_k = 2$.

1. Dot products: $q \cdot k_{\text{trophy}} = 2$ and $q \cdot k_{\text{suitcase}} = 0.5$.
2. Divide by $\sqrt{2} \approx 1.414$: scores $1.414$ and $0.354$.
3. Softmax: $e^{1.414} = 4.11$ and $e^{0.354} = 1.42$, giving weights $0.743$ and $0.257$.
4. If the values are trophy $= (1, 0)$ and suitcase $= (0, 1)$, the output is $0.743 \times (1, 0) + 0.257 \times (0, 1) = (0.743, 0.257)$.

The new representation of "it" is about three-quarters "trophy". In a language model each position may only attend to itself and earlier positions (a causal mask), because it must not peek at the tokens it is trying to predict. Real transformers run many of these attention calculations side by side (multi-head attention), each free to track a different kind of relationship.

Sampling in code

import numpy as np

def softmax(logits, temperature=1.0):
    z = np.asarray(logits, dtype=float) / temperature
    z = z - z.max()                   # subtract the max for numerical stability
    e = np.exp(z)
    return e / e.sum()

tokens = ["blue", "clear", "falling"]
logits = [2.0, 0.5, -0.5]
for T in (0.5, 1.0, 2.0):
    p = softmax(logits, T)
    print(f"T={T}: " + ", ".join(f"{t} {q:.3f}" for t, q in zip(tokens, p)))

rng = np.random.default_rng(0)
draws = rng.choice(tokens, size=1000, p=softmax(logits))
print({t: int((draws == t).sum()) for t in tokens})

The loop reproduces the temperature table. The last two lines draw 1,000 tokens at $T = 1$; the counts come out close to 766, 171 and 63 (here 752, 188 and 60), which is why the same prompt gives different continuations.

Common pitfalls

• Expecting temperature 0 to mean "correct". It makes the choice more repeatable, not more accurate. The most likely token can still be wrong.
• Comparing token counts across models. Different tokenizers split the same text differently, so prices and limits per token are not directly comparable.
• Reading confident wording as calibrated confidence. The tone of a reply tells you little about the probability that it is right.
• Asking letter-level questions without tools. Because models see tokens rather than letters, counting characters or reversing words is surprisingly error-prone.
• Assuming everything in a long prompt is used equally well. Quality can drop when key facts are buried in very long contexts; put the important material clearly and close to the question.
• Using a base model as an assistant. Without instruction tuning it continues text rather than following instructions.

In practice

• Expect variation: if an answer matters, ask again or ask the model to check its own work.
• Use lower-temperature settings, where available, for factual or technical tasks, and higher for brainstorming.
• Give the model the source material (paste the document) instead of relying on its memory; it is far more accurate that way.
• Verify names, numbers, quotes and citations independently before you rely on them.