Natural Language Processing: From Pipelines to Pretrained Models

Natural language processing (NLP) is the study and engineering of systems that read, classify, extract from, transform and generate human language. It runs spam filters, search engines, contract review, support-ticket routing, clinical coding and, now, chat assistants. Large language models (LLMs) have absorbed many NLP tasks, but a working engineer still needs the older ideas: they explain why LLMs work, they are the right tool surprisingly often, and their metrics are still how much of the field reports results.

The picture in your head

Imagine teaching a new clerk to process mail. First you teach them to split text into words and recognize that "running" and "ran" are forms of "run." Then grammar: which word is the verb, which noun it acts on. Then names: "Apple" the company versus "apple" the fruit. For decades NLP built exactly this clerk, one hand-designed stage at a time. The deep-learning era replaced the stages with representations: vectors that encode what a word means in context, learned from huge amounts of raw text. LLMs are that idea taken to its limit: one model whose internal representations are good enough to do every stage, and the task itself, from an instruction.

The classic pipeline

A traditional pipeline (the shape of tools such as spaCy, NLTK and Stanford CoreNLP) runs these steps in order, each consuming the previous one's output.

1. Tokenization: splitting text into tokens. Even this is not trivial: "don't", "U.S.", "e-mail", URLs, and languages without spaces (Chinese, Japanese) all need rules or models. LLMs use subword tokenization (byte-pair encoding, WordPiece, unigram LM) so that any string can be represented by a fixed vocabulary of frequent pieces.
2. Normalization: lowercasing, Unicode normalization, removing markup.
3. Stemming or lemmatization. A stemmer chops affixes with rules (the Porter stemmer maps "connection", "connected", "connecting" to "connect", but also maps "university" and "universe" to "univers"). A lemmatizer uses a dictionary and the part of speech to return the dictionary form ("better" as an adjective becomes "good"; "saw" as a verb becomes "see"). Stemming is fast and crude; lemmatization is slower and more accurate.
4. Part-of-speech (POS) tagging: labeling each token as noun, verb, adjective and so on.
5. Named-entity recognition (NER): finding spans that name people, organizations, locations, dates, amounts. Usually framed as sequence labeling with BIO tags: B-ORG marks the first token of an organization, I-ORG a continuation, O anything outside an entity.
6. Parsing: recovering sentence structure. Dependency parsing links each word to its grammatical head ("Apple ← nsubj ← acquired → dobj → startup"), which is what lets a system answer "who acquired whom."

import spacy   # pip install spacy && python -m spacy download en_core_web_sm

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple quietly acquired a London startup for $50 million in 2024.")

for t in doc:
    print(f"{t.text:10s} lemma={t.lemma_:10s} pos={t.pos_:6s} dep={t.dep_:8s} head={t.head.text}")
for ent in doc.ents:
    print(ent.text, "->", ent.label_)     # e.g. Apple -> ORG, London -> GPE, $50 million -> MONEY

Statistical language models: n-grams

A language model assigns a probability to a sequence of tokens $w_1, \dots, w_T$. By the chain rule,

An n-gram model makes the Markov assumption that only the previous $n - 1$ tokens matter. For a bigram model ($n = 2$), the maximum-likelihood estimate is a ratio of counts:

where $C(\cdot)$ counts occurrences in the training corpus. Unseen pairs get probability zero, which makes any sentence containing them impossible, so we smooth. Add-$k$ (Laplace when $k = 1$) smoothing uses

with $V$ the vocabulary size. (Practical systems used better schemes such as Kneser–Ney; add-one is shown because it is easy to compute by hand.)

Language models are compared with perplexity, the exponentiated average negative log-likelihood per predicted token: $\text{PPL} = \exp\big(-\frac{1}{N}\sum_{t} \log P(w_t \mid \text{context})\big)$. Intuitively, a perplexity of 4 means the model is as uncertain as if it were choosing uniformly among 4 tokens at each step. Lower is better, and it is only comparable between models that share a tokenizer and test set.

Worked example. Training corpus of three sentences, with start and end markers:

• <s> i like green tea </s>
• <s> i like black coffee </s>
• <s> you like green tea </s>

The vocabulary is {<s>, </s>, i, you, like, green, black, tea, coffee}, so $V = 9$. "like" appears 3 times and is followed by "green" twice. The MLE estimate is $P(\text{green} \mid \text{like}) = 2/3$. With add-one smoothing it is $(2 + 1)/(3 + 9) = 3/12 = 0.25$, because probability mass has been spread to the eight continuations never seen after "like." Running the code below, the familiar sentence "i like green tea" gets perplexity about 3.8, the novel-but-grammatical "you like black coffee" about 5.4, and the scrambled "tea like i" about 11.5: the model has learned something about word order from twelve bigrams.

import math
from collections import Counter

corpus = ["<s> i like green tea </s>", "<s> i like black coffee </s>",
          "<s> you like green tea </s>"]
tokens = [s.split() for s in corpus]
unigrams = Counter(w for s in tokens for w in s)
bigrams = Counter((a, b) for s in tokens for a, b in zip(s, s[1:]))
V = len(unigrams)

def p_bigram(prev, w, k=1.0):                     # add-k smoothed P(w | prev)
    return (bigrams[(prev, w)] + k) / (unigrams[prev] + k * V)

def perplexity(sentence, k=1.0):
    words = sentence.split()
    lp = sum(math.log(p_bigram(a, b, k)) for a, b in zip(words, words[1:]))
    return math.exp(-lp / (len(words) - 1))

for s in ["<s> i like green tea </s>", "<s> you like black coffee </s>", "<s> tea like i </s>"]:
    print(f"{s:32s} perplexity = {perplexity(s):.2f}")

A transformer LLM is still a language model in exactly this sense; it simply conditions on thousands of previous tokens through learned representations instead of counting the last one or two.

Representing documents: TF-IDF

For retrieval and classification, a classic representation is the bag of words weighted by TF-IDF. For term $t$ in document $d$ from a collection of $N$ documents:

where $\text{tf}(t, d)$ is how often $t$ occurs in $d$ (raw or length-normalized) and $\text{df}(t)$ is the number of documents containing $t$. Words that appear everywhere ("the") get weight near zero; words concentrated in few documents get high weight.

Worked example. $N = 3$ documents: $d_1$ = "the cat sat on the mat", $d_2$ = "the dog sat on the log", $d_3$ = "cats and dogs play". Using length-normalized tf and the natural log, in $d_1$ (6 tokens):

• "cat": $\text{tf} = 1/6$, $\text{df} = 1$, idf $= \ln 3 = 1.099$, weight $= 0.183$.
• "the": $\text{tf} = 2/6$, $\text{df} = 2$, idf $= \ln 1.5 = 0.405$, weight $= 0.135$.
• "sat": $\text{tf} = 1/6$, $\text{df} = 2$, weight $= 0.068$.

"cat" ends up the most distinctive term even though "the" occurs twice as often. Note that "cat" and "cats" are different features here, which is exactly the gap stemming or lemmatization fills. Libraries vary the formula (scikit-learn adds smoothing, $\ln\frac{1+N}{1+\text{df}} + 1$, then L2-normalizes each document vector), so hand calculations will not match library output exactly. BM25, the default ranking function in many search engines, is a refined TF-IDF with term-frequency saturation and document-length normalization.

A TF-IDF plus logistic-regression classifier is the baseline every text-classification project should run first:

from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

texts = ["refund not received", "love the new update", "app crashes on login",
         "great support team", "charged twice this month", "works perfectly"]
labels = ["billing", "praise", "bug", "praise", "billing", "praise"]

clf = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True),
                    LogisticRegression(max_iter=1000))
clf.fit(texts, labels)
print(clf.predict(["I was charged for a refund that never arrived"]))

Word embeddings

Bag-of-words treats "car" and "automobile" as unrelated. The distributional hypothesis ("you shall know a word by the company it keeps," usually attributed to J. R. Firth) says words in similar contexts have similar meanings, so we can learn dense vectors from context.

• word2vec (Mikolov, Chen, Corrado and Dean, 2013) trains a shallow network either to predict a word from its neighbors (CBOW) or neighbors from a word (skip-gram). A companion paper the same year added negative sampling: for each real (word, context) pair, also sample a few random "noise" words and train a logistic classifier to tell real from noise, maximizing $\log \sigma(\mathbf{u}_c^\top \mathbf{v}_w) + \sum_{k} \log \sigma(-\mathbf{u}_{n_k}^\top \mathbf{v}_w)$, where $\mathbf{v}_w$ is the word's vector, $\mathbf{u}_c$ the context vector and $n_k$ the noise words.
• GloVe (Pennington, Socher and Manning, 2014) fits vectors so that their dot products approximate the logarithm of global word co-occurrence counts, combining count-based and prediction-based ideas.

These vectors famously encode analogies approximately (vector("king") − vector("man") + vector("woman") lands near vector("queen")), and they also encode the social biases of their training text, which became one of the first well-studied fairness problems in NLP. Their key limit: one vector per word type, so "bank" in "river bank" and "bank loan" is the same point.

Contextual embeddings

• ELMo (Peters et al., 2018) ran deep bidirectional LSTM language models over the sentence and used their internal states as word representations, so each occurrence of "bank" gets its own vector.
• BERT (Devlin et al., 2018 preprint; NAACL 2019) used a transformer encoder pretrained with masked language modeling (hide about 15% of tokens and predict them from both sides) plus a next-sentence objective. The recipe "pretrain once, fine-tune a small head per task" set new results on most NLP benchmarks and made transfer learning the default in NLP. Encoder models of this family are still widely used for classification, NER and retrieval embeddings.
• GPT-style decoders are trained left-to-right to predict the next token. With scale, instruction tuning and preference training, they became the general-purpose LLMs covered in the LLM course.

Tasks and their metrics

F1 is the harmonic mean of precision $P$ and recall $R$: $F_1 = 2PR/(P + R)$. For NER it is usually computed over whole entities: a prediction counts only if span boundaries and type both match. Worked example: a document has 4 gold entities; the model predicts 5, of which 3 match exactly. $P = 3/5 = 0.60$, $R = 3/4 = 0.75$, $F_1 = 2(0.60)(0.75)/1.35 = 0.667$. Macro-F1 averages F1 across classes (each class equal), micro-F1 pools all decisions (frequent classes dominate); report macro-F1 when rare classes matter.

BLEU (Papineni et al., 2002) compares a candidate translation to one or more references using clipped n-gram precision for $n = 1..4$, combined by a geometric mean and multiplied by a brevity penalty that punishes candidates shorter than the reference. Clipping stops gaming: the candidate "the the the the" against the reference "the cat is on the mat" has unigram precision 4/4 unclipped, but "the" appears only twice in the reference, so the clipped precision is 2/4.

ROUGE (Lin, 2004) is recall-oriented: ROUGE-N counts reference n-grams that appear in the system summary; ROUGE-L uses the longest common subsequence.

Their limits. Both measure surface overlap. A correct paraphrase ("the meeting was postponed" vs "the meeting was delayed") is penalized; a fluent summary that inverts a fact ("did not approve" vs "approved") can score well. Scores depend on tokenization and on how many references exist, so numbers from different papers are often not comparable (the SacreBLEU tool exists to standardize BLEU). Learned metrics (BERTScore, COMET) correlate better with human judgment, and LLM-as-judge evaluation is common for open-ended generation, but all of these need spot-checking by people and are covered in depth in the evaluation lessons.

Hugging Face pipelines

The transformers pipeline API wraps tokenization, model and post-processing. Pin explicit model names so results do not change when defaults change. Task coverage also changes between major versions: transformers v5 removed the dedicated summarization and translation pipelines, so the last part of the example uses the version-independent tokenize, generate(), decode pattern, which works for any encoder-decoder model.

from transformers import pipeline, AutoTokenizer, AutoModelForSeq2SeqLM

sentiment = pipeline("text-classification",
                     model="distilbert/distilbert-base-uncased-finetuned-sst-2-english")
print(sentiment("The update fixed my crash, finally!"))

ner = pipeline("token-classification", model="dslim/bert-base-NER",
               aggregation_strategy="simple")          # merge B-/I- word pieces into spans
print(ner("Ada Lovelace worked with Charles Babbage in London."))

qa = pipeline("question-answering", model="distilbert/distilbert-base-cased-distilled-squad")
print(qa(question="Who worked with Babbage?",
         context="Ada Lovelace worked with Charles Babbage in London."))

# Summarization: call generate() directly (the summarization pipeline was removed in v5)
name = "sshleifer/distilbart-cnn-12-6"
tok, s2s = AutoTokenizer.from_pretrained(name), AutoModelForSeq2SeqLM.from_pretrained(name)
long_text = "Replace this string with the report or article you want to condense. " * 20
inputs = tok(long_text, return_tensors="pt", truncation=True, max_length=1024)
ids = s2s.generate(**inputs, max_new_tokens=60, min_length=20, num_beams=4)
print(tok.decode(ids[0], skip_special_tokens=True))

How LLMs absorbed NLP, and where classic methods still win

A modern instruction-tuned LLM can do classification, extraction, QA, summarization and translation from a prompt with zero or a few examples, handles messy inputs gracefully, and lets a single engineer prototype in an afternoon what used to take a labeled dataset and a team. For low-volume, varied or generative tasks it is usually the right starting point.

Classic and small-model NLP still wins when:

• Volume and cost dominate. Classifying 50 million short messages a day with a TF-IDF model or a fine-tuned small encoder costs a tiny fraction of LLM calls and runs in milliseconds on a CPU.
• Determinism and auditability matter. Regular expressions and rules for IBANs, dates or part numbers are exact, explainable and testable; an LLM may "correct" a digit.
• Latency and offline use. On-device or real-time pipelines often cannot wait for a large model.
• Search. Lexical retrieval (BM25) remains a strong component of production search and RAG, typically combined with embeddings in hybrid search.
• You have labels. With a few thousand labeled examples, a fine-tuned BERT-class model frequently matches or beats prompting a much larger model on a narrow classification or NER task, and is cheaper to run.

The pragmatic pattern is to prototype with an LLM, use it to help label data, then distill the high-volume path into a small model while keeping the LLM for the long tail.

Common pitfalls

• Tokenization mismatch. Using a different tokenizer from the one a model was trained with produces garbage silently; always load tokenizer and model from the same checkpoint.
• Leakage through near-duplicates. Templated emails or reposted articles in both train and test inflate scores. Deduplicate before splitting.
• Reporting accuracy on imbalanced data. 95% accuracy when 95% of tickets are "other" means nothing; report per-class and macro-F1.
• Comparing BLEU/ROUGE across setups. Different tokenization, casing or reference counts make numbers incomparable.
• Lemmatizing or lowercasing blindly. Case carries information for NER ("Apple" vs "apple"), and stop-word removal can delete negations ("not") that flip sentiment.
• Assuming English. Many tools and benchmarks are English-first; morphology-rich and low-resource languages need different tokenization and evaluation.

In practice

Start every text task by writing down the input, the output schema and the metric, then build a cheap baseline (TF-IDF + logistic regression, or zero-shot LLM) and a small labeled test set of a few hundred examples. Compare a prompted LLM against a fine-tuned small encoder on that test set with cost and latency included, not accuracy alone. Use spaCy-style pipelines for preprocessing and rule-based extraction, Hugging Face for pretrained encoders and fine-tuning, and keep humans in the loop for metrics on generative output. In production, monitor drift in vocabulary and topics (new product names, new slang), and re-label a fresh sample regularly.