Capstone: LoRA Fine-Tune a Small Open Model

Project brief

A support team wants a small model that runs on their own laptops, follows instructions in a consistent house style, and never sends customer data to an outside service. Prompting a general model gets close, but answers drift in format and length. You are asked to fine-tune a small open-weight model with LoRA, prove with numbers that it improved without breaking general ability, and deliver a single quantized GGUF file they can run in llama.cpp or Ollama.

This is parameter-efficient fine-tuning as practitioners do it: pick a base model whose licence allows your use, budget memory before you start, train small adapter matrices instead of all weights, evaluate before and after on held-out data, merge, and quantize for local inference.

Learning goals

• Choose a base model by size, licence, chat template and tooling support, and justify the choice.
• Budget GPU memory for full fine-tuning, LoRA and QLoRA from first principles.
• Format an instruction dataset with the model's chat template and mask the prompt tokens out of the loss.
• Train LoRA adapters with Hugging Face transformers and peft, and evaluate before and after with held-out loss, task checks and a regression check.
• Merge adapters into the base weights and convert the result to a quantized GGUF file for local use.

Required background

Review post-training-and-alignment, fine-tuning-vs-prompting, parameters-and-model-size, quantization, local-runtimes-and-formats and open-weights-and-licensing.

Setup

Base model. The default is Qwen/Qwen2.5-1.5B-Instruct (about 1.54 billion parameters, 28 layers, hidden size 1536, 12 query heads and 2 key-value heads of dimension 128), released by the Qwen team under the Apache 2.0 licence according to its model card. It is small enough for free hosted GPUs, uses a standard chat template, and is supported by llama.cpp's converter. A larger alternative is IBM's Granite 4.2 3B, listed in this site's model index as Apache-2.0 as of November 2026. Whatever you choose, read the model card and licence yourself, record the exact repository and revision, and check whether the instruct and base variants differ in licence.

Dataset. databricks/databricks-dolly-15k on the Hugging Face Hub: about 15,000 human-written instruction and response pairs with fields instruction, context, response and category, released under CC BY-SA 3.0. The share-alike terms mean that if you publish a derived dataset you must license it compatibly, and you must credit the source; consider what that means for your model card. For the house-style scenario, you may replace or augment it with your own examples (a few hundred good ones often move format more than thousands of mediocre ones), as long as you have rights to them.

Environment. Python 3.10+ with torch, transformers, peft, datasets and accelerate; add bitsandbytes for QLoRA on NVIDIA GPUs. For GGUF conversion you need a clone of the llama.cpp repository (for its convert_hf_to_gguf.py script and its Python requirements) and a llama.cpp build that includes the llama-quantize tool. Library APIs in this area change often: when an argument name in the code below is rejected, check the current documentation rather than downgrading blindly. The trl library's SFTTrainer wraps much of this; the capstone uses the plain Trainer so that every step, especially loss masking, is visible.

Hardware. A single GPU with 16 GB or more trains LoRA in 16-bit at sequence length 1024 and batch size 4 comfortably. With less memory, lower the batch size (and raise gradient accumulation) or use QLoRA. Free hosted notebook GPUs are time-limited and often entry-level cards that may lack bfloat16 support; the code below falls back to float16 for them. Apple-silicon Macs can run LoRA through PyTorch's mps device slowly; bitsandbytes QLoRA targets NVIDIA GPUs. Save adapters to persistent storage often: hosted sessions end without warning.

Milestone 1: Memory budget and baseline

Full fine-tuning with AdamW in mixed precision needs roughly 16 bytes per parameter before activations: 2 for 16-bit weights, 2 for gradients, 4 for an FP32 master copy, and 8 for AdamW's two FP32 moment estimates (some recipes differ, but this is the standard estimate). For $P = 1.54 \times 10^9$ that is about $24.6$ GB, too much for a 16 GB card.

LoRA freezes the pretrained weight $W_0 \in \mathbb{R}^{d_\text{out} \times d_\text{in}}$ and learns a low-rank update:

$B$ starts at zero, so training begins exactly at the base model. Each adapted matrix adds $r(d_\text{in} + d_\text{out})$ trainable parameters.

Worked example. Adapt all seven linear projections in each of the 28 layers with $r = 16$. With hidden size 1536, key/value projection width $2 \times 128 = 256$ and MLP width 8960:

Per layer: $2(49{,}152) + 2(28{,}672) + 3(167{,}936) = 659{,}456$. Times 28 layers: $18{,}464{,}768$ trainable parameters, about 1.2% of the model. Their weights, gradients and AdamW states in FP32 take about $18.5\text{M} \times 16 = 295$ MB. The frozen base in bfloat16 takes $1.54\text{B} \times 2 = 3.08$ GB. Activations come on top and grow with batch size times sequence length; gradient checkpointing trades extra compute for much less activation memory.

QLoRA stores the frozen base in 4-bit NormalFloat (NF4); Dettmers et al. report about 4.127 bits per parameter including quantization constants when double quantization is used. Embedding layers are usually left in 16-bit, and for this model the token embedding is $151{,}936 \times 1536 \approx 233$M parameters ($\approx 0.47$ GB in bfloat16). The rest, about 1.31B parameters, costs $1.31\text{B} \times 4.127 / 8 \approx 0.67$ GB, so the base drops to roughly 1.1 GB. These are estimates; measure with torch.cuda.max_memory_allocated() and report the measured peak.

Before training, record the baseline: generate answers for 50 held-out prompts, compute held-out loss (milestone 4) and save everything to disk.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"
DTYPE = torch.bfloat16 if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else torch.float16
tok = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=DTYPE, device_map="auto")

@torch.no_grad()
def generate(user_text, max_new_tokens=256):
    prompt = tok.apply_chat_template([{"role": "user", "content": user_text}],
                                     tokenize=False, add_generation_prompt=True)
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    out = model.generate(**ids, max_new_tokens=max_new_tokens, do_sample=False)
    return tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True)

print(generate("Give three tips for writing a clear bug report."))

Older transformers releases call the dtype argument torch_dtype; use whichever your installed version documents.

Milestone 2: Data formatting and loss masking

The model must learn to produce responses, not to reproduce prompts. Tokenize the templated prompt and the response separately, concatenate them, and set the label of every prompt token to $-100$, the index that PyTorch's cross-entropy ignores. The training loss is then the mean negative log-likelihood of the response tokens only:

where $R$ is the set of response positions and $p_\theta$ is the model's next-token distribution.

from datasets import load_dataset

MAX_LEN = 1024
ds = load_dataset("databricks/databricks-dolly-15k", split="train")
ds = ds.train_test_split(test_size=300, seed=0)

def build(row):
    user = row["instruction"] + (f"\n\nContext:\n{row['context']}" if row["context"] else "")
    prompt = tok.apply_chat_template([{"role": "user", "content": user}],
                                     tokenize=False, add_generation_prompt=True)
    p_ids = tok(prompt, add_special_tokens=False)["input_ids"]
    r_ids = tok(row["response"] + tok.eos_token, add_special_tokens=False)["input_ids"]
    ids = (p_ids + r_ids)[:MAX_LEN]
    labels = ([-100] * len(p_ids) + r_ids)[:MAX_LEN]
    return {"input_ids": ids, "labels": labels}

train = ds["train"].map(build, remove_columns=ds["train"].column_names)
test = ds["test"].map(build, remove_columns=ds["test"].column_names)
train = train.filter(lambda r: any(l != -100 for l in r["labels"]))  # drop fully truncated

def collate(batch):
    n = max(len(b["input_ids"]) for b in batch)
    pad = tok.pad_token_id if tok.pad_token_id is not None else tok.eos_token_id
    ids = torch.tensor([b["input_ids"] + [pad] * (n - len(b["input_ids"])) for b in batch])
    labels = torch.tensor([b["labels"] + [-100] * (n - len(b["labels"])) for b in batch])
    lengths = torch.tensor([len(b["input_ids"]) for b in batch])
    mask = (torch.arange(n)[None, :] < lengths[:, None]).long()
    return {"input_ids": ids, "labels": labels, "attention_mask": mask}

Tokenizing the two parts separately avoids a subtle bug: tokenizing the concatenated string can merge characters across the boundary, so the prompt length you compute would not match the real one. Appending the end-of-sequence token teaches the model to stop. Decode one example and print which tokens have labels, as a check.

Milestone 3: LoRA training

from peft import LoraConfig, get_peft_model
from transformers import Trainer, TrainingArguments

model.enable_input_require_grads()   # needed with gradient checkpointing on a frozen base
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                  "gate_proj", "up_proj", "down_proj"])
model = get_peft_model(model, lora)
model.print_trainable_parameters()     # expect about 18.5M trainable

args = TrainingArguments(
    output_dir="lora-out", per_device_train_batch_size=4, gradient_accumulation_steps=4,
    learning_rate=2e-4, num_train_epochs=1, lr_scheduler_type="cosine", warmup_steps=30,
    logging_steps=20, save_steps=200, save_total_limit=2, gradient_checkpointing=True,
    bf16=(DTYPE == torch.bfloat16), fp16=(DTYPE == torch.float16), report_to="none")
trainer = Trainer(model=model, args=args, train_dataset=train, data_collator=collate)
trainer.train()
model.save_pretrained("adapter")       # writes only the small adapter weights
print("peak GB:", torch.cuda.max_memory_allocated() / 1e9 if torch.cuda.is_available() else "n/a")

The effective batch size is $4 \times 4 = 16$ sequences per optimizer step. A learning rate around $10^{-4}$ to $2 \times 10^{-4}$ is a common LoRA starting point because only the small adapter matrices move; full fine-tuning typically uses much smaller rates. For QLoRA, load the base with BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=DTYPE) passed as quantization_config, call peft.prepare_model_for_kbit_training(model) before get_peft_model, and keep everything else the same.

Milestone 4: Evaluate before and after

Use three kinds of evidence, all on data the model never trained on.

1. Held-out loss and perplexity on the 300 test rows, response tokens only. Perplexity is $\exp(\mathcal{L})$; a drop from, say, $\mathcal{L} = 1.90$ (perplexity $e^{1.90} \approx 6.69$) to $\mathcal{L} = 1.55$ (perplexity $\approx 4.71$) means the model assigns the reference responses much higher probability. Lower loss on this dataset does not by itself mean better answers.
2. Task checks for the behaviour you wanted: for a house style, automatic checks (length limit, required sections, no forbidden phrases) plus a blind pairwise comparison of 50 base and fine-tuned answers, with the order randomized, judged by people or by a strong model with a rubric that you spot-check.
3. Regression check: 50 to 100 questions from a general benchmark or your own general-knowledge set, scored for base and fine-tuned models, to detect forgetting.

import math

@torch.no_grad()
def heldout_loss(model, rows, bs=8):
    model.eval()
    total, count = 0.0, 0
    for i in range(0, len(rows), bs):
        batch = {k: v.to(model.device) for k, v in collate(rows[i:i + bs]).items()}
        loss = model(**batch).loss
        n = (batch["labels"][:, 1:] != -100).sum().item()   # tokens actually scored
        total += loss.item() * n; count += n
    return total / count, math.exp(total / count)

rows = [test[i] for i in range(len(test))]
print("loss, perplexity:", heldout_loss(model, rows))

The model shifts labels by one position internally, so the number of scored tokens is the count of non-ignored labels after the first position; weighting each batch's mean loss by that count gives the exact token-level average across batches of different sizes.

Milestone 5: Merge, convert to GGUF, quantize

Merging folds $\frac{\alpha}{r}BA$ into $W_0$ so the result is an ordinary model with no extra inference cost. Merge into a 16-bit copy of the base, never into a 4-bit one, because adding the update to quantized weights loses precision.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(MODEL_ID, dtype=torch.float16)  # CPU is fine
merged = PeftModel.from_pretrained(base, "adapter").merge_and_unload()
merged.save_pretrained("merged")
AutoTokenizer.from_pretrained(MODEL_ID).save_pretrained("merged")

Then, from your llama.cpp checkout and build:

python convert_hf_to_gguf.py ../merged --outfile house-1.5b-f16.gguf --outtype f16
./llama-quantize house-1.5b-f16.gguf house-1.5b-Q4_K_M.gguf Q4_K_M

(Depending on how you built llama.cpp, llama-quantize may live in a build/bin folder.) To use it in Ollama, write a Modelfile containing FROM ./house-1.5b-Q4_K_M.gguf and run ollama create house-1.5b -f Modelfile; check that the chat template Ollama applies matches the one you trained with.

Sizing the deliverable with the site's AI Index method: Q4_K_M averages about 4.8 bits per weight, so the file is about $1.54\text{B} \times 4.8 / 8 \approx 0.92$ GB, or $0.97$ GB with the 5% allowance. The KV cache is $2 \times 28 \times 2 \times 128 \times 2 = 28{,}672$ bytes per token in 16-bit, so 4,096 tokens of context need about 0.12 GB. Adding 1.5 GB of runtime overhead gives roughly 2.6 GB of memory. On a laptop with 100 GB/s of memory bandwidth, decode speed is about $1 / (0.92\text{ GB} / (0.55 \times 100\text{ GB/s}) + 1.5\text{ ms}) \approx 55$ tokens per second. Measure the real numbers and compare them with this estimate in your report. Finally, re-run 20 of your task checks on the quantized model; quantization can shift behaviour, and you must ship what you measured.

Evaluation criteria and target metrics

These are course thresholds, not published benchmark results.

Grading rubric

Stretch goals

• Compare ranks $r \in \{4, 16, 64\}$ and plot held-out loss against trainable parameters.
• Train only attention projections versus all linear layers and compare quality and memory.
• Repeat with QLoRA and report the memory saved and any quality difference.
• Replace Dolly with 300 to 1,000 high-quality examples in your house style and compare.
• Write a model card covering intended use, data, licence obligations, evaluation and limitations.

Write-up template

1. Goal. The behaviour you wanted to change and why prompting was not enough.
2. Base model and licence. Repository, revision, licence, and why this size.
3. Data. Source, licence, filtering, template, masking check, split sizes.
4. Memory and hardware. Estimates, measured peak, GPU, wall-clock time.
5. Training. LoRA configuration, optimizer, schedule, curves.
6. Evaluation. Before and after on loss, task checks, pairwise preference and regression; examples of wins and losses.
7. Deployment. Merge, GGUF quantization type, file size, measured memory and speed versus estimates.
8. Limitations and model card. Known failure modes, data licence obligations, intended and out-of-scope uses.

Common pitfalls

• Training on prompt tokens, which teaches the model to imitate questions and wastes capacity.
• Training with one chat template and serving with another, which quietly degrades every answer.
• Merging adapters into a 4-bit base, or quantizing before evaluating, and then shipping an unmeasured model.
• Evaluating only on training-like prompts and missing catastrophic forgetting of general skills.
• Ignoring licences: the base model's licence and the dataset's share-alike or non-commercial terms both bind your result.

In practice

Teams fine-tune small models when they need consistent format, tone or domain behaviour at low cost, offline or on-device; they keep using retrieval for facts that change. A typical workflow is exactly this one: a pinned base model, a curated dataset, LoRA or QLoRA with a modest rank, a frozen evaluation suite run before and after, and a quantized artifact that is evaluated again before release. Adapters are often kept separate in serving systems that can load several on one base model, and merged only for single-purpose local deployment.