Local Runtimes and Model Formats

Once you know a model fits your hardware, you need software to run it and a copy of the model in a format that software understands. The ecosystem has a handful of well-established tools, and choosing among them is mostly about who will use it and on what machine.

What a runtime does

A runtime (or inference engine) loads model weights into memory, turns your text into tokens, runs the model, and turns the output tokens back into text. Good runtimes also manage the KV cache (stored context for the conversation), split work between CPU and GPU, and offer an API so other programs can use the model.

An analogy: model files are like video files, and runtimes are like video players. Some players are friendly apps, some are command-line engines, and some are built to stream to thousands of viewers at once. Not every player opens every file format.

The main runtimes

Platform details below are as of November 2026.

A rough rule: Ollama or LM Studio for one person on a desktop, MLX for the best results on a Mac, llama.cpp directly when you want maximum control or unusual hardware, and vLLM or SGLang when a GPU server is answering many requests at once.

Model formats

Models are distributed as files, and the format determines which runtime can use them.

• safetensors is the standard format for publishing original model weights. It stores the raw numbers (tensors) plus a small header describing their names, types, and shapes, and no executable code, which makes it safer to load than older formats based on Python's pickle. Server engines like vLLM and SGLang typically load safetensors directly, often at 16-bit or with GPU-oriented quantization.
• GGUF is the file format of the GGML library behind llama.cpp. It bundles tensors with metadata (architecture, tokenizer, settings) in one binary file and supports many quantization levels, meaning weights stored with fewer bits to save memory. Ollama, LM Studio, and llama.cpp use GGUF.
• MLX format is weights converted for Apple's MLX framework, usually published as safetensors files plus configuration, at quantization levels such as 4-bit or 8-bit.

Reading a GGUF file name

GGUF names usually include a quantization tag. Hugging Face's documentation lists the bits per weight for common types; for example:

Real files mix types across layers, so the file size on disk is the best guide; the site's AI Index uses about 4.8 bits for a typical 4-bit download. Lower bits mean a smaller, faster model with some loss of quality, which is usually small at 4–6 bits and grows quickly below about 3 bits.

Where to download models

Hugging Face is the main public hub for open-weight models. Model creators publish original safetensors weights there, and community members publish converted GGUF and MLX versions. You can filter for GGUF files on the Hub. Ollama also runs its own model library, and LM Studio has a built-in search that pulls from Hugging Face.

Before downloading, check three things: the license (some models restrict commercial use), who published the conversion (prefer the original creator or well-known converters), and the file size compared with your usable memory.

Local OpenAI-compatible APIs

Most of these runtimes can run as a local server that accepts requests in the same shape as OpenAI's API, such as the /v1/chat/completions endpoint. As of November 2026, Ollama's documentation uses http://localhost:11434/v1, and LM Studio's examples use http://localhost:1234/v1. vLLM and SGLang offer OpenAI-compatible endpoints as well.

That means many chat apps, coding assistants, and scripts written for a hosted API can be pointed at your local model by changing the base address and model name. One caution: these local servers generally assume a trusted machine or network. Do not expose them to the internet without proper authentication in front. Ollama, for example, has no built-in login, and vLLM's optional API key protects only some of its endpoints, so its own security guide recommends a reverse proxy and firewall.

Going deeper

How block quantization stores a weight

Most local formats use block quantization: weights are split into small blocks, and each block stores low-bit integers plus one or two higher-precision numbers that rescale them. In the simplest symmetric form, a block of $k$ weights $w_1,\dots,w_k$ gets a scale

where $q_{\max} = 2^{b-1}-1$ for $b$-bit signed integers (7 for 4-bit). Rounding error per weight is at most $d/2$, so blocks with one large outlier get a coarse scale and lose precision for everything else. Asymmetric variants add a per-block offset $m$, so $\hat w_i = d\,q_i + m$. llama.cpp's block formats differ in their exact layout, but the idea is the same.

Deriving bits per weight

The overhead of the scales is what turns "4-bit" into 4.5 bits per weight (bpw):

• Q4_0 / Q8_0 style: blocks of 32 weights with one 16-bit scale. Q4: $(32\times4 + 16)/32 = 4.5$ bpw. Q8: $(32\times8+16)/32 = 8.5$ bpw.
• Q4_K: a 256-weight super-block of eight 32-weight sub-blocks, each with a 6-bit scale and a 6-bit minimum, plus a 16-bit super-scale and super-minimum: $(256\times4 + 8\times12 + 32)/256 = 1152/256 = 4.5$ bpw.
• Q6_K: 256 weights at 6 bits, sixteen 8-bit sub-block scales, one 16-bit super-scale: $(1536 + 128 + 16)/256 = 6.5625$ bpw.
• Q3_K: 256 weights at 3 bits, sixteen 6-bit scales, one 16-bit super-scale: $(768 + 96 + 16)/256 = 3.4375$ bpw.

These match the figures in Hugging Face's GGUF table above. "Mixed" recipes such as Q4_K_M keep some sensitive tensors at a higher type (for example Q6_K), which is why a typical 4-bit download averages nearer 4.8 bpw, the Index's default. MLX's common 4-bit setting uses groups of 64 weights with a 16-bit scale and a 16-bit bias, also about $4 + 32/64 = 4.5$ bpw.

Experiment: block size versus error

import numpy as np

def quantize_blocks(w, bits=4, block=32):
    """Symmetric absmax quantization: one 16-bit scale per block of weights."""
    qmax = 2 ** (bits - 1) - 1                     # 7 for 4-bit
    w = w.reshape(-1, block)
    scale = np.abs(w).max(axis=1, keepdims=True) / qmax
    scale[scale == 0] = 1.0
    q = np.clip(np.round(w / scale), -qmax - 1, qmax).astype(np.int8)
    return q, scale.astype(np.float16)

def dequantize(q, scale):
    return (q * scale.astype(np.float32)).ravel()

rng = np.random.default_rng(0)
w = rng.normal(0, 0.02, size=256 * 4096).astype(np.float32)
for block in (32, 256):
    q, s = quantize_blocks(w, 4, block)
    rmse = np.sqrt(np.mean((w - dequantize(q, s)) ** 2))
    bpw = 4 + 16 / block
    print(f"block={block:3d}  bits/weight={bpw:.4f}  RMSE={rmse:.6f}")

On this synthetic Gaussian data, 32-weight blocks cost 4.5 bpw and give a root-mean-square error of about 0.0019; 256-weight blocks cost only 4.0625 bpw but the error rises to about 0.0025, because each scale must cover a wider range. K-quants get the best of both by giving 32-weight sub-blocks cheap 6-bit scales. Real weights have heavier-tailed outliers than a Gaussian, which is why calibration-based methods such as GPTQ, which adjust remaining weights to compensate for rounding error, do better than plain rounding at low bit widths.

Choosing a bit width

For a fixed memory budget you can trade parameters against bits: a 14B model at 8 bits and a 28B model at 4 bits occupy similar space. Dettmers and Zettlemoyer (2023) studied this trade across many model families and found that, for a fixed total number of model bits, 4-bit precision was almost always the best choice for zero-shot accuracy. In practice that supports the common habit of running the largest model that fits at roughly 4–5 bits per weight rather than a smaller model at 8 or 16 bits, while dropping below about 3 bits usually costs more quality than the extra parameters buy back. Check quality on your own tasks, since results vary by model and method.

What a safetensors file actually contains

A safetensors file is: 8 bytes holding a little-endian unsigned 64-bit integer $N$; then $N$ bytes of JSON mapping each tensor name to its dtype, shape, and data_offsets; then the raw tensor bytes. Nothing in it is executable, which is the security point. You can inspect one with only the standard library (see the exercises), and you can compute parameter counts from the shapes without loading any weights.

Talking to a local server in code

Because the servers mimic the OpenAI API, the official openai Python client works by changing base_url:

from openai import OpenAI

# Ollama's documented local endpoint; the client requires a key, which Ollama ignores.
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
    model="llama3.2",  # replace with a model you have pulled locally
    messages=[{"role": "user", "content": "Explain GGUF in one sentence."}],
)
print(reply.choices[0].message.content)

Swap the base URL for LM Studio (http://localhost:1234/v1) or a vLLM server, and the rest of the code is unchanged. Runtime-specific options (context length, GPU layers, sampling extras) are not part of the shared API and are set in each runtime's own configuration.

For interactive apps, pass stream=True and iterate over the returned chunks so text appears as it is generated; time to first token then becomes visible to the user, which is exactly the prefill cost discussed in the tokens-per-second lesson. For batch jobs, sending several requests concurrently lets server runtimes such as vLLM batch them, raising total throughput even though each request is no faster.

Common pitfalls

• Loading pickle-based checkpoints from strangers. .bin, .pt, and .pth files can execute code when loaded; prefer safetensors or GGUF from trusted publishers.
• Assuming the model's maximum context is in use. Runtimes often default to a context window much smaller than the model supports, and silently drop the start of long prompts; check and set it.
• Comparing quant names across ecosystems. "4-bit" GGUF, MLX, GPTQ, and AWQ files differ in bpw and quality; compare file sizes and measured quality instead.
• Wrong chat template. A model run with the wrong prompt format gives poor answers; GGUF files usually embed the template, but converted or merged models sometimes do not.
• Binding the server to all interfaces. Exposing a local API on 0.0.0.0 without authentication puts it on your network or the internet.

In practice

• Start with Ollama or LM Studio; both can download a model and have you chatting in minutes.
• On a Mac, try an MLX version of the same model and compare speed.
• Pick a GGUF quantization whose file size, plus KV cache and about 1.5 GB, stays under 85% of your usable memory; the AI Hardware & Model Index shows sizes for common models.
• Point your existing tools at the local OpenAI-compatible address to run them privately.