Why Run AI Locally?

Most people meet AI through a website or app where the model runs in someone else's data center. Running a model on your own computer is now practical for many everyday tasks, and knowing when it makes sense can save money, protect sensitive data, and keep you working when the internet does not.

What "running locally" means

A local model is an AI model whose files (its weights, the billions of learned numbers that make up the model) sit on your own disk and run on your own processor or graphics card. A hosted model runs on a provider's servers; you send your text over the internet and get an answer back.

The software that loads the weights and produces text is called a runtime or inference engine. Inference simply means using a trained model to produce output, as opposed to training it. Later lessons in this track cover runtimes and hardware in detail.

An analogy: hosted AI is like taking a taxi, and local AI is like owning a car. The taxi is always the newest model and needs no maintenance, but you pay every trip and the driver knows where you went. The car costs money up front and needs some care, but after that each trip is cheap, private, and available whenever you want it.

The reasons to run locally

• Privacy. Your prompts and documents never leave the machine. For medical notes, legal files, source code under contract, or personal journals, this can be the deciding factor. It also simplifies compliance, because there is no third party processing the data.
• Cost at volume. Hosted models charge per token (a chunk of text, roughly three-quarters of an English word). Light use is cheap. Heavy, repetitive use, such as summarizing thousands of documents or running an agent that loops all day, adds up. A local machine costs money once, then mostly electricity.
• Offline and reliable. A local model works on a plane, in a basement lab, or during an outage. It is never rate-limited, deprecated, or changed without notice.
• Control and customization. You choose the exact model version, its settings, and its system prompt. You can run fine-tuned or specialized variants, swap models freely, and keep a known-good setup unchanged for years.
• Latency. There is no network round trip and no queue behind other customers. On capable hardware, a small model can start answering almost instantly, which matters for autocomplete, voice assistants, and tools that call the model many times in a row.

A worked cost example

The numbers below are illustrative, not current prices. Check real prices before deciding.

Suppose a small team processes 5 million tokens a day through a hosted model priced at a hypothetical $2 per million tokens. That is $10 a day, or about $3,650 a year.

Now suppose a local machine draws 300 watts while working, for 8 hours a day. That is 2.4 kilowatt-hours a day. At an illustrative $0.15 per kWh, electricity costs about $0.36 a day, or roughly $130 a year. The question becomes whether the machine's purchase price, minus what you would have spent on the hosted service, pays back within its useful life, and whether a model small enough for that machine does the job well enough.

At 50,000 tokens a day, the hosted bill in this example drops to about $36 a year, and buying hardware purely to save money makes little sense. Volume is what changes the math.

The honest downsides

• Hardware cost. Running capable models needs a lot of fast memory, and memory is the expensive part. A machine that comfortably runs mid-sized models costs far more than a typical laptop.
• Smaller, weaker models. The strongest hosted models are much larger than anything a single desktop can hold, and some are not available to download at all. Open-weight models you can run at home have improved quickly, but on the hardest reasoning, coding, and research tasks the biggest hosted models generally still lead.
• Setup and upkeep. You pick models, choose a compressed version, match it to your memory, and update software. Modern tools make this much easier than it was, but it is still your job.
• Power, heat, and noise. A large graphics card under load can draw several hundred watts and sound like a hair dryer. Compact unified-memory machines are quieter and more efficient, but slower at some tasks.
• Speed limits. Generation speed depends on memory bandwidth. A model that streams text quickly on a hosted service may crawl on modest hardware, especially with long documents in the prompt.

The hybrid approach

Most people who run local AI do not run everything locally. A common pattern:

This works smoothly because many local runtimes expose an OpenAI-compatible API, meaning they accept requests in the same format that many hosted services use. As of November 2026, for example, Ollama documents a local endpoint at http://localhost:11434/v1 for this purpose. Software written for a hosted service can often be pointed at a local model by changing one address, and routing logic can send each request to whichever side fits it best.

Going deeper

Total cost of ownership

The worked example above compared a daily hosted bill with a daily electricity bill. A fair comparison uses total cost of ownership (TCO): everything the local machine costs over its useful life, spread per year. A simple model:

where $H$ is the hardware price, $L$ its useful life in years, $P$ the average power draw in watts while working, $h$ the working hours per day, and $e$ the electricity price per kWh. This deliberately ignores your time, resale value, and cooling, so treat it as a lower bound.

The hosted cost per year for $T$ tokens per day at a blended price $p$ per million tokens is

Setting the two equal gives the break-even volume, the daily token count above which local is cheaper:

Worked example

Take an illustrative machine at USD 4,000, a 3-year life, 300 W for 8 hours a day, and USD 0.15 per kWh, against a hosted price of USD 2 per million tokens (the same hypothetical numbers as above).

1. Depreciation: 4,000 ÷ 3 ≈ 1,333 per year.
2. Energy: 0.3 kW × 8 h × 365 = 876 kWh, × 0.15 ≈ 131 per year.
3. $C_{\text{local}} \approx$ 1,465 per year.
4. $T^{*} = 1{,}465 \times 10^6 / (365 \times 2) \approx 2.0$ million tokens per day.

So at 5 million tokens a day the machine is comfortably cheaper; at 50,000 a day it never pays back.

The capacity check people forget

Break-even assumes the machine can actually do the work. Token volume has two parts with very different speeds: input tokens are processed in the fast, parallel prefill phase, and output tokens are generated one at a time in the slower decode phase (the tokens-per-second lesson explains why). Suppose the 5 million daily tokens are 90% input and 10% output, and the machine prefills at about 1,000 tokens per second and decodes at about 30 for one stream:

That fits in an 8-hour day, but notice that the 10% of tokens that are output take almost four times as long as the 90% that are input. A workload that is output-heavy (long generated reports, agents that think aloud) can exceed a single machine's daily capacity even when the cost math looks good. Server runtimes recover capacity by batching several requests, which the later lessons cover.

Encoding the decision

def local_annual_cost(hw_price, life_years, watts, hours_per_day, usd_per_kwh):
    energy_kwh = watts / 1000 * hours_per_day * 365
    return hw_price / life_years + energy_kwh * usd_per_kwh

def breakeven_tokens_per_day(annual_local, usd_per_million):
    return annual_local * 1e6 / (365 * usd_per_million)

def hours_needed(tokens_in, tokens_out, prefill_tps, decode_tps):
    return (tokens_in / prefill_tps + tokens_out / decode_tps) / 3600

annual = local_annual_cost(4000, 3, 300, 8, 0.15)
print(round(annual))                                   # 1465
print(round(breakeven_tokens_per_day(annual, 2.0)))    # 2006484
print(round(hours_needed(4.5e6, 0.5e6, 1000, 30), 1))  # 5.9

The three functions mirror the three questions: what local costs per year, how much volume makes it worthwhile, and whether the box can keep up. Hosted providers usually price input and output tokens differently; to model that, replace the single price $p$ with separate input and output prices weighted by your mix.

Sensitivity analysis

Every input in the break-even formula is uncertain, so professionals check how the answer moves. $T^{*}$ is inversely proportional to $p$: if hosted prices halve, which has happened repeatedly for models of a given capability, the break-even volume doubles. It is proportional to $C_{\text{local}}$, which is dominated by depreciation: a machine kept for 5 years instead of 3 cuts the yearly hardware share by 40%. Energy matters less unless electricity is expensive or the machine runs around the clock. A sensible habit is to compute $T^{*}$ under pessimistic, central, and optimistic assumptions and decide only if all three point the same way.

Privacy as a threat model

"Local is private" is true in a specific sense: it removes the provider as a party that receives, processes, and possibly retains your data. It does not remove every risk. Security engineers reason about this with a threat model: list who could see the data and how.

Local AI shifts responsibility rather than eliminating it: disk encryption, access control, where chat history is stored, and which model files you trust all become your job.

Latency, decomposed

Time to first token on a hosted service is roughly network round trip plus queueing plus the provider's prefill; locally it is your own prefill. A hosted provider often prefills faster than a desktop because it runs on larger accelerators, so local wins on latency mainly for short prompts and many small calls, where round trips and queues dominate, not for very long prompts.

Common pitfalls

• Comparing hardware price with one month of API spend. Use annual TCO against annual hosted cost, and include electricity.
• Skipping the capacity check. A machine that is cheaper per token on paper may not be able to generate enough output tokens per day.
• Assuming a smaller local model is good enough. Test the quality on your real tasks before buying; a cheaper model that fails the job costs more.
• Treating "local" as automatically secure. Exposed API ports, untrusted downloads, and unencrypted chat logs are common local failures.
• Forgetting your own time. Setup, updates, and troubleshooting are real costs that the formula leaves out.

In practice

• Write down what you actually use AI for and sort each task into "sensitive", "high-volume", "latency-sensitive", or "needs the very best model". The first three lean local; the last leans hosted.
• Estimate your monthly token volume before buying hardware. If it is small, privacy and offline use are better reasons to go local than cost.
• Try a small model on the computer you already own before spending anything. The next lessons explain how much memory you need and how fast it will run.
• Use the AI Hardware & Model Index to see which models fit on which machines.