Routing work between local and hosted AI

The best answer to "local or cloud?" is usually "both". Small and private tasks run well on your own hardware for free; the hardest problems still benefit from the largest hosted models. Routing is the practice of sending each request to the right one automatically.

Why not pick just one?

Neither column wins everywhere, so a single fixed choice either overpays or underdelivers.

The routing questions

A router asks a few questions about each request, roughly in this order:

1. Is there a hard rule? If the content is sensitive or the task is pinned to local, it stays local. Rules come first and are never overridden by cost or convenience.
2. How hard is the task? Classifying an email, extracting fields, summarizing a page or drafting a short reply are well within the reach of a good small or mid-size model. Multi-step reasoning across a large codebase may need a frontier model.
3. How much context does it need? Long contexts cost memory locally (the KV cache, see KV cache and context cost) and money when hosted.
4. How fast must it be? An interactive chat needs a quick first token; an overnight batch job does not.
5. What is the budget? Spending caps per task, per agent or per month.

A simple, effective policy many people use: local first, escalate when needed. Try the local model; if the task is classified as hard, or the local answer fails a check, escalate to a hosted model — unless a rule forbids it.

A worked example

Suppose an agent processes 2,000 support emails a day. Each email plus instructions is about 1,500 input tokens and the reply about 300 output tokens.

• All hosted, at an illustrative $3 per million input and $15 per million output tokens: 2,000 × (1,500 × $3 + 300 × $15) ÷ 1,000,000 = $18 a day, about $540 a month.
• Routed: 90% are routine and handled by a local model; 10% are escalated. Hosted cost drops to about $1.80 a day.

The local side has to keep up: 2,000 replies × 300 tokens = 600,000 generated tokens a day. At 40 tokens per second that is about 4 hours of generation, comfortably done in a working day on one machine. (Prices here are round illustrative numbers; check current provider pricing.)

Sizing the local side

Routing only works if your local model is good enough for the routine work and fast enough for the volume. Three numbers decide that:

• Memory decides which models fit.
• Memory bandwidth decides how fast they generate.
• Compute decides how fast long prompts are read.

The AI Hardware & Model Index answers "What can this machine run?" for every machine it lists: the largest model that runs well, the models that fit but are slow, and the ones that need more memory or a cluster. Use it to choose a main local model plus a smaller, faster one for quick routing decisions and simple tasks.

Routing inside Prometheus OS

Prometheus OS routes across the providers you enable, hosted and local, and treats an explicit provider choice as absolute: if a task or agent is set to a particular provider, it does not silently fall back to another. Each request is logged with the model that handled it, so cost and behaviour can be reviewed later. Pooling several of your own machines into one local compute resource is planned and in development.

Going deeper

Routing as an optimisation problem

Stated precisely, a router picks for each request $x$ the model $m$ that minimises expected cost, but only among the models the rules allow:

$m^{*}(x) = \arg\min_{m \in A(x)} \Big[ c_m(x) + \lambda \,\big(1 - q_m(x)\big) \Big]$

Here $A(x)$ is the set of models permitted for this request after hard rules are applied (privacy, pinning, region), $c_m(x)$ is the cost of running request $x$ on model $m$, $q_m(x)$ is the estimated probability that $m$ handles it acceptably, and $\lambda$ (lambda) is what a bad answer costs you, in the same units as $c$. Rules shrink the set before any optimisation, which is why cost can never override them. When a bad answer is cheap (a draft you will edit), $\lambda$ is small and cheap models win; when it is expensive (a customer-facing legal reply), $\lambda$ is large and the router pays for quality. In practice $q_m(x)$ comes from a small classifier, simple rules (length, keywords, task type) or past results; research systems such as FrugalGPT, Hybrid LLM and RouteLLM study how to learn it.

Two architectures: pre-routing and cascades

A pre-router decides before any model runs. A cascade tries the cheaper model first, checks the result, and escalates only on failure. Write $f$ for the fraction escalated, $c_H$ for the hosted cost per request, and $t_L, t_H$ for local and hosted response times. Taking the local marginal cost as roughly zero:

• Cascade: expected cost $= f\,c_H$; expected latency $= t_L + f\,t_H$, because escalated requests pay for both attempts.
• Pre-router: expected cost $= f\,c_H$; expected latency $= (1-f)\,t_L + f\,t_H$, but only if its predictions are right.

Worked example, continuing the support-email case. Per email, hosted cost is $(1{,}500 \times 3 + 300 \times 15) / 10^6 = 0.009$ dollars. With $f = 0.10$ and 2,000 emails a day, hosted spend is $2{,}000 \times 0.10 \times 0.009 = 1.80$ dollars a day, as above. Suppose $t_L = 6$ s and $t_H = 4$ s (illustrative). The cascade averages $6 + 0.1 \times 4 = 6.4$ s per email; a perfect pre-router averages $0.9 \times 6 + 0.1 \times 4 = 5.8$ s. The cascade is slower but has evidence (a failed check) for every escalation; a pre-router is faster but can silently send a hard task to a model that cannot do it. Many systems combine them: pre-route the obvious cases and cascade the uncertain ones.

Sizing the local side with the Index formulas

The site's AI Index methodology uses these approximations:

• Memory needed $\approx$ model file $\times 1.05$ + KV cache + 1.5 GB, where the file is parameters $\times$ bits $\div 8$. It fits if that is at most 85% of usable memory; on macOS the GPU can use about 75% of unified memory by default.
• KV cache bytes per token $= 2 \times \text{layers} \times \text{KV heads} \times \text{head dim} \times \text{bytes per element}$.
• Decode speed $\approx 1 \div \big(\text{bytes per token} \div (0.55 \times \text{bandwidth}) + 1.5\ \text{ms}\big)$, where bytes per token $=$ active parameters $\times$ bits $\div 8$.
• Prefill $\approx 2 \times$ active parameters FLOPs per input token.

Worked example. An 8-billion-parameter model at 4 bits with 32 layers, 8 KV heads and head dimension 128, a 16-bit KV cache and an 8,192-token context, on a hypothetical Mac with 16 GB of unified memory and 400 GB/s of bandwidth.

1. File: $8 \times 10^9 \times 4 / 8 = 4.0$ GB; with overhead $4.2$ GB.
2. KV cache: $2 \times 32 \times 8 \times 128 \times 2 = 131{,}072$ bytes per token; $\times 8{,}192$ tokens $= 1.07$ GB.
3. Need: $4.2 + 1.07 + 1.5 = 6.77$ GB. Limit: $16 \times 0.75 \times 0.85 = 10.2$ GB. It fits.
4. Decode: $4.0\ \text{GB} / (0.55 \times 400\ \text{GB/s}) = 18.2$ ms, plus $1.5$ ms, gives $19.7$ ms per token, or about 51 tokens per second.

Now the workload. The 1,800 routine replies need $1{,}800 \times 300 = 540{,}000$ output tokens, which at 51 tokens per second is about 10,600 s, or 2.9 hours. Prefill for $1{,}800 \times 1{,}500 = 2.7$ million input tokens costs $2 \times 8 \times 10^9 \times 2.7 \times 10^6 = 4.3 \times 10^{16}$ FLOPs; assuming the machine sustains 20 TFLOPS of dense compute in practice (an assumption, check your own hardware), that is about 2,160 s, or 0.6 hours. Roughly 3.5 hours of work a day: one machine copes. A 70-billion-parameter model at 4 bits, by contrast, is a 35 GB file that does not fit in 16 GB at all, and even on a machine with enough memory at the same bandwidth it would decode at about 6 tokens per second, taking about 24 hours for the same 540,000 tokens. Bigger is not better if it cannot keep up.

A rules-first router in code

def route(task, local_ok=True):
    # 1. Hard rules first: never overridden by cost or speed.
    if task["sensitive"] or task.get("pinned") == "local":
        return "local" if local_ok else "fail: local required but unavailable"
    if task.get("pinned") == "hosted":
        return "hosted"
    # 2. Capacity: does the prompt fit the local context budget?
    if task["context_tokens"] > 32_000:
        return "hosted"
    # 3. Difficulty estimate (0..1) from a classifier or heuristics.
    if task["difficulty"] >= 0.7:
        return "hosted"
    return "local" if local_ok else "hosted"

tasks = [
    {"sensitive": True,  "context_tokens": 3_000,  "difficulty": 0.9},
    {"sensitive": False, "context_tokens": 1_800,  "difficulty": 0.2},
    {"sensitive": False, "context_tokens": 90_000, "difficulty": 0.3},
    {"sensitive": False, "context_tokens": 2_500,  "difficulty": 0.8},
    {"sensitive": False, "context_tokens": 1_500,  "difficulty": 0.1, "pinned": "local"},
]
print([route(t) for t in tasks])
print(route(tasks[0], local_ok=False))

The order of checks encodes the priorities: rules, then capacity, then difficulty. It prints ['local', 'local', 'hosted', 'hosted', 'local']: the sensitive task stays local despite being hard, and the long-context and hard tasks escalate. When the local model is down, the sensitive task returns a failure rather than leaking to a hosted model.

Measuring whether routing works

Track four numbers per week: the escalation rate (share sent to hosted), the local failure rate (local answers that later failed review), cost per completed task, and 95th-percentile latency. Then sweep the difficulty threshold (0.7 in the code) up and down on a labelled sample and plot cost against quality. Pick the point where extra spend stops buying noticeable quality. Also have a person read a small random sample of local answers each week. The local failure rate is only as good as the checks that measure it, and human review catches kinds of failure that automated checks were never written to detect.

Common pitfalls

• Cost logic overriding rules. If a budget check or a fallback can send sensitive data to a hosted model, the rules are not hard rules. Filter allowed models first.
• Uncapped cascades. A task that fails locally, escalates, fails again and retries can cost more than going hosted directly. Cap attempts and log each one.
• Weak checks. A cascade is only as good as its check. "The output is valid JSON" is not "the answer is right"; use task-specific tests where possible.
• Ignoring prefill. Long prompts make local prefill slow, especially on machines with modest compute. Size for input tokens as well as output tokens.
• Stale prices and models. Provider prices and model line-ups change often; re-check before relying on cost projections.
• Sizing for the average. Work arrives in bursts. Size the local side for peak hour, or allow overflow to a queue.
• Not logging the routing decision. Without the model, reason and cost recorded per request, you cannot tune the router or explain its behaviour.

In practice

1. Write down your rules first: what must never leave your machine.
2. List your recurring AI tasks and mark each as routine or hard.
3. Use the AI Index to check which models your machine runs well, and test one on your routine tasks.
4. Send routine work local, escalate the rest, and review the log after a week: what escalated, why, and what it cost.