APIs, Tokens and Pricing

If you build anything on top of a hosted AI model, every request costs money and takes time. Understanding how those requests are priced lets you estimate a monthly bill before you ship, instead of discovering it afterwards.

Calling a model over an API

An API (application programming interface) is a defined way for one program to ask another program to do something. A hosted model's API is a web address your code sends an HTTPS request to. The request contains the conversation so far and some settings; the response contains the model's reply.

Most providers use a shape similar to the one below. Field names differ between providers, so treat this as a generic, OpenAI-compatible illustration rather than any one vendor's exact format.

{
  "model": "example-model",
  "max_tokens": 500,
  "messages": [
    {"role": "system", "content": "You are a concise support assistant."},
    {"role": "user", "content": "How do I reset my password?"}
  ]
}

The system message sets standing instructions; user and assistant messages hold the conversation. An important detail: the API is stateless, meaning the server does not remember your previous call. To continue a conversation you send the whole history again each time. That is why long chats get more expensive per turn.

Tokens: the unit you pay for

Models read and write tokens, which are chunks of text: often a whole short word, sometimes part of a longer word or a punctuation mark. A common rule of thumb for English is that 1,000 tokens is roughly 750 words, but it varies by language and by the model's tokenizer.

Every request has two token counts:

• Input tokens: everything you send, including the system prompt, the conversation history, any documents you paste in, and tool definitions.
• Output tokens: everything the model writes back.

Providers price these separately, quoted per million tokens. Output is usually priced higher than input, often several times higher, because generating each output token takes a full pass through the model, while input tokens can be processed in parallel. As one example, as of November 2026 Anthropic's published API prices put output at five times the input price for its current models.

A worked monthly cost example

The prices below are illustrative round numbers, not any provider's actual rates. Check the current pricing page of whichever provider you use.

Assume a support bot with:

• Input price: $3 per million tokens; output price: $15 per million tokens.
• 10,000 conversations per day, 30 days a month = 300,000 requests.
• Each request sends 2,000 input tokens and receives 500 output tokens.

Notice that output costs more than input here even though there is four times less of it. Capping response length is one of the simplest cost controls.

Prompt caching

Many requests start with the same long prefix: a big system prompt, a policy document, a set of tool definitions. Prompt caching lets the provider store the processed form of that prefix so later requests that begin with exactly the same text are cheaper and faster. Some providers cache automatically; others need you to mark what to cache. Writing to the cache can cost more than normal input, and cached prefixes are kept for anything from a few minutes to many hours, depending on provider, model and settings. As of November 2026, for example, Anthropic charges 1.25 times the input price for a five-minute cache write (2 times for a one-hour cache) and 0.1 times for a cache read, or less on some newer models.

Continuing the example: if 1,500 of the 2,000 input tokens are an identical system prompt, and cache reads cost 10% of the input price, then 450 million tokens cost $0.30 per million ($135) and the remaining 150 million cost $3 per million ($450). Input drops from $1,800 to about $585, before the small cost of cache writes. The rule: put stable content first and variable content (the user's question) last, because caching only works on an exact matching prefix.

Batch discounts

If you don't need an answer immediately, for example nightly document tagging or evaluating thousands of test cases, many providers offer a batch interface. You upload many requests at once and collect results later, typically within 24 hours. As of November 2026, both OpenAI and Anthropic advertise 50% off standard prices for batch jobs. In the example, a latency-tolerant workload of the same size would cost about $2,025 instead of $4,050.

Rate limits

Providers cap how much you can send, usually as requests per minute and tokens per minute, with limits that typically rise as your account's payment and usage history grows. Exceed them and you get an error (HTTP status 429, "too many requests"). Well-behaved clients honour a Retry-After header if the provider sends one, and otherwise retry with exponential backoff: wait a moment, retry, and double the wait after each failure, with a little randomness so many clients don't retry in lockstep.

Streaming

By default the API returns the whole reply once it is finished. With streaming, the server sends tokens as they are generated, usually over a long-lived HTTP connection using server-sent events. Streaming does not make the total answer faster or cheaper; it makes the first words appear sooner, which matters a lot for how responsive a chat interface feels. Two latency numbers are worth tracking separately: time to first token and total time to complete.

Going deeper

A cost model you can reason with

For a workload of $N$ requests, each with $t_{\text{in}}$ input tokens and $t_{\text{out}}$ output tokens, and prices $p_{\text{in}}$ and $p_{\text{out}}$ quoted per million tokens, the bill is

Everything else in this lesson modifies one of those terms. Caching lowers the effective $p_{\text{in}}$ for part of $t_{\text{in}}$; batching multiplies the whole thing by roughly 0.5; trimming history lowers $t_{\text{in}}$; a tighter max_tokens caps $t_{\text{out}}$. When a bill surprises you, find out which term moved.

Why chat history grows faster than you expect

Because the API is stateless, turn $k$ of a conversation resends the system prompt ($S$ tokens), all $k$ user messages (about $u$ tokens each) and the $k-1$ earlier assistant replies (about $a$ tokens each). Summing over a conversation of $K$ turns gives total input

The last two terms grow with $K^2$, so a conversation twice as long costs roughly four times as much input once history dominates.

Worked example. Take $S = 1{,}000$, $u = 50$, $a = 250$ and $K = 20$ turns:

• System prompt: $20 \times 1{,}000 = 20{,}000$ tokens.
• User messages: $50 \times 210 = 10{,}500$ tokens (since $20 \times 21 / 2 = 210$).
• Earlier replies: $250 \times 190 = 47{,}500$ tokens (since $20 \times 19 / 2 = 190$).
• Total input: 78,000 tokens. Output: $20 \times 250 = 5{,}000$ tokens.

At the illustrative prices from earlier (USD 3 and USD 15 per million), that is USD 0.234 of input and USD 0.075 of output, about USD 0.31 per conversation. If the server remembered history you would only pay for about 21,000 input tokens; resending is why input dominates. The standard fixes are to drop or summarize old turns past a budget, and to keep the stable part (system prompt, tool definitions) at the front so it can be cached.

When does caching pay off?

Let a cache write cost $w$ times the normal input price and a cache read cost $r$ times it. If an identical prefix of $P$ tokens is used by $n$ requests before the cache expires, the prefix costs $P\,(w + r(n-1))$ with caching and $P\,n$ without. With the multipliers quoted earlier for Anthropic's five-minute cache ($w = 1.25$, $r = 0.1$):

• $n = 1$: 1.25 versus 1. Caching a prefix you never reuse costs 25% more.
• $n = 2$: 1.35 versus 2. You already save about a third.
• $n = 100$: 11.15 versus 100. You save about 89%.

Solving $w + r(n-1) < n$ gives $n > (w - r)/(1 - r) \approx 1.28$, so a prefix reused even twice within the cache lifetime is worth caching. What breaks caching in practice is not the arithmetic but tiny changes to the prefix: a timestamp at the top of the system prompt, tools listed in a different order, or a user's name inserted early. Any change ends the matching prefix at that point.

Latency: two numbers, one formula

For a single streamed request, a useful first approximation is

where $T_{\text{first}}$ is the time to first token (queueing, network and processing the input) and $R$ is the generation speed in tokens per second. With $T_{\text{first}} = 0.5$ s, $R = 100$ tokens/s and a 500-token answer, the user waits about 5.5 s for the full reply but sees text after 0.5 s. Halving the answer to 250 tokens saves 2.5 s; a model that is 20% faster at generating saves about 0.8 s. Long inputs raise $T_{\text{first}}$, because the model must process every input token before it writes anything (see the lesson on KV cache and context cost).

Retries without stampedes

When many clients hit a rate limit at once and all retry after the same fixed delay, they collide again. Exponential backoff with full jitter draws each wait at random:

where $j$ is the retry number. The sketch below estimates monthly cost and retries a rate-limited call. It is provider-neutral: plug in your own client call and the exception your SDK raises for HTTP 429.

import random, time

def monthly_cost(requests, t_in, t_out, p_in, p_out,
                 cached_frac=0.0, read_mult=0.1, batch=False):
    """Prices are per million tokens. Ignores cache-write cost."""
    cached = t_in * cached_frac
    input_cost = cached * p_in * read_mult + (t_in - cached) * p_in
    total = requests * (input_cost + t_out * p_out) / 1e6
    return total * (0.5 if batch else 1.0)

print(monthly_cost(300_000, 2_000, 500, 3, 15))                    # 4050.0
print(monthly_cost(300_000, 2_000, 500, 3, 15, cached_frac=0.75))  # 2835.0

class RateLimited(Exception):
    def __init__(self, retry_after=None):
        self.retry_after = retry_after

def call_with_backoff(fn, max_retries=6, base=0.5, cap=30.0):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except RateLimited as e:
            if attempt == max_retries:
                raise
            if e.retry_after is not None:      # the server knows best
                delay = e.retry_after
            else:                              # full jitter
                delay = random.uniform(0, min(cap, base * 2 ** attempt))
            time.sleep(delay)

The second figure reproduces the caching example from earlier: 75% of input served from cache brings the month from USD 4,050 to USD 2,835 (USD 585 input plus USD 2,250 output), before cache-write costs.

Counting tokens properly

The 750-words-per-1,000-tokens rule is only a planning figure. Tokenizers are usually learned with variants of byte-pair encoding, which merges frequent character sequences into single tokens, so common English words are cheap while code, numbers, rare names and many non-English languages can take noticeably more tokens for the same content. Before committing to a budget, run a sample of real prompts through the provider's own token-counting tool or tokenizer, and log the token counts the API reports with each response; those reported counts, not your estimates, are what you are billed for.

Common pitfalls

• Estimating with toy prompts. Real requests carry system prompts, tool definitions, retrieved documents and history. Measure with production-shaped inputs.
• Forgetting hidden output. Reasoning models can generate internal reasoning tokens that are billed as output even when they are not shown; budget for them and cap them where the API allows.
• Breaking the cache by accident. Timestamps, request IDs or reordered tools near the top of the prompt silently disable prefix caching.
• Retrying everything. Retry 429s and transient server errors, not 400-class errors caused by a bad request, which will fail the same way again.
• Unbounded loops. An agent or retry loop without a step and spend cap can run up a large bill overnight.
• Hard-coding prices. Prices change; keep them in configuration and re-check the provider's pricing page.

In practice

• Before building, estimate: requests per month × (input tokens × input price + output tokens × output price). Do it with real prompt sizes, including history and documents.
• Set a sensible max_tokens and trim conversation history you no longer need.
• Order prompts as stable-first, variable-last so caching can help.
• Move anything that can wait into batch jobs.
• Handle 429 errors with backoff, and stream responses in user-facing interfaces.
• Re-check prices periodically; they change often, usually downward.