Tool Use and Function Calling

A language model on its own can only produce text. It cannot check today's weather, look up an order or send an email. Tool use (also called function calling) is the mechanism that connects a model to those actions, and it is the foundation of every AI agent.

The key idea: the model asks, your code acts

The model never runs anything. When it decides a tool would help, it outputs a structured request, essentially "please call get_order_status with order_id = 88123", and stops. Your application reads that request, decides whether to honour it, runs the real function, and sends the result back to the model, which then continues.

An analogy: the model is a skilled office manager who can't leave the desk. It writes requests on forms ("pull file 88123"), and a clerk (your code) checks each form, fetches the file and hands it back. The manager never touches the filing cabinet directly, and the clerk can refuse forms that break the rules.

Describing tools with JSON Schema

You tell the model which tools exist by sending definitions along with the conversation. Each has a name, a plain-language description, and a parameter specification written in JSON Schema, a standard format for describing the shape of JSON data (which fields exist, their types, which are required). The example below is a generic, OpenAI-compatible shape; providers differ in field names.

{
  "type": "function",
  "function": {
    "name": "get_order_status",
    "description": "Look up the shipping status of one order by its ID.",
    "parameters": {
      "type": "object",
      "properties": {
        "order_id": {"type": "string", "description": "Order number, e.g. 88123"}
      },
      "required": ["order_id"]
    }
  }
}

The description matters as much as the schema. The model chooses tools by reading descriptions, so write them as you would for a new colleague: what the tool does, when to use it, and what it does not do. Tool definitions also count as input tokens on every request.

What the model sends back

When the model wants a tool, its reply contains a tool call instead of (or as well as) ordinary text. Again in a generic shape:

{
  "role": "assistant",
  "tool_calls": [{
    "id": "call_1",
    "type": "function",
    "function": {"name": "get_order_status", "arguments": "{\"order_id\": \"88123\"}"}
  }]
}

Note that the arguments arrive as text that should parse as JSON. Your code must parse and check it; the model can produce invalid or unexpected values.

The tool loop

A single question may need several tool calls. The standard pattern is a loop:

# Pseudocode: call_model and run_tool_safely stand in for your own helpers.
def run_task(system_prompt, user_message, tools, max_steps=10):
    messages = [system_prompt, user_message]
    for step in range(max_steps):          # always cap iterations
        reply = call_model(messages, tools)
        messages.append(reply)
        if not reply.tool_calls:
            return reply.content           # final answer
        for call in reply.tool_calls:
            result = run_tool_safely(call) # parse, validate, authorize, execute
            messages.append({"role": "tool", "tool_call_id": call.id,
                             "content": result})
    raise RuntimeError("Step limit reached")

Each pass sends the growing history back to the model, so token costs rise with every step. Many models can request several tools at once (parallel tool calls), which your code can run concurrently. Most APIs also let you control whether the model may, must, or must not use tools on a given turn.

Structured outputs

The same machinery is useful even when nothing is executed. If you need the model to return data in a fixed shape, such as an invoice's vendor, date and total, you can supply a JSON Schema and ask for output that matches it. Many providers now offer a structured output mode that constrains generation so the result parses and fits the schema (a reply cut off by a token limit, or a refusal, can still fall short, so handle those cases). Even then, check the values: a schema can guarantee that total is a number, not that it is the right number.

Safety: treat every tool call as untrusted input

The model's request is shaped by everything in its context, including web pages, emails and documents it has read, any of which may contain malicious instructions (prompt injection). So treat a tool call the way you would treat a form submitted by an anonymous website visitor:

• Validate inputs: check types, ranges and formats against the schema and your own business rules. Reject an order_id with SQL in it; never pass model output straight into a shell, a database query or a file path.
• Authorize per user: the tool should act with the permissions of the person using the app, not with a powerful service account. If the user can only see their own orders, the tool must enforce that, regardless of what the model asks for.
• Least privilege: give the model the narrowest tools that do the job. A "read calendar" tool is safer than a "run any API call" tool. Separate read tools from write tools.
• Confirm consequential actions: require human approval before anything that sends, pays, deletes or publishes.
• Limit and log: cap the number of steps and the spend per task, set timeouts, and record every call and result so you can audit what happened.
• Return errors usefully: if a call fails validation, return a clear error message as the tool result. Models often correct themselves when told what went wrong.

Going deeper

What the model is really doing

A tool call is ordinary next-token generation that has been trained, and often constrained, to produce a particular structure. During post-training, models see many conversations in which a tool definition appears in the context and the correct continuation is a well-formed call, followed by a tool result and a final answer. At inference time the model reads your tool definitions as part of its input and predicts that a call is the most useful continuation.

When a provider offers strict or structured modes, it can go further and constrain decoding. At each step the model produces a probability $p(t)$ for every token $t$ in its vocabulary. A constrained decoder masks out tokens that would make the output impossible to complete validly under the schema, then renormalizes:

This is why constrained output is guaranteed to parse (as long as generation is not cut off), and also why it cannot guarantee correct values: the mask only removes structurally invalid tokens; among valid ones, the model still chooses by its own probabilities.

The tool loop as a state machine

Think of the loop as a state machine over the message history $h_t$:

1. Model step: produce either a final answer or a set of tool calls $\{c_1, \dots, c_m\}$ given $h_t$ and the tool definitions.
2. Execution step: for each call, parse, validate, authorize and execute, producing results $\{r_1, \dots, r_m\}$.
3. Append: $h_{t+1} = h_t + \text{calls} + \text{results}$, and repeat until a final answer or a limit.

Two numbers follow from this structure.

Reliability. If the model picks the correct tool with probability 0.97 and fills the arguments correctly with probability 0.95, each call is right about $0.97 \times 0.95 \approx 0.92$ of the time, and a task needing four correct calls in a row succeeds about $0.92^4 \approx 0.72$ of the time (assuming independence, which is a simplification). Validation errors returned as tool results let the model repair many of these mistakes, which is why good error messages raise end-to-end success.

Cost. Tool definitions are input tokens on every model step. Worked example: 20 tools averaging 150 tokens each is 3,000 tokens per request. A task that takes 6 model steps sends those definitions 6 times, 18,000 tokens. At an illustrative USD 3 per million input tokens that is USD 0.054 per task for definitions alone, or USD 5,400 a month at 100,000 tasks. If the definitions are cached (written once at 1.25× and read 5 times at 0.1×), the same definitions cost $3{,}000 \times (1.25 + 5 \times 0.1) = 5{,}250$ token-equivalents per task, about USD 1,575 a month. Fewer, sharper tools help reliability and cost at once.

Latency. If one model step requests three independent lookups taking 0.8 s, 1.2 s and 0.5 s, running them sequentially takes 2.5 s; running them concurrently takes about 1.2 s, the slowest one. Parallel calls only help when the calls really are independent.

Validation in code, not in the prompt

The run_tool_safely helper in the loop above does the important work. Here is a minimal, runnable version using Pydantic, which validates arguments against a typed model and can also generate the JSON Schema you send to the model (OrderStatusArgs.model_json_schema()), so the schema and the validator never drift apart.

import json
from pydantic import BaseModel, Field, ValidationError

class OrderStatusArgs(BaseModel):
    order_id: str = Field(pattern=r"^\d{5,10}$")   # digits only

ORDERS = {"88123": {"owner": "u1", "status": "shipped"}}

def get_order_status(args: OrderStatusArgs, user_id: str) -> dict:
    order = ORDERS.get(args.order_id)
    if order is None or order["owner"] != user_id:   # authorize in code
        return {"error": "Order not found for this user."}
    return {"order_id": args.order_id, "status": order["status"]}

REGISTRY = {"get_order_status": (OrderStatusArgs, get_order_status)}

def run_tool_safely(name: str, raw_args: str, user_id: str) -> str:
    if name not in REGISTRY:
        return json.dumps({"error": f"Unknown tool: {name}"})
    schema, fn = REGISTRY[name]
    try:
        args = schema.model_validate_json(raw_args)   # parse + validate
    except ValidationError as e:
        return json.dumps({"error": "Invalid arguments",
                           "details": e.errors(include_url=False)})
    return json.dumps(fn(args, user_id))

print(run_tool_safely("get_order_status", '{"order_id": "88123"}', "u1"))
print(run_tool_safely("get_order_status", '{"order_id": "88123"}', "u2"))
print(run_tool_safely("get_order_status", '{"order_id": "1; DROP TABLE"}', "u1"))

Points worth noticing:

• The tool name is looked up in an allow-list (REGISTRY); the model cannot invoke anything that is not registered.
• user_id comes from your authenticated session, never from the model's arguments. The second call returns "not found" rather than another customer's order, and the message does not reveal whether the order exists.
• Malformed input never reaches business logic; the error comes back as data the model can read and correct.

Designing good tools

Tool design is interface design for a reader that is very literal and cannot ask follow-up questions:

• One clear job per tool. search_orders and cancel_order are better than manage_orders(action=...), because read and write permissions can then differ.
• Constrain arguments. Use enums for fixed choices, formats for dates, minimums and maximums for numbers, and additionalProperties: false to reject unexpected fields.
• Return compact, structured results. A 40,000-token raw API dump costs money on every later step and can bury the useful part. Return the fields the model needs, with IDs it can use in follow-up calls, and paginate large results.
• Make errors actionable. "date must be ISO 8601, e.g. 2026-11-05" teaches the model; "error 400" does not.

Common pitfalls

• Trusting the model's identity claims. Passing user_id or role as a model-filled argument lets injected text impersonate anyone.
• Over-broad tools. A generic run_sql or http_request tool turns any prompt injection into full database or network access.
• Huge tool catalogues. Dozens of overlapping tools increase wrong choices and token costs; load only those relevant to the task.
• Swallowing errors. Raising an exception that kills the loop, or returning an empty string, prevents the model from recovering.
• No idempotency on writes. If a call times out and the loop retries, a non-idempotent create_payment may run twice. Use idempotency keys.
• Assuming structured output means correct output. Validate business rules (totals add up, dates are in range) after parsing.

In practice

• Start with two or three well-described tools rather than twenty overlapping ones.
• Write tool descriptions and parameter descriptions carefully, then test with realistic and adversarial requests.
• Put validation, authorization and confirmation in code, never only in the prompt.
• Use structured outputs for data extraction, and still validate the values.
• Watch token usage per task: each loop iteration resends the history.