LLM Function Calling: A Production Engineering Guide

Function calling is not a special mode or a separate API endpoint. It is a pattern built on top of the standard completions interface. You send a request that includes both your conversation and a list of tool definitions formatted as JSON Schema objects. The model processes everything together — the conversation, the tool descriptions, the parameter schemas — and then decides how to respond.

When the model decides to call a function, it does not return a text reply. Instead it returns a structured completion that names the function and provides a JSON object with the argument values it has inferred from the conversation. Your application code receives this completion, extracts the function name and arguments, executes the actual function call, and appends the result back to the conversation as a new message. The model then receives the updated conversation — now including the function result — and continues reasoning from there.

This loop can run multiple times in a single user interaction. A single request might trigger three or four function calls before the model has enough information to produce a final response. Each iteration extends the conversation, which means the input token count grows with every round trip. OpenAI refers to this mechanism as function calling. Anthropic refers to it as tool use. Google refers to it as function declarations. The underlying loop is identical across all three providers.

Every tool definition has three parts: a name, a description, and a parameter schema. The name and description are the primary signal the model uses to decide whether to call this tool at all. A description that says "handles data" is functionally useless. A description that says "retrieves the current account balance for a given customer ID from the billing system" gives the model enough context to make a correct tool selection decision even when similar tools are registered alongside it.

Parameter surfaces should be as narrow as production requirements allow. A common mistake is registering a tool that accepts a raw SQL query string as a parameter because it is flexible for development. In production, that wide surface gives the model room to construct queries that are syntactically valid but semantically wrong. Replace free-form parameters with structured alternatives wherever possible: specific field names instead of query strings, typed numeric ranges instead of open string fields, explicit boolean flags instead of instruction-bearing text fields.

Enumerate values whenever the valid set is finite. If a parameter accepts one of six status codes, define it as an enum in the schema rather than a string. If a parameter represents a priority level with four valid values, enumerate them. Enum constraints are enforced at the schema level and prevent an entire class of argument hallucination where the model invents a plausible-sounding value that does not exist in the actual system. The multi-agent design patterns guide covers how tool schema quality affects orchestration reliability when multiple agents share a tool registry.

Tool definitions are serialized into the prompt on every request. A minimal tool definition — a short name, a two-sentence description, and two or three parameters — consumes roughly 800 to 1,000 input tokens. That cost is paid on every single request, regardless of whether the model uses the tool. At 10 tools, the overhead is approximately 9,500 tokens. At 20 tools, approximately 19,000 tokens. At 58 tools, which is not an unusual count for an enterprise agentic system with integrations across multiple services, the tool list alone consumes roughly 55,000 input tokens before a single word of conversation has been included.

Tool routing is the standard mitigation. Instead of registering the full tool set on every request, the system maintains a registry of all available tools and selects a relevant subset — typically 10 to 15 tools — based on the current conversation context before making the LLM call. The routing layer itself is a lightweight classifier that examines the user intent and the current agent state. Implemented well, tool routing cuts the tool-related token overhead by 70 to 85 percent. For additional context on evaluating this in production, see the post on LLM agent evaluation in production.

The first principle is to validate before you execute. When the model returns a function call, run the arguments through your parameter validation logic before calling the actual function. If the arguments are malformed — a required field is missing, a value is outside the acceptable range, a referenced entity does not exist — return a structured error description to the model rather than letting the function fail at runtime. The model can recover from explicit, structured feedback. It cannot recover from a swallowed exception or a generic 500 error with no context.

The second principle is to cap retries per function call, not per conversation turn. Most teams implement retry logic at the conversation level: they catch a failed turn and retry the entire thing. The problem is that function calling loops can have two or three independent retry layers — one in the application code, one in the agent framework, and sometimes one in the LLM client library. An uncapped loop across two retry layers with a limit of five each can generate 25 or more LLM calls before timing out. Cap retries at the function invocation level: three attempts maximum per function call, then surface a specific failure message to the model or escalate to a human fallback.

The third principle is to build an explicit fallback path. Every agent that calls functions should have a defined behavior for the case where a function is unavailable, consistently failing, or returning results the agent cannot use. The fallback does not need to be sophisticated — it can be as simple as responding to the user with a specific error message and a suggested next step. What it cannot be is silence, a false success, or an infinite loop. Define the fallback explicitly and test it with chaos injection before deploying to production.

Parallel function calling allows the model to return multiple function invocation specifications in a single response. Your runtime executes them concurrently, then appends all results to the conversation before the next model call. For independent read operations — fetching user profile data and account balance simultaneously, or running three separate search queries in parallel — this is unambiguously the right approach. Latency drops by the factor of the parallelism, and correctness is unaffected because the operations have no state dependencies on each other.

For write operations or any pair of functions with state dependencies, parallel execution is dangerous. If a model returns two function calls where the second logically depends on the result of the first, executing them in parallel produces a race condition. The second function runs against stale state, potentially writing incorrect data or producing a result that the model then reasons from incorrectly. This is particularly insidious because the error does not surface as an exception — it surfaces as subtly wrong agent behavior that is hard to trace back to the concurrent execution. Enforce sequential execution for any function that writes state or has explicit ordering dependencies.

Schema drift occurs when a function's actual interface changes but the registered tool definition does not — the model continues generating arguments for the old schema, producing failures that look like model errors but are really deployment coordination failures. Parallel explosion happens when a model with broad tool access returns five or six parallel function calls for a request that warranted one, overwhelming downstream services. Context accumulation compounds across multi-turn sessions as function results pile up in the conversation, eventually degrading reasoning quality and inflating costs. Retry storms arise from uncapped retry layers as described above. Argument hallucination produces syntactically valid but semantically wrong arguments that fail silently. Tool deregistration — removing a tool the model has learned to rely on without retraining or adjusting the system prompt — produces persistent wrong tool selection until the prompt is updated. Each of these failure modes is predictable, and each has a specific design response. None of them are edge cases.