AI Agent Evaluation: A Framework That Actually Works

Most teams evaluating AI agents run the agent, check whether the final answer looks right, and call it done. That works for a simple LLM call. It fails completely for an agent.

The reason is structural. A language model takes one input and produces one output. An agent takes an objective and produces a sequence of decisions, tool calls, intermediate observations, and recoveries before it produces a final output. The final output can look correct while the path to it was expensive, brittle, or unsafe.

Agent evaluation breaks the standard LLM eval model in four ways. First, the output depends on a sequence of decisions, not a single one. Second, tool calls are side effects — when an agent writes to a database or sends an email, wrong decisions have consequences that do not disappear when the agent produces its next token. Third, the cost of the path matters: two agents that both succeed are not equivalent if one uses three tool calls and one uses fifteen. Fourth, safety and refusal are behaviors that require explicit testing.

The companion post on LLM agent evaluation in production covers the tooling side — tracing, logging, and the infrastructure you need to capture agent runs. This post is about the methodology: what to measure, how to score it, and how to build the datasets that make the scoring meaningful.

Each layer requires its own evaluation dataset, its own scoring logic, and its own pass/fail criteria. Teams that try to collapse all four into a single end-to-end pass/fail lose the diagnostic signal that tells them which layer is failing.

Task success evaluation asks one question: did the agent complete the task it was given? The scoring function depends on the task type. For tasks with deterministic correct answers (data extraction, calculation, lookup), a string match or structured comparison is appropriate. For tasks with open ended outputs (research summaries, code generation, plan drafts), you need a judge. LLM as judge scoring has real limitations, but it is the practical option for open ended agent tasks at scale. The key is to give the judge a rubric, not just the question and the answer.

A reasonable task success dataset has at minimum: 30 to 50 cases covering the task types the agent is actually deployed for; 10 to 15 edge cases where the task is ambiguous or requires the agent to ask a clarifying question; and 5 to 10 cases where the task cannot be completed and the correct behavior is to report failure rather than guess.

Trajectory quality evaluation looks at the steps the agent took, not just the endpoint. It scores three properties:

Trajectory scoring requires storing full run traces, not just inputs and outputs. The agentic AI production architecture guide covers how to structure traces for downstream evaluation use.

Tool use accuracy is the layer most teams miss, and it is where most agent bugs actually hide. An agent can complete a task successfully in eval but fail in production because the eval tasks happened to trigger correct tool use and the production cases trigger edge cases in how the agent selects tools or constructs arguments.

Tool selection accuracy measures whether the agent chose the right tool for each step. Score it by comparing the agent's tool selection at each step against an annotated reference trajectory. A tool selection accuracy of 0.95 or above is a reasonable production bar for agents with a clearly defined scope. Below 0.90 means the agent is regularly selecting wrong tools, which will surface as production failures on edge cases.

Argument quality measures whether the arguments passed to each tool were correct. Use a rubric: required fields present, types correct, values within expected range, no hallucinated argument keys. Tool call sequencing measures whether the agent called tools in the correct order when order matters.

Safety evaluation tests the agent's refusal and flagging behavior. A safety evaluation dataset has three categories of cases: refusal cases (instructions the agent must refuse entirely), flagging cases (instructions the agent should flag for human review before proceeding), and boundary cases (instructions at the edge of the agent's defined scope where the correct behavior is to clarify, not proceed silently).

Score safety evaluation as a binary per case: the agent either behaved correctly or it did not. There is no partial credit for almost refusing. A safety pass rate of less than 100% on refusal cases is a blocking issue before production deployment.

Start from production traces, not synthetic cases. Synthetic evaluation cases are easy to generate and systematically miss the long tail of real behavior. Real production traces capture the actual distribution of tasks, the actual ambiguities in user instructions, and the actual edge cases that production throws at the agent. Even a small set of real traces — 50 to 100 — produces a far more diagnostic evaluation dataset than 500 synthetic cases.

If the agent is not yet in production, generate traces from a pilot with internal users or domain experts who are willing to try adversarial and edge case inputs. Do not use the same people who built the agent — they will not find the edge cases.

Annotate at the step level, not just the task level. A trace annotation that says "this run succeeded" is almost useless for building trajectory and tool use evaluation datasets. Annotate at the step level: which tool calls were correct, which were suboptimal, which were wrong, and what the correct action at that step would have been.

A useful minimum viable evaluation dataset for a production agent has: 40 to 60 task success cases covering the full task distribution; 20 to 30 trajectory annotation cases with step level labels; 30 to 40 tool use annotation cases with labels at the argument level; and 20 to 30 safety cases covering refusal, flagging, and boundary behavior. That is 110 to 160 cases in total — the minimum that gives you meaningful coverage across all four layers.

Agent evaluation is expensive: each case requires a full agent run, which consumes tokens, takes time, and calls external tools. A dataset of 150 cases might cost $5 to $15 per full evaluation run depending on the model and the agent's average step count. That is affordable. It is not so cheap that you want to run it carelessly.

Structure your regression suite by severity tier. Tier 1 cases — safety cases and critical task success cases — run on every pull request that touches agent logic. If any Tier 1 case fails, the pull request cannot merge. Tier 2 cases — trajectory quality, tool use accuracy, and secondary task success cases — run on every merge to the main branch and block the next deployment unless explicitly overridden. Tier 3 cases — the full evaluation suite including boundary and edge cases — run on a scheduled cadence (nightly or weekly) and generate a report without blocking deployment.

The consulting side of agent evaluation — scoping the evaluation framework for a specific agent, defining the annotation schema, and wiring evaluation into the CI/CD pipeline — is work that benefits from a structured engagement. If you are standing this up for the first time, the agentic AI consulting service covers evaluation framework design as part of the architecture phase.