“It got the right answer, so why did it take fourteen steps?” A site engineer asked this out loud last spring. An agent trajectory is the full path of reasoning, tool calls, and decisions an agent follows toward a goal. Outcome scores show where it landed. The path shows whether it can be trusted to land there again.
That question came from a regional power utility. Its dispatch agent scored 93 percent on the standard outcome check, so leaders called the pilot a success. The engineers kept reading the logs anyway. What they found changed how the whole team measures agents, and it is simpler than it sounds.
This guide walks through that story in the order the team lived it. You will see how to read an agent trajectory, score it, and start without new tooling. A five-check sequence called the Detour Test sits at the center.
THE HIDDEN PATH
The utility built an agent to triage outage tickets after storms. It read each ticket, checked the feeder map, and assigned a repair crew. Leaders tracked one number: did the right crew type get dispatched? On 400 sampled tickets, the answer was yes 93 percent of the time.
Then an engineer opened the replay of a single ticket. The agent had taken fourteen steps to reach a correct answer. Six would have done the job. It queried the same substation record three times and refreshed a feeder map it had just loaded.
That is the gap between a result and a habit. A correct outcome can come from sound reasoning or from luck. Outcome scores cannot tell the two apart. An agent trajectory can, because it records every decision the agent made along the way.
The gap matters most on bad days. In a normal week, extra steps cost seconds. During a storm with 31,000 open tickets, those seconds pile up. The agent’s detours added an estimated 40 minutes before the first crews rolled out.
We tell clients the same thing every time. Do not throw out your outcome checks. They tell you whether the work landed. They cannot tell you whether the work was done well, or whether it will hold up next week.
WHY SCORES MISLEAD
Final-answer scoring is popular because it is cheap. One label, right or wrong, can be checked by a script. The trouble is that three very different agents earn the same passing grade. The utility found all three hiding in its own data.
PATTERN 1
Some agents reach the right crew by guessing well. They skip the feeder map and infer from the street name. It works until two streets share a name. The agent trajectory in these runs shows no lookup at all. In the replay set, 11 percent of correct dispatches came from a guess.
PATTERN 2
Other agents get there the long way. Median path length was 14 steps against a practical minimum of six. Every extra step added a call to a busy outage system. A long agent trajectory also hides cost, because the final answer looks fine while the delay builds.
PATTERN 3
The third group works only under ideal conditions. These agents succeed when the map is fresh and fail when it lags. Scores gathered in calm weeks never showed this. The storm did, and by then crews were waiting on instructions that pointed to the wrong feeder.
None of this means outcome metrics are wrong. It means they are incomplete. Treat the final answer as the headline and the path as the story behind it. Operators need both before they trust an agent with a storm night.
THE RECORD
Good judgment needs good records. Evaluation traces are the saved logs of single runs, covering the input, each reasoning step, each tool call, and each result. Most teams already store something close. It usually sits in a log file nobody reads.
The utility started small. It pulled 200 runs from the previous month and wrote each trace into a shared sheet, one row per step. Two engineers and one operator did the work in four days. No new software was bought, and no outside help was needed.
Then came the first useful change. They stopped scoring runs and started scoring steps. Step-level evaluation asks a narrow question of each row: was this step needed, correct, and well timed? Reading a full agent trajectory this way takes about six minutes per run once the sheet is set up.
That pace sounds slow, yet it scales better than people expect. Reviewers learn the patterns fast. By the fiftieth trace, most could spot a detour in under two minutes. Pair each reviewer with an operator who knows the real work, and the quality of judgment rises quickly.
Keep the sample honest. Pull traces from calm days and storm days, and include a few the agent got wrong. A sample made only of failures teaches you what broke. A mixed sample teaches you what normal looks like.
THE SEQUENCE
The Detour Test is the five-check sequence the utility built from its first 200 traces. You score each run on five yes or no questions. Each question targets a different part of the agent trajectory. Together they fit inside the six minutes a reviewer already spends per run.
CHECKS 1 AND 2
Check one asks whether every step was necessary. Divide the steps taken by the practical minimum. That quotient is the detour ratio. A ratio near 1.0 is clean. The utility’s agent averaged 2.3, which meant more than half its steps were wasted.
Check two asks whether the steps came in a sensible order. A map lookup should come before a crew choice, never after. Order errors are rarer than detours, about 8 percent of runs. Each one tends to produce a wrong dispatch.
CHECKS 3 AND 4
Check three covers tool call accuracy. Did the agent call the right tool with the right inputs? A call to the correct system with the wrong feeder code still fails. Tool call accuracy is the cleanest number in the test, because a reviewer can verify it from the log alone.
Check four asks about recovery. When a call failed or returned stale data, did the agent adjust? In 62 of 200 runs, a map refresh lagged. Only 19 of those runs showed a retry or a fallback. The rest pressed on with old data.
CHECKS 5
Check five asks whether the agent stopped at the right moment. Some agents keep checking after the answer is settled. Others quit early with a guess. A clean stop is brief and confident. It is also one of the easiest checks for operators to learn.
Score a run as a pass if it clears at least four of the five checks. The utility’s first batch passed 38 percent of runs. After prompt and tool changes aimed at the failures, the next batch passed 81 percent. Dispatch accuracy barely moved, from 93 to 94 percent.
Read that again. The agent trajectory improved sharply while the headline score stayed flat. A team watching only the final number would have called the work finished and missed the whole gain.
THE HABIT
A test only helps if someone runs it. The utility settled on a simple rhythm. Every Thursday, one operator and one engineer review ten evaluation traces together. They log the detour ratio and the five checks. The session takes thirty minutes and needs no slides.
In the first month, the team found four recurring detours and fixed three. In the second month, it added step-level evaluation to the release checklist. No new agent version ships until its traces pass at the same bar as the last one.
Autonomy decisions also changed. The team had been debating freely how much freedom the dispatch agent should have during storms. Once paths were visible, the debate got concrete. Questions about how much autonomy the agent deserves could finally be answered with evidence.
Sequencing helps as well. When several agents hand work to each other, one detour upstream becomes three downstream. Notes on planning multi-step agent work explain how to order those handoffs so traces stay readable for reviewers.
Expect resistance in week one. Operators may see trace review as extra work. Show them one replay of a detour, and most change their minds. Nobody enjoys learning that an agent spent forty minutes on a fifteen-second task.
The next test is already forming. Agents are starting to rewrite their own plans mid-run. Once the agent trajectory shifts while it is being walked, the practical minimum stops being a fixed number. Teams that can score a moving path will keep their operators’ trust. Teams that cannot will be guessing again.
How many agent trajectory samples should a team review each week?
Ten to fifteen traces per agent per week is enough to spot recurring detours. Pull them from different days and include a few failed runs. Review each agent trajectory with one engineer and one operator together. After a release change, raise the sample to 30 for two weeks, then settle back.
What is a good detour ratio for a production AI agent?
A ratio between 1.0 and 1.5 is a sensible target for most operational tasks. Anything above 2.0 suggests wasted calls or weak planning. Set the practical minimum by having an expert complete the same task by hand, then count the steps required for a typical agent trajectory.
How is step-level evaluation different from checking final answers?
Final-answer checks label a whole run as right or wrong. Step-level evaluation scores each action inside the run for need, correctness, and timing. It takes longer per run but shows exactly where an agent wastes effort, which a single final label can never reveal.
Which tool call accuracy errors show up most often in agent logs?
Wrong parameters appear most often, such as a stale identifier or a mistyped region code. Repeated identical calls come next, followed by calls to a tool the task never needed. Tool call accuracy improves quickly once reviewers tag each error type in the shared sheet.
Can small teams review an agent trajectory without special software?
Yes. A shared spreadsheet with one row per step works well for the first few hundred runs. Add columns for tool, inputs, result, and a pass or fail mark per check. Move to dedicated tooling only when review volume outgrows what two people can read.