Four in five design decisions in a production workflow concern what happens when a step fails, yet the diagram shows only the fifth. An agent orchestrating workflows is judged by how it routes around failure: conditional branching, error handling, retry logic, and workflow versioning decide whether a bad step becomes a recovered step or an incident.
A workflow that coordinates several agents, tools, and approval steps is a distributed system with a language model in the middle. Distributed systems fail in partial, uneven ways. A call times out after its side effect has already landed. A retrieval returns stale content. A model answers fluently and incorrectly. Orchestration is the layer that decides what happens next in each of those cases, which makes it the place where reliability is won or lost.
Four capabilities carry most of that weight. Conditional branching routes work according to evidence rather than sequence. Error handling classifies what went wrong so the response fits the cause. Retry logic spends a bounded budget on faults likely to clear. Workflow versioning lets a running process survive the edits its owners keep making. The sections below take each in turn, then move to the tests that separate a platform built for failure from one built for the demo.
THE ASSUMPTION UNDERNEATH THE DIAGRAM
Practitioners tend to assume that orchestration means arranging steps in the right order: fetch the record, classify the request, decide, act, notify. The resulting diagram is a clean line with a few diamonds on it. It is easy to review, easy to present, and misleading about where the engineering effort goes.
Underneath, every step is a small state machine with more exits than the diagram admits. A step can succeed. It can fail loudly with an error code. It can time out with its effect unknown, which is the worst case, because the work may or may not have happened. It can return partial output, such as eleven of fifteen records. And any step involving a model can succeed in form while being wrong in substance.
A step with five exits, repeated across a twelve-step workflow, produces sixty transitions, and the diagram usually draws twelve. The other forty-eight are where on-call engineers spend their time. Skipping their design does not remove them. It leaves each one to be improvised during an incident by whoever happens to be awake.
The effort ratio confirms the pattern. A first working prototype of a multi-step agent workflow often takes a week. Hardening it for production, which means designing those other forty-eight transitions, commonly takes six to ten weeks. Roughly 70 percent of the logic written after the prototype exists to handle something other than success.
Seen this way, an agent orchestrating workflows is closer to a state-and-transition system with explicit rules than to a script. When dozens of such workflows run side by side, those rules also need a shared control plane, so that branching thresholds, retry limits, and escalation paths are set once and inherited rather than reinvented by whoever built each workflow.
THE ROUTING DECISION
Conditional branching sends work down different paths depending on what the workflow has learned so far. The basic form is a threshold: if the amount exceeds a limit, route to a person. The useful form branches on evidence about the work itself, and four signals do most of the lifting. Output confidence says how sure the step is. Context completeness says whether the inputs were fresh and whole. Value at stake says what a mistake would cost. Reversibility says whether the next action can be undone.
Consider an invoice exception workflow. An agent matches an invoice to a purchase order and emits a match score. A score above 0.92, an amount under 5,000 dollars, and a clean supplier history resolve automatically. A score between 0.75 and 0.92 triggers a second agent to gather receiving records and contract terms. Anything lower goes to a human queue with the evidence attached. In a well-tuned version of this design, roughly 65 to 75 percent of exceptions resolve without a person, 15 to 20 percent are enriched and then resolved, and the remainder reach a reviewer already assembled.
A tension sits underneath every branching design: should rules decide the route, or should the model? Rule-based routing is auditable and predictable, and it fails on inputs nobody anticipated. Model-decided routing handles novelty, and it fails in a way that is harder to catch, because a wrong route arrives with a confident justification. The workable answer in most enterprise settings is to let the model propose and let rules dispose. The model classifies the case and suggests a branch, then a deterministic gate checks the proposal against policy, value limits, and tool permissions before the branch is taken. That split keeps flexibility where it helps and predictability where an auditor will look.
Every conditional needs an explicit default. A missing else branch is the most common reason a workflow stalls without raising an alert: the input matches no condition, nothing fires, and the run sits in limbo until a customer asks about it. Designing the default branch first, usually a parked state with a named human owner, costs an hour and prevents stalls that take a week to notice.
THE FAILURE CONTRACT
Error handling goes wrong when it is treated as one thing. A better framing is a failure contract attached to each step, with four clauses. What does the step promise on success? How does it signal each class of failure? Which failures may it retry on its own? And where does it escalate, and what state does it leave behind?
The second clause depends on sorting failures into classes, because each class calls for a different response. Transient faults, such as timeouts, rate limits, and brief outages, usually clear on their own and justify a retry. Deterministic faults, such as a schema mismatch, a missing permission, or malformed input, fail identically every time and need a fix or a reroute. Semantic faults produce well-formed output with wrong content and need verification or a different approach. Policy faults produce output that breaks a rule and need a stop and an approval.
The common shortcut is a single catch-all: on any error, retry three times, then fail. That shortcut triples the cost of every deterministic fault and delays its detection. It does nothing for semantic faults, which raise no error at all. In typical workflow logs, perhaps 70 percent of raised errors are transient, 25 percent deterministic, and 5 percent policy related, while semantic faults raise nothing and surface downstream. That last group is small by count and can account for more than half of correction cost, since nobody notices it until a wrong decision has already propagated.
The fourth clause, the state a step leaves behind, is the one most often left blank. Suppose step seven of nine fails after steps five and six wrote to external systems. The workflow now holds a half-finished business transaction. It needs either compensating actions that reverse the earlier writes, or a parked state that preserves everything and tells an operator exactly what is done and what is not. Items that exhaust their handling should land in a visible dead-letter queue with full context attached, not vanish into a log file.
THE RETRY BUDGET
Retry logic earns its place only for transient faults, and even there it needs limits. A reasonable default is up to three attempts per step, with exponential backoff starting near two seconds, a ceiling around thirty, and random jitter so that many runs do not retry in lockstep. Beyond three attempts, the odds that a transient fault clears on its own fall sharply, and the delay stops being a retry and becomes an outage the workflow is hiding.
Per-step limits are not enough, because retries multiply across concurrent runs. If 400 runs are active when a retrieval service degrades and each retries three times, the struggling service receives 1,600 calls instead of 400, the reverse of what it needs. A retry budget fixes this by capping retries at a fraction of total calls, commonly 10 percent over a rolling window, for each downstream dependency. When the budget is spent, the workflow stops retrying and takes its fallback branch.
Idempotency decides whether retries are safe at all. An ambiguous timeout on a payment, a ticket creation, or a message send leaves the workflow unsure whether the action occurred. Every action with an external effect should carry an idempotency key derived from the run identifier and step identifier, so the receiving system recognizes a repeat and does nothing. Without that key, a retry policy is a duplication policy.
Model calls add a cost dimension. A self-correction loop that asks an agent to try again after a failed check feels clever and spends tokens without limit. A loop that runs on 2 percent of requests, with up to five passes each, can account for 15 to 20 percent of model spend. The sound rule is that a retry on a semantic fault must change something: added context, a different tool, a narrower question. Repeating the identical call and hoping for different output is not retry logic. It is a coin flip with an invoice.
Fallbacks complete the picture. When retries are exhausted, a degraded answer from cached data, a smaller model, or a human queue is usually better than a failed run. The fallback should be chosen in advance, per step, by someone who understands what a degraded answer costs the business.
THE LONG-RUNNING PROBLEM
Some workflows finish in seconds. Others wait. A supplier reassessment that requests documents, waits for a reply, routes to a reviewer, and waits again can run for two weeks. A fair planning assumption is that 30 to 40 percent of enterprise workflows involving a human approval last longer than a day, which creates a problem short workflows never face: the definition changes while instances are still running.
Picture a run on day nine. Meanwhile the owning group has released the workflow twice. One release moved an escalation threshold, and another added a verification step and swapped the model behind classification. The run in flight began under the old definition. What should it do now? Three options exist. It can run to completion on the version it started with. It can migrate to the new version at a safe checkpoint. Or it can restart from the beginning and discard its progress.
Pinning is the safest default, since an in-flight run then completes under the rules it started with and its behavior stays explainable. Migration is appropriate only after a compatibility check confirms that the new version’s state schema accepts the data the run has accumulated and that no step it already passed has changed meaning. Restart is a last resort, because it can repeat side effects the run has already caused.
All of this assumes that a version means more than a tag on a diagram. The bundle logic that governs agent rollback applies to workflow versioning as well: the graph, every branch threshold, the retry and escalation policy, the prompts, the tool permissions, and the model identifiers should travel together as one promotable and reversible unit. A rollback that restores only part of the definition restores nothing reliable.
Auditability is the quieter benefit. When each run records the version it executed under, a decision made three weeks ago can be explained with the rules of that day, not the rules of today. That matters for governance reviews, and it matters for tuning. A threshold change from 0.92 to 0.88 is a policy change with business consequences. It deserves the same review as any other policy change, not a quiet edit.
THE EVALUATION
Evaluating an orchestration approach is easiest when the demo is bent toward failure. Forward motion looks the same everywhere: a workflow is drawn, a run completes, a green indicator appears. Differences surface only when something breaks, so the evaluation should break something on purpose. Five requests reveal most of what matters:
Inject a timeout into a mid-workflow step, and watch whether the platform classifies it, retries within a stated budget, and records every attempt.
Return malformed output from a tool, and confirm the workflow stops retrying and routes the case instead of looping.
Release a new workflow version while a run waits on approval, and ask what happens to that run.
Stop a run after a step that wrote to an external system, and ask what state it leaves and who can see it.
Request the full history of a single run, including its version, the branches taken, and every retry.
Two kinds of platforms emerge from these tests. One is optimized for authoring: it makes the first workflow fast to draw and impressive to present. The other is optimized for operating: it makes the hundredth workflow safe to change on a Tuesday afternoon. The first looks better in a demo, and the second looks better at month three. The choice turns on how many workflows are expected to be live in eighteen months and who will be operating them.
Four measures show whether the investment is paying back. Mean time to detect a stuck run should be minutes, not days. The share of runs ending in an unhandled state should stay below half a percent. The escalation rate to humans should trend down as branches are tuned, without reaching zero, since zero usually means failures are being absorbed silently. And the ratio of retries to eventual successes shows whether retries are rescuing work or only delaying the inevitable.
Tradeoffs deserve equal attention. Thorough failure handling adds latency, since gates and verification steps take time. It adds complexity: once a workflow passes about fifteen branches, reviewers stop reading it carefully, and splitting it into sub-workflows with their own failure contracts becomes the better design. It also adds up-front cost that a program with three simple workflows may never recover. At that scale, a shared checklist and a disciplined review can substitute. At thirty workflows, human memory stops working as an index, and the machinery pays for itself.
A harder question is arriving behind these. Agents are beginning to propose edits to their own branches and retry settings based on what they observe, which is attractive, since the failure paths are where the learning is. But a workflow that tunes its own thresholds produces a version history nobody authored. When the plan changes itself between runs, which version is the one an accountable person approved, and how would anyone know?
What does an agent orchestrating workflows do?
It coordinates the steps, tools, and approvals that make up a multi-step process, deciding what runs next based on the result of each step. That includes routing work down different branches, handling failures, retrying transient faults, and escalating to people when needed. In production, most of its logic concerns non-success paths rather than the ideal sequence.
How does conditional branching work in an AI agent workflow?
Conditional branching routes work down different paths based on signals such as output confidence, context completeness, value at stake, and reversibility of the next action. A sound design lets the model propose a branch while a deterministic gate validates it against policy before it executes. Every conditional also needs an explicit default branch so unmatched inputs never stall silently.
What is the difference between error handling and retry logic?
Error handling classifies a failure and chooses the right response, which might be retry, reroute, verify, or escalate. Retry logic is one of those responses, and it suits only transient faults such as timeouts and rate limits. Applying retries to deterministic or semantic faults raises cost and delays detection without fixing the cause.
How many times should an agent workflow retry a failed step?
Three attempts per step is a reasonable default, using exponential backoff with jitter. A retry budget at the dependency level, often about 10 percent of calls over a rolling window, prevents retries from overwhelming a degraded service. After the limit, the workflow should take a predefined fallback branch.
Why does workflow versioning matter for long-running workflows?
Long-running workflows with human approvals can stay in flight for days or weeks, so their definition often changes while instances are still active. Versioning lets each run finish under the rules it started with or migrate safely after a compatibility check. It also lets decisions be explained later using the exact thresholds and prompts in force at the time.
How can you test whether an orchestration platform handles failure well?
Inject faults during the evaluation: force a timeout, return malformed tool output, release a new version mid-run, and stop a run after an external write. Watch whether failures are classified, retries stay within a budget, and the resulting state is visible to an operator. A platform that only demonstrates successful runs has not shown how it behaves under pressure.