This is the second article in a series about designing agents. Read The Purpose of an Agent Is What It Does before this one. It will provide more context.
An agent’s most consequential decision may be one nobody asked it to make.
Give it a question with two reasonable answers. It doesn’t ask which one you wanted. It picks one, and hands back an answer formatted so cleanly you’d never guess it was a guess.
There is no visual difference between the agent that knows and the agent that is improvising. No hesitation. No change of font. No little light that comes on to tell you that the system has crossed from knowledge into invention.
I’ve spent much of this year designing agents, workflows, skills and systems. I’ve become reasonably good at describing what they should do when everything is clear.
I’m less sure I’ve been designing what happens when things go sideways.
My Substack - Some Assembly Required - has always been where I work things out in public. It’s where the ideas are longest, sharpest, and least edited for palatability. If you want to stay connected to where the thinking is going, not just the occasional post that survives the algorithm, please subscribe.
The first article in this series was about designing an agent around the job it is supposed to do, rather than rebuilding the workflow that came before it.
This one starts where that design usually runs out. What should the agent do when the source is thin? When two sources disagree? When the instruction is ambiguous? When the system can’t tell whether it has completed the task? What should the agent do when the answer it wants to give is plausible, but not necessarily true?
If nobody answers those questions, the agent will.
The Agent Will Decide for You
The designs I see, and the ones I’ve written, often carry the same unexamined assumption: that failure is something that happens to the system, rather than something the system does. A bug. An outlier. An edge case. Something to patch after the fact and move on from.
That language is convenient because it makes failure sound external and unusual. It lets us design the happy path and treat everything else as a rare event.
But an agent doesn’t experience failure as an exception. It reaches a moment where it has to choose. Continue? Retry? Search somewhere else? Ask a question? Stop? Hand the decision to a person?
Or fill the gap with whatever answer sounds most confident?
Nobody writes a ticket for that moment. Nobody sits in a design review and says, “When the evidence is incomplete, please quietly choose one of two reasonable interpretations and present it as fact.” But if the system has been built to always produce an output, then an output is what it produces.
Every time.
It keeps going. It fills the gap. It dresses the guess in the same tone it uses for everything else. That isn’t simply a technical limitation. It is a design decision made by default instead of on purpose.
Fail Safely, Not Forward
A fallback is often treated as what happens when the system can’t do the real thing. I’ve started to think that’s backward. A fallback isn’t a consolation prize. It is part of the real thing—the part of the design that decides what “good” looks like once the primary path is gone.
Sometimes the right response is a retry, because the failure was noise rather than signal. Sometimes it is a different source, because the first source was thin and a second opinion changes the picture. Sometimes it is a request for more information, because the system has reached a genuine ambiguity rather than a temporary problem.
Sometimes the right response is to stop and explain the problem in plain language to a person who can decide what to do next.
That third option tends to get built last, if it gets built at all. It admits that the system has a ceiling. Teams will happily engineer three layers of retry logic and never build the one fallback that says:
I don’t have this, and pretending I do will cost you more than waiting would.
Escalation is what happens when the fallback itself needs backup. Who does the agent escalate to? When? With what information? What has it already tried? What exactly does the person need to decide?
Those questions are less glamorous than adding another capability. They are also where the system’s actual behavior gets designed.
Take one situation. An agent produces a client-facing campaign report, pulling from the ad platform’s own reporting and an internal attribution model. On a good day, the two agree closely enough to round together. On a bad day they don’t, and the report still needs to go out.
The instruction set that handles that moment might read like this:
If the platform and the attribution model disagree by more than fifteen percent, don't average them, and don't default to the platform's number because it's the one the client expects to see. State both figures and flag the disagreement in the report.If the brief doesn't specify which metric the client actually cares about — leads, cost per lead, something further down the funnel — do not infer it from last month's report. Ask.If the recommended action is to shift live budget past a set threshold, or pause a running campaign, do not execute it. Draft the recommendation, explain the reasoning, and route it to a named approver.If the same disagreement between the two sources shows up three times in thirty days, escalate the pattern, not the instance. Someone needs to look at why the sources keep disagreeing, not approve the fourth report.Four sentences. None of them are hard to write. All of them are easy to skip. The system that includes them isn’t smarter. It just knows what it doesn’t know, and says so before it costs anyone anything.
Set the threshold too high, and the agent bluffs its way past problems that needed a person, because asking for help was never a live option. Set it too low, and you’ve built an expensive tap on the shoulder. The humans on the other end start ignoring it within a week, gutting the fallback underneath it.
A system that escalates everything is not cautious. It is useless. A system that escalates nothing is not autonomous. It is making decisions it was never given permission to make.
The Honest Pause
Knowing its limits is a feature.
An agent that says, “I’m not sure. Here’s what I found,” is more useful, more often, than one that always has an answer. A confident wrong action costs more than an honest pause. It can also take a decision away from the human without announcing it.
Most systems are built with the opposite incentive. A system that always answers demos better than one that sometimes doesn’t. Certainty sells. Certainty also eventually breaks something you can’t easily un-break.
“I’m not sure” on a low-stakes draft can be a useful signal. “I’m not sure” on every output, regardless of what is at risk, is a system that never learned to have an opinion. A human will eventually stop trusting it for the opposite reason they stopped trusting the overconfident one.
The real question was never confidence versus uncertainty. It is calibration. Does the honesty match the stakes, or has it been stapled to everything as a blanket disclaimer so nobody can be blamed for whatever comes out from under it?
Calibration isn’t only about what the agent says. It’s about what it’s allowed to do next.
A draft email with the wrong tone is recoverable. A recommendation built on a weak source may need another pass. A transaction executed on a bad assumption is a different category of event. The fallback has to know the difference.
Safety is not a setting you dial in once. It is a judgment about reversibility, made fresh for whatever the agent is about to do.
Make the Failure Legible
Another requirement makes the other responses useful. You need to see why the system failed. A system that fails loudly, with a clear trail, is a system you can improve.
This source conflicted with that one. The evidence did not support the conclusion. The instruction could be interpreted in two ways. The tool returned incomplete information.
This was the point where the system stopped knowing and started guessing.
A system that fails silently, producing something wrong but fluent, with no seam showing where the certainty ran out, is a system you only discover has failed after it has already cost you something.
Legible failure is unglamorous work. Failure doesn’t show up in the demo. A demo is, by definition, the happy path performed for an audience.
It shows up three days later, in the ten minutes it takes—or doesn’t take—to figure out why the agent told a client something that wasn’t true. It shows up in whether anyone can reconstruct the decision. It shows up in whether the system can be improved, or whether everyone simply changes the prompt and hopes.
The Postmortem Is the System
Zoom out one more level and POSIWID, the idea that a system’s purpose is what it does, stops being about the agent. It starts being about the people around it.
Picture the room after a failure has been flagged. In one version of that room, someone patches the prompt, closes the ticket, and moves on by Friday. In another, someone asks what the failure revealed about everything upstream.
Was the source selection wrong? Was the escalation rule missing? Was the system rewarded for producing an answer when it should have asked a question? Did the human checkpoint exist only because a person used to occupy that box? Did the architecture make the wrong behavior more likely?
Both rooms think they handled the failure. Only one of them has learned from it.
A team that treats every failure as a one-off has built an organization whose real function is not looking too closely at its own design. A team that treats failure as data has built something that can become harder to break over time.
That is the only kind of safe worth having. The thing you run after the agent gets something wrong isn’t merely a bug report. It is the only honest performance review your system will ever get—the moment it stopped performing and told you what was actually happening.
Design the Break
My earlier article argued that we should design agents backward from the consequence we want. This one adds the second half — and it’s actually two halves, not one.
There’s what the agent does when it runs out of certainty: the source is thin, the brief is ambiguous, the answer it could produce isn’t necessarily true. And there’s what it does when it has plenty of certainty and still shouldn’t act: the transaction it could execute but that needs a name attached to it first, the campaign it could pause but that isn’t its call to make alone. Not knowing what to do, and knowing exactly what to do but being required to stop anyway, are two different failures. Most systems are only designed for the first one.
What does it do when it reaches either edge? What verb does it use? Does it ask? Does it retry? Does it search? Does it stop? Does it escalate?
Or does it guess?
That last verb is often hiding inside an otherwise impressive system. It is not written anywhere. It does not appear on the architecture diagram. Nobody names the Guessing Agent. But if the system has no designed response to ambiguity — and no designed restraint where restraint was actually the job — guessing, or overreaching, is what it will do by default.
You can write a beautiful system prompt, chart a clean diagram, demo the agent flawlessly in front of the CMOs, and still have built a system whose real purpose only becomes visible the first time it hits something it wasn’t ready for — or something it was ready for and shouldn’t have touched.
Nobody Wants to Hear, “I Am Not Sure Yet.”
Design the agent all you want. It’s the part you didn’t design — the fallback, the escalation, the honest pause, the moment everything forks — that will end up doing the talking.
That’s true. It’s also not the whole truth.
Every gate in that instruction set requires the agent to say, out loud, in front of someone: I don’t know. Flag the discrepancy. Ask instead of guessing. Route it to a person before it goes any further. That only survives inside a real organization if the humans in the room can say the same thing without it costing them something.
Most can’t. Sit through enough status meetings, and you’ll notice which answer gets rewarded — the confident number, even the wrong one, over the honest “I’m not sure yet.” Nobody designs that incentive on purpose either. It’s just what earns a nod in the room, and what earns a follow-up question nobody needs.
The agent is listening, in the only way an agent can listen: through the humans who keep adjusting its instructions. The first time “I’m not sure” produces an uncomfortable silence instead of an answer someone can act on, somebody edits the prompt. Not out of malice. Out of the same instinct that wants the meeting to end on time. A few edits later, the hedge is gone, and nobody remembers deciding to remove it.
You can design a system for what it does. But you must also design for what it doesn’t, or shouldn’t do. This will be the hardest part of agent design, because we operate in a culture that requires answers, not ambiguity or indecision. The agent doesn’t succeed or fail on how smart it is. It succeeds or fails on how much judgment actually made it into the instructions.
Judgment has to come from somewhere. The next article in this series is about the human as the loop — the experience it actually takes to know what good looks like before you can ever write it into an instruction set.
