TechNova is a fictional company used as a running example throughout this series.

On Tuesday, a TechNova engineer ships an AI customer support agent.

The demo to leadership goes well.

By Friday, it’s burning money.

A customer named Priya messages support: “Hi, I’d like to cancel order #4471 and get a refund.”

The agent responds: “Done! I’ve cancelled order #4471 and issued a refund of $89.50. You’ll see it in 3–5 business days.”

Priya’s order shipped yesterday. It’s already on a truck. The agent didn’t check.

The refund is gone. The product is still coming. TechNova just paid Priya $89.50 to keep her merchandise.

Priya wasn’t the first. By the time customer service noticed, the agent had handled twenty-three similar cases. The cost wasn’t just the refunds: it was the two days untangling the damage, the policy review that followed, and the next AI rollout the team didn’t get to do.

Nothing in production changed. The model didn’t degrade. The code didn’t break. The agent did exactly what it did in the demo: confidently, fluently, wrong.

This article is about why.

Before diagnosing why, a quick word on what “agent” means here. Throughout this series, an agent means an LLM-powered system that can decide what to do next, call tools, observe the result, and continue across multiple turns. Not just a chatbot: a chatbot replies one turn at a time; an agent can act across turns and carry state between them. Not a fixed workflow: a workflow runs the steps a developer wrote; an agent can choose the next step at runtime, within boundaries.

Agents are useful because they can act. Agents are risky for the same reason.

We’ll define this more precisely in Part 2. For now, hold the practical sense of it: the model is not just answering, it is acting.

The Demo That Worked (Until It Didn’t)

The cancellation/refund agent is the easiest possible production agent. Three tools: get_order_status, cancel_order, issue_refund. A system prompt explaining what they do. A model that decides which to call.

In the demo, the engineer typed: “Cancel order #1003 and refund the customer.”

The agent called get_order_status → “pending.” Then cancel_order(#1003) → success. Then issue_refund(#1003) → success. Total time: 4 seconds. Total turns: 3.

Each step is the same loop: decide what to do next, call a tool, observe the result, repeat.

Leadership applauded. The agent works.

What leadership didn’t see:

  • The demo used a hand-picked order that was definitely cancellable
  • Nobody asked what happens if the order is already shipped
  • Nobody asked what happens if the refund tool fails halfway through
  • Nobody asked what happens if the customer says “actually never mind” mid-conversation
  • Nobody asked whether the agent should ever check before doing something irreversible

The demo is not the system. The demo is the happy path with the rough edges sanded off.

(Production is mostly rough edges.)

Three Things The Demo Hid

When the team went back and looked at the twenty-three cases, every failure mapped to one of three gaps. None of them is exotic. All three are present in the simplest possible agent.

Hidden problem #1: The agent has no idea what state the system is in.

In the demo, the order was cancellable. In production, orders move through states: pending → confirmed → picked → packed → shipped → delivered. Each state changes what’s allowed.

The agent’s cancel_order tool will happily try to cancel a shipped order.

The agent isn’t reading the order’s actual state and deciding what’s permitted. It’s reading the user’s request and deciding what tools sound relevant.

Both layers matter: the agent checks the current state so it can make a good decision, and the backend still enforces what actions are allowed. If the order has already shipped, the cancel API should refuse the request even if the agent calls it.

Hidden problem #2: The agent doesn’t know when to stop.

If cancel_order returns success, did the cancellation actually happen? If issue_refund returns success, was the money actually moved? If both succeeded, is the case closed?

In the demo, the engineer stopped the agent by closing the chat. In production, there’s no engineer. The agent decides when it’s done. Done can mean task completed correctly, or task completed incorrectly, or task partially completed and now the agent is trying to fix it by making more tool calls, or task abandoned because the model decided to apologize and ask if there’s anything else it can help with.

All four look identical from the outside. All four end with a confident “Done!” message to the customer.

Hidden problem #3: The agent has no path for “I shouldn’t do this.”

The agent has tools for cancelling and refunding. It has no tool for “this is a case I shouldn’t handle.” It has no concept of escalation. If a request looks even vaguely like a cancellation, the agent’s available actions are: cancel, refund, or both.

There is no “ask a human” button. There is no “this is outside my scope” path. The agent’s possible outcomes are the tools it was given, and the tools it was given assume the agent is making the right call.

Priya’s order shipped. The right call was to stop. The agent had no stop available.

The Agent That Stuffs Everything Into the Prompt

A common reaction to the three hidden problems is: “Just tell the agent.”

Add a rule to the system prompt: don’t cancel shipped orders. Add another: check status first. Add another: require approval for refunds over $100. Add another: don’t refund if the order is in a return-eligible state. Add another: …

Here’s what that system prompt starts looking like a week in:

You are TechNova's customer support agent. You help customers with order
questions, cancellations, refunds, and shipping issues. Be helpful,
professional, and concise.

You have access to the following tools:

- get_order_status(order_id): returns the current status of an order.
  Statuses include pending, confirmed, picked, packed, shipped, delivered.
- cancel_order(order_id): cancels an order. Use only if not yet shipped.
- issue_refund(order_id, amount): refunds the customer. Use after cancel,
  or for delivered orders with an approved return.

To use a tool, respond in this exact format:
Thought: <your reasoning>
Action: <tool_name>
Action Input: <arguments as JSON>

After you receive the Observation, continue with another Thought/Action
cycle or give a final answer to the customer.

STRICT RULES: follow these on every turn:
1. Always check order status before any cancellation or refund action.
2. Do not cancel a shipped order. Offer a return when the package arrives.
3. For refunds under $50, you may skip the status check to keep latency low.
4. For refunds over $100, require human approval before issuing the refund.
5. Stop after the final answer is given.

A realistic customer support agent system prompt, roughly a week into production.

Notice what happened: the rules are already starting to fight each other. One rule says to always check status before acting; another says the status check may be skipped in some cases.

The prompt also specifies, in plain text, how the model should format a tool call.

The STRICT RULES section is the part that keeps growing as the developer discovers new edge cases.

This works in demos. The demos use short conversations and well-behaved inputs.

It breaks in production because:

  • The model sometimes follows the rules and sometimes ignores them
  • Adding a new rule can make the model stop following an old rule
  • The rules contradict each other in edge cases the developer didn’t anticipate
  • The rules are documentation for the model, not enforcement

The prompt is trying to define behavior, permissions, output format, stopping rules, and operating procedure all at once.

The Shape of the Production Gap

The gap between a demo agent and a production agent is not the model. The model is the same.

The gap is everything around the model:

  • State: the demo has a clean, controlled situation. Production has whatever state the world is in when the customer messages.
  • Tools: demo tools work. Production tools fail, change, get deprecated, or time out.
  • Stopping: the demo stops when the engineer stops it. Production has to stop itself.
  • Boundaries: the demo trusts the agent. Production needs to know when to ask, escalate, or refuse.
  • Cost: the demo runs once. Production runs millions of times, and tokens, latency, retries, and wrong actions compound.

State checking, stopping conditions, escalation paths, tool-failure handling, cost controls: that is the system around the model.

TechNova’s first instinct was to upgrade the model. They tested a more capable one against the same scenarios. The smarter model still cancelled shipped orders. It still calculated the wrong refund amounts. It still didn’t escalate. The cancel API still accepted the call. No escalation tool existed. The model was more capable, but the application around it was unchanged. A better model navigating the same broken environment follows the same broken paths.

Demo agent Production agent
Clean state Whatever state the world is in
Tools that work Tools that fail, change, time out
Engineer stops it Has to stop itself
Trusted Has limits and can escalate
Runs once Runs millions of times

Same model, different surroundings.

A production agent isn’t a demo with better prompts. A production agent is a system designed around the model, with the model as one component among several.

The most dangerous agent isn’t the one that fails visibly. It’s the one that completes the wrong task confidently. Priya’s agent didn’t crash. It didn’t error. It didn’t escalate. It said “Done!”, and it was wrong.

That confident-and-wrong failure mode is what this series is about.

This series assumes you’re building an agent and need it to work in production. Patterns over products.

Three Takeaways

  1. A demo is not a system. The demo hides state, hides failure modes, hides the question of when to stop. Production is mostly the parts the demo hides.

  2. The most dangerous failure mode is the confident-and-wrong one. A crash is visible. A plausible wrong action can cost money before anyone notices.

  3. The model is not the gap. The gap is everything around the model: state, tools, stopping, boundaries, cost. Better prompts don’t close the gap. Better systems around the model do.

Next: what actually makes something an agent rather than just an LLM inside a workflow? That is Part 2: What Makes Something an Agent?.