Rarely the model. What breaks first is usually the surrounding system: a tool contract that assumed clean inputs, permissions borrowed from a shared account, retries that quietly duplicated work, and an evidence trail too thin to reconstruct what happened. The agent then gets blamed for a failure it merely exposed.
The tool contract meets real data
In a pilot the inputs are chosen. In production they are whatever the estate contains: a record with a null where the contract expected a value, a description field holding pasted email, a currency code that does not appear in any list.
An agent handles these worse than a form does, because a form refuses and an agent improvises. It will pick the closest plausible option and proceed, and the resulting proposal looks as confident as a correct one. The fix is on the contract rather than the prompt: validate inputs at the boundary, return structured refusals the agent can reason about, and make missing data an explicit outcome rather than something to interpret.
Permissions and retries
Two failures dominate the early weeks, and both are architectural.
- The borrowed identity. A shared service account with broad rights makes every exploratory read succeed. The failure surfaces the first time an agent assembles context for one customer using data belonging to another, and it surfaces as a disclosure rather than an error. Enforcement has to sit underneath the agent, which is the subject of stopping an agent reaching data it should not see.
- The silent retry. Orchestrators retry on timeout. If the write underneath is not idempotent, the retry creates a second record, and because the first call timed out nobody sees the duplicate until reconciliation. A client-supplied key on the write prevents this and costs almost nothing to add.
Evidence, and the operating gap
The third failure is quieter and worse. Something goes wrong, someone asks why the agent did it, and the answer cannot be assembled. The logs show that a call happened but not what evidence the agent relied on, which version of the instructions it was running, or who approved the action.
That question arrives more often than teams expect, and it arrives from auditors and customers rather than engineers. Recording actor, reason, evidence, artifact, and cost per run is much easier to do at the start than to backfill. What an audit trail needs to contain sets out the shape.
The last thing to break is ownership. A pilot has an engineer watching it; production does not, unless someone was named. When the model provider changes a version, or a downstream schema shifts, the agent degrades rather than stopping, and degradation has no alert unless one was built. Who operates an agent after it is built is a question worth answering before launch rather than after the first incident.
Frequently asked questions
Does the model itself ever cause the first failure?
Sometimes, but less often than expected, and usually through change rather than capability. A provider updates a version and behaviour shifts, or a response format moves and parsing fails. Both are operational events with operational answers: pin versions, validate output against a schema, and treat a provider change as a release requiring the same checks as your own.
If the model is not the problem, why do these projects still stall?
Because the work that makes an agent survivable is ordinary systems engineering: interfaces, permissions, idempotency, logging, ownership. Teams staff the modelling and under-staff that, so the agent works in a demonstration and cannot be operated. The stall is an integration and governance gap wearing the costume of a model problem.
What single control prevents the most damage early on?
A human approval in front of every write, with a separate executor performing the approved action. It does not prevent poor proposals, but it stops a poor proposal from becoming a change to real data, and it produces the record you need to judge whether the agent is improving.
