To validate AI agent output before it reaches your systems, treat every model response as untrusted input. Enforce a typed schema at the model boundary, then check values against your own reference data and business rules, and only then let the write, API call or message go ahead. Decide in advance whether a failed check retries, falls back or escalates.
Why agent output needs its own checks
A chat answer is read by a person, who notices when something looks wrong. An agent's output is read by software. It becomes a row in a database, a payload sent to a partner API, a refund request or an email to a customer. Nothing downstream was built to question it.
Models are good at producing output that looks right. The failures that cause trouble are small: a missing field, a date in the wrong format, a list where a single value was expected, a customer ID that does not exist, or an amount that is valid as a number but far outside what the business allows. Each one passes a casual glance and breaks something later, often far from the agent that caused it.
The fix is the same discipline you already apply to user input and third-party data. Validate at the boundary, reject what does not fit, and never assume the source behaves.
Check structure at the model boundary
Ask for a schema, not free text
Most model providers and agent frameworks now support structured outputs, where the response must follow a JSON schema you supply. Libraries such as Pydantic AI and Instructor let you describe that schema as typed code. Use them for anything a system will consume. Free text that you parse afterwards is where most format errors begin.
Still validate what comes back
Constrained output reduces format errors, but it does not remove the need to check. Parse the response against the same schema in your own code, with required fields, types, allowed values and length limits. If parsing fails, send the error back to the model once or twice with a clear message, then stop retrying and move to the failure path.
Check values against what you know
A response can match the schema perfectly and still be wrong. The second layer compares values with your own records:
- Identifiers exist. Every customer, order or product ID the agent names should resolve in your system, and belong to the account the agent is working for.
- Values are in range. Amounts, quantities and dates should sit inside limits the business sets, not just inside what the data type allows.
- References match the source. If the agent quotes a policy, a price or a document, check that it matches the version the agent was given.
These checks are ordinary code. They are cheap, predictable and easy to test, which is why they belong outside the model.
Check business rules before the action
The final layer asks whether the action is allowed, not only whether the data is valid. A refund may be correctly formatted and still exceed the agent's approval limit. An email may be well written and still go to a customer who has opted out. Put these rules in the same place you enforce permissions, so the agent cannot talk its way past them. Our guide on stopping an agent taking a wrong action covers where those gates should sit.
For writes that are hard to reverse, add a confirmation step or route the action to a person. The post on using confidence scores to send work to human review explains how to decide which ones.
Decide what happens when a check fails
Validation is only useful if the failure path is clear. For each check, choose one of these responses in advance:
- Retry with feedback for format errors the model can fix on its own.
- Fall back to a safe default or a simpler path when the task can still complete without the agent's output.
- Escalate to a person when the output is plausible but breaks a rule, and keep the task open rather than half done.
Log every failure with the input, the output and the check that rejected it. That record shows which prompts, tools or models produce bad output most often, and it gives your evaluation suite real cases to test against. Rising failure rates are also an early sign of a model change or a drifting prompt, so watch them alongside your other production monitoring.
Where CodeDTX fits
We build validation layers into the agents we deliver, so model output is checked against your schemas, records and rules before it touches a live system. If you are connecting an agent to systems of record, our enterprise AI integration work starts from that boundary. Talk to us about the systems your agent will write to.
Frequently asked questions
Do structured outputs remove the need for validation?
No. Structured outputs make the model follow a schema, which cuts format errors sharply, but they cannot tell whether a customer ID exists, an amount is allowed or an action breaks a business rule. Keep parsing the response in your own code, then check values against your records and rules. Treat structured outputs as the first layer of validation, not the whole of it.
Should we use another model to check the agent's output?
A second model can help with checks that need judgement, such as tone, whether a summary matches its source, or whether a reply contains sensitive data. It should not replace deterministic checks. Schema, range and permission checks are cheaper, faster and predictable when written as ordinary code. Use a model checker as an extra layer for the questions code cannot answer, and log its decisions too.
How many times should an agent retry after a failed check?
Keep it low and fixed. One or two retries with the validation error included in the prompt fixes most format problems. Beyond that, repeated attempts add cost and delay without much benefit, and they can hide a real fault in the prompt or the model. After the limit, move to the fallback or escalation path you chose in advance, and log the failure for review.
Where should validation code live?
Put it between the agent and the systems it writes to, in a layer the agent cannot bypass or switch off. That is often the tool or API wrapper the agent calls, or a gateway in front of your services. Keeping it there means the same checks apply whichever model or prompt is in use, and your engineering team can test and change the rules without retraining or re-prompting anything.
