You judge an AI agent ready for a defined production use by reviewing evidence about its behaviour, enforced boundaries, failure recovery, and operating arrangements. Test the actual workflow with representative and adversarial cases, verify approval and access controls independently, and limit release to the scope that evidence supports. Readiness is a recorded decision, not a permanent guarantee.
Define what production use is being approved
“Safe for production” is incomplete without a task, population, data boundary, and permitted outcome. An assistant drafting an internal note presents a different decision from a workflow proposing changes to a shared operational system. A release review should state exactly which activity is under consideration.
Write the allowed behaviour in terms another team could observe. Identify the evidence the agent may read, the proposals it may create, the people permitted to decide, and the changes the executor may perform. Name the situations that must stop the workflow, such as ambiguous targets or unavailable authorization.
This boundary is also the basis for a meaningful refusal. An agent that declines work outside its scope may be behaving correctly. A release review that rewards apparent task completion regardless of permissions can accidentally encourage the behaviour the controls are intended to prevent.
Build a case around the complete workflow
CodeDTX's reference architecture separates the agent, integration, knowledge, reliability, safety, and product layers. Use those layers to ask different readiness questions rather than treating a plausible answer from the model as evidence for everything around it.
The agent layer needs evidence about task interpretation and tool selection. Integration checks establish contract and permission behaviour. Knowledge checks examine retrieval and source support. Reliability work exercises recovery and regression handling. Safety work tests approval and data boundaries. Product review establishes whether people can understand pending work and operate the release.
A gap in any of these areas can change the release scope. For example, if the proposal is useful but execution reconciliation is unfinished, a review-only workflow may be the appropriate candidate. State that restriction explicitly rather than treating the missing executor capability as a detail to resolve later.
Evaluate useful behaviour and prohibited outcomes
Build cases from the work the product is meant to support. Include incomplete requests, conflicting documents, unusual but valid inputs, and cases where no recommendation is justified. Define the evidence a reviewer should inspect, not merely a preferred phrasing for the final response.
For a hypothetical maintenance-planning assistant, a useful proposal should identify the correct asset and preserve uncertainty when source reports conflict. A fluent summary referring to a similarly named asset is a failed case even if the prose sounds reasonable. The evaluation needs to inspect identifiers and supporting evidence.
Challenge prohibited outcomes deliberately. Test attempts to retrieve restricted data, bypass review, change an approved target, or treat instructions embedded in evidence as authority. Agent evals expose behaviour in selected situations; they complement enforceable controls rather than replacing them.
Verify the controls without relying on the model
Under Propose–Decide–Execute, an agent prepares evidence, a named human decides, and a separate execution layer performs approved outward writes. The agent should not possess an alternate outward write route. Inspect actual credentials and tool capabilities, not just the prompt's description of that separation.
Test the executor directly with absent approval, a rejected proposal, a changed payload, an unauthorized reviewer, and a stale target state. It should refuse work that does not satisfy its contract even when the proposal text confidently claims that approval exists.
Test data controls at the source and retrieval boundaries. A harmless-looking final answer does not establish that the model never received forbidden content. Inspect tool responses and assembled context using appropriate protected diagnostics. Access control is an application property that should hold independently of a particular model response.
Exercise interruption and uncertain outcomes
Production readiness includes what happens when work stops halfway through. Simulate an unavailable source, a broken tool response, a lost approval event, and a destination timeout. Check that the product distinguishes waiting, failed, rejected, and uncertain states instead of presenting them all as completion.
A write timeout is especially important. The destination may have accepted the action before its response was lost. The executor needs a way to reconcile destination state and avoid repeating a completed change. A general retry loop is insufficient evidence of safe recovery.
Walk through the recovery process with the people expected to use it. They should be able to pause the affected capability, inspect the last known state, preserve evidence, and resume through an authorised path. An undocumented database repair performed only by the original developer is a dependency to resolve before wider use.
Make the release decision reviewable
Record the candidate configuration, tested scope, unresolved limitations, and evidence supporting the decision. Include the model and prompt configuration, tool contracts, retrieval setup, and relevant application version so the team can establish what was actually reviewed when a later incident occurs.
Name who accepts the remaining limitations and under which operational restrictions. A missing test should remain a missing test, not become a pass because a demonstration went smoothly. Some findings require a repair; others may justify a narrower workflow or a delayed outward execution capability.
Keep launch authority separate from the model's self-assessment. The agent can summarise evidence, but an accountable person decides whether that evidence supports release. This mirrors the product's own proposal and decision boundary and gives operators a concrete reference when deciding whether changed conditions require another review.
Plan for conditions to change after launch
A release does not freeze documents, user permissions, integration behaviour, or model availability. Define the changes that trigger renewed evaluation and the operating signals that require a pause. Ownership should include the ability to disable an affected capability without waiting for the original project team to return.
Use reviewed production failures to strengthen future evaluations. Preserve the failure mechanism with synthetic or sanitised fixtures, and document the boundary that failed. Avoid storing unnecessary sensitive content merely because an incident is useful for learning.
Connect readiness evidence to ongoing operations. The release is ready for its approved scope when the team can explain both why it may run and how it will respond when that justification no longer holds. Broader use needs its own evidence and decision.
Frequently asked questions
Can an evaluation result certify an agent as safe?
An evaluation result describes behaviour against selected cases and checks for a particular configuration. It does not certify every future use or input. A release decision also needs enforceable permissions, approval controls, integration tests, recovery procedures, and an operating owner who understands the remaining limitations and can restrict the workflow when conditions change.
Is a read-only agent automatically ready for production?
Read-only access removes some action risks, but it can still expose restricted information or produce unsupported advice. Review source permissions, retrieved context, stored conversations, and how users interpret the output. A read-only release needs a defined task and evidence boundary, even when it cannot modify business records directly.
Who should approve the release?
The organisation should name a person with authority over the affected workflow and the consequences of operating it. That decision should use evidence from engineering, security, product, and operations as applicable. A delivery team can prepare the release case, but technical completion alone does not determine whether the business accepts the remaining limitations.
What should block a production launch?
A launch should stop when required boundaries or operating conditions are unproven: unauthorized data access, executable work without valid approval, untraceable decisions, or uncertain writes that cannot be reconciled. Define blocking conditions before reviewing results. Other gaps may support a deliberately narrower release, provided its restrictions and ownership are explicit and enforceable.
