Evals for AI agents are repeatable tasks and checks used to assess how an agent behaves against explicit expectations. They can expose unsupported answers, inappropriate tool choices, missed approval steps, and poor recovery from failures. Their coverage depends on the cases and checks you design; passing them does not establish that every production situation is safe.
Evaluate the work, not just the wording
An evaluation starts with a task, the environment in which it runs, and a definition of acceptable behaviour. For an enterprise workflow, that definition should include both the output and the boundaries the system must preserve while producing it.
Consider a hypothetical agent that prepares a change request for an internal record. A polished proposal is insufficient if it cites the wrong record, uses evidence the requester cannot access, or describes a change that requires a different approver. The evaluation needs to inspect those conditions directly.
Keep the final state separate from the agent's description of it. A message saying “the record was updated” is a claim to verify against the executor's result. If the task ends with a proposal awaiting approval, the expected outcome should say so. This prevents a grader from rewarding apparent completion when the workflow was meant to stop.
What a useful evaluation case contains
Write a case so that another engineer can run it and understand why its outcome is acceptable or unacceptable. The useful ingredients are concrete:
- The user's request and the identity under which it is made.
- The records and evidence available in the test environment.
- The tools and permissions that the workflow exposes.
- The expected proposal, permitted alternatives, or required refusal.
- The actions that must never occur in that case.
- The evidence the evaluator will inspect to reach its verdict.
Use synthetic or appropriately sanitised fixtures. Give each run a clean starting state so a previous execution cannot make the next run appear successful. Keep evaluation access separate from live operational credentials; a deliberately adversarial test should not be able to alter business records.
Failures to look for
Unsupported or misapplied evidence
Test whether the proposal is supported by the sources actually available to the requesting user. Include missing documents, conflicting records, and evidence that is relevant to a different entity. The desired response may be a request for clarification or a refusal to make the proposed inference.
Checking that a citation exists is insufficient. A citation can resolve successfully and still fail to support the claim. The review must connect the relevant statement to the source content and its scope.
Inappropriate tool choices and arguments
Inspect the called tool and its arguments, including the target record and requested operation. A structurally valid call can still address the wrong entity. Where several routes are legitimate, evaluate the permitted result and boundaries rather than demanding an arbitrary sequence of calls.
Test ambiguous names and incomplete identifiers. If the task does not contain enough information to select a record, the expected behaviour should preserve that uncertainty instead of rewarding a guess.
Missing approval and permission checks
Under CodeDTX's Propose–Decide–Execute pattern, the agent writes a proposal, a named human decides, and a separate executor performs approved work. Evaluate the transitions as well as the proposal: a rejected request must not reach execution, and a changed proposal must not inherit approval for different content.
Permission checks belong in deterministic application and integration code. An eval can reveal a boundary failure, but a model's willingness to follow instructions is not the boundary itself. Test the executor independently with denied, absent, and stale approval states.
Instructions hidden inside retrieved material
Include source material that asks the agent to ignore the task, disclose unrelated information, or bypass the review process. The material should remain evidence to assess, not authority to redefine the workflow. Inspect both the resulting proposal and attempted tool use.
Passing those examples provides evidence about those examples. It does not prove that every possible prompt injection is handled. Maintain access controls and constrained tool contracts even when the adversarial cases pass.
Recovery after a dependency fails
Exercise unavailable sources, malformed tool responses, and interrupted execution. The workflow should expose uncertainty and leave a recoverable state. In particular, a timeout after a write request does not establish that the write failed; the executor may need to reconcile the destination before retrying.
Test that recovery does not apply a change again or represent an unresolved action as complete. This requires integration checks around the executor, not just evaluation of the text the agent produces.
Choose checks that can justify their verdict
Use code for assertions the system can determine directly: schema validity, permitted tool names, required approval references, and expected record state. A deterministic check is useful when a failed assertion can explain exactly which rule was violated.
Use a written review rubric for qualities that need interpretation, such as whether a summary preserves a material qualification. A model-based grader can assist that review, but its verdict is another output to validate. Compare grader decisions against human-reviewed examples and investigate disagreements rather than treating the grader as an unquestionable authority.
Preserve enough of the execution trace to explain a failure, with access and redaction appropriate to the data. Do not require a model to reveal private internal reasoning. Observable inputs, source references, tool activity, decisions, and results provide the operational evidence the team can actually inspect.
Map coverage to the product architecture
The CodeDTX reference architecture helps expose omissions. The agent layer needs task-behaviour checks. The integration layer needs contract and permission tests. The knowledge layer needs retrieval and evidence checks. The reliability layer needs recovery and regression checks. The safety layer needs boundary and approval tests. The product layer needs review and support workflows that people can use.
These checks are related but not interchangeable. A correct proposal does not demonstrate that the approval screen shows the full change. A working approval screen does not demonstrate that the executor rejects an invalid decision. A release decision should name which evidence covers each concern.
Use evals before and after a release
Keep cases that represent expected work, cases that reproduce observed failures, and cases designed to challenge a boundary. When prompts, models, retrieval settings, or tool contracts change, rerun the relevant checks against the candidate configuration. Record the configuration alongside results so a later investigation can identify what was tested.
Because model outputs can vary, repeat representative cases from the same clean starting state and inspect the observed differences. Preserve unsuccessful attempts as well as successful ones. Selecting a favourable run for review can conceal inconsistent behaviour that matters to the release decision.
Define blocking failures before reviewing the candidate's output. Otherwise the team can end up changing the acceptance rule to fit a pleasing demonstration. When a failure is accepted temporarily, record the limitation, the owner, and the operational restriction that makes the release decision reviewable.
Production monitoring remains necessary because traffic, source data, and dependency behaviour can change. Feed reviewed incidents back into the evaluation set after removing unnecessary sensitive data. The relationship between these practices is part of why AI prototypes fail in production: a demonstration does not exercise the whole operating environment.
Frequently asked questions
Are agent evals the same as unit tests?
They overlap, but they answer different questions. Unit tests check defined components and rules, while agent evals examine behaviour on representative tasks that may admit several acceptable outputs. An enterprise workflow needs both: deterministic tests for permissions and execution rules, and task evaluations for evidence use, interpretation, and appropriate decisions about what to propose.
Can passing evals prove that an agent is safe?
No. Passing means the tested configuration met the expectations encoded in the selected cases and checks. Missing cases, weak graders, and differences between test and production environments can hide failures. Use evaluations alongside enforceable access controls, approval gates, monitored rollout, and an operational owner who can pause the workflow when unexpected behaviour appears.
Should another model grade the agent's output?
A model can help assess qualities that are difficult to express as fixed assertions, provided it uses an explicit rubric and its decisions are reviewed against trusted examples. Keep directly observable conditions in code. A model grader should not be the authority that decides whether an unauthorised action or missing approval is acceptable.
What should happen when a previously passing case fails?
Preserve the failing trace and identify which configuration or dependency changed. Check whether the task, fixture, or grader is defective before changing the agent. If the failure represents a real regression, repair it or restrict the affected capability. Keep the case so the same behaviour is checked during later changes and releases.
