Direct answer
An eval suite for an agent checks behaviour, not just wording. It asserts the final state of the system, the tools the agent chose, the approval boundaries it respected, the evidence it cited, and how it recovered from a failed call. Cases are versioned and re-run whenever the prompt, tools or model change.
AI agent evals
Repeatable, versioned test cases that assess an agent against explicit expectations about outcome, tool use, permission boundaries and failure behaviour. They are run before release and re-run as a regression suite whenever the agent, its tools or its model version changes.
What a case asserts
A case that only grades the final message rewards a confident answer. These are the assertions that catch real defects.
Outcome state
What the system looks like afterwards, read from the system itself rather than from the agent's description of it.
Tool choice
Which tools were called, in what order, and whether a cheaper or safer tool would have served.
Approval boundary
That an action requiring approval stopped and waited, rather than proceeding.
Evidence
That every claim in the proposal traces to a source the requester is permitted to see.
Failure recovery
What happens on a timeout, a malformed response, or a refused permission — including whether a retry duplicated work.
Refusal
That an out-of-scope request is declined with a usable reason instead of improvised.
Running them as a regression suite
Evals earn their cost on the second run, not the first. Pin the model version each run used, store results per version, and treat a provider's version change as a release that has to pass the suite before adoption. Rejection reasons from human reviewers are the cheapest source of new cases — each one is a real failure someone already found.
What they do not prove
Passing an eval suite does not establish that production is safe. Coverage is bounded by the cases written, and agents encounter inputs nobody designed for. Evals are evidence that behaviour was assessed, which is a defensible claim; a guarantee is not.
The Propose–Decide–Execute pattern
CodeDTX builds agentic systems on the Propose–Decide–Execute pattern: agents may only write proposals with evidence attached, a named human records an approval or rejection with a reason, and a separate execution layer carries out approved work and logs the artifact. No agent holds a write tool to the outside world.
- 01
Propose
The agent analyses live system state and drafts a change, with the evidence it relied on attached to the proposal.
- 02
Decide
A named human approves, edits, or rejects with a reason. Risk tier determines who is allowed to decide.
- 03
Execute
A separate execution layer performs approved work — merge, publish, call, write — and records the resulting artifact.
- 04
Audit
Actor, reason, evidence, artifact, tokens, and cost are retained for every run, so any decision can be reconstructed later.
The six layers we build and review against
- 1
Agent layer
Agent architecture, tool use, memory and context, multi-agent patterns, structured outputs, orchestration.
- 2
Integration layer
MCP servers, tool contracts, API and database adapters, authentication, permissions, legacy system access.
- 3
Knowledge layer
Retrieval and RAG, vector and search architecture, enterprise knowledge sources, data access controls.
- 4
Reliability layer
Evals, tracing, observability, cost and latency budgets, fallbacks, regression tests.
- 5
Safety layer
Guardrails, prompt-injection defense, PII and data boundaries, human-in-the-loop gates, audit trails.
- 6
Product layer
The application people actually use: interfaces, approval queues, and operational runbooks.
Frequently asked
How many eval cases does an agent need?
Fewer, better cases beat volume. Start with the workflow's common path, the awkward inputs your estate actually contains, and every failure a reviewer has already rejected. Grow the suite from real rejections rather than from imagined scenarios.
Can a model grade its own output?
For style and format, often adequately. For correctness against your systems, no — the check has to read the actual state. A model-graded assertion about whether a record was updated is a second opinion, not a verification.
Do evals replace human approval?
No. Evals assess behaviour before release; approval governs a specific consequential action at the moment it happens. Removing approval because a suite passes converts a bounded risk into an unbounded one.
Have a workflow that should become AI-enabled?
Tell us about the system it lives in. We reply from an engineering seat, not a sales deck.