Direct answer
Agent observability records each run as a trace: the tools called, the evidence retrieved, the decision reached, and the cost. On top of that sit the signals that matter operationally — refusal rate, approval rate, input freshness and cost per run — because agents degrade quietly rather than failing loudly.
AI agent observability
The instrumentation that makes an agent's behaviour visible: a trace per run covering tool calls, retrieved evidence and outcome, plus aggregate signals that reveal degradation before a person reports it.
The signals worth alerting on
Traces explain a single run. These aggregates tell you something changed.
Refusal rate
A rise usually means an interface or the input data changed underneath the agent.
Approval rate with sampling
Rate alone hides rubber-stamping, so read a sample of approved items in full.
Rejection reasons
Clustering on one theme names the next thing to fix, and each is a free eval case.
Input freshness
A stopped sync is silent; the agent keeps producing confident output from stale data.
Cost per run
A quiet climb usually means retries or growing context.
Version in force
Which model and prompt version each run used, so behaviour ties to a release.
Why degradation is the hard case
A crashed service pages someone. An agent whose proposals have become slightly worse produces no error at all, and the first report comes from a frustrated reviewer weeks later. That is why the aggregate signals matter more here than in conventional services, and why they need a named owner rather than a dashboard nobody opens.
The Propose–Decide–Execute pattern
CodeDTX builds agentic systems on the Propose–Decide–Execute pattern: agents may only write proposals with evidence attached, a named human records an approval or rejection with a reason, and a separate execution layer carries out approved work and logs the artifact. No agent holds a write tool to the outside world.
- 01
Propose
The agent analyses live system state and drafts a change, with the evidence it relied on attached to the proposal.
- 02
Decide
A named human approves, edits, or rejects with a reason. Risk tier determines who is allowed to decide.
- 03
Execute
A separate execution layer performs approved work — merge, publish, call, write — and records the resulting artifact.
- 04
Audit
Actor, reason, evidence, artifact, tokens, and cost are retained for every run, so any decision can be reconstructed later.
The six layers we build and review against
- 1
Agent layer
Agent architecture, tool use, memory and context, multi-agent patterns, structured outputs, orchestration.
- 2
Integration layer
MCP servers, tool contracts, API and database adapters, authentication, permissions, legacy system access.
- 3
Knowledge layer
Retrieval and RAG, vector and search architecture, enterprise knowledge sources, data access controls.
- 4
Reliability layer
Evals, tracing, observability, cost and latency budgets, fallbacks, regression tests.
- 5
Safety layer
Guardrails, prompt-injection defense, PII and data boundaries, human-in-the-loop gates, audit trails.
- 6
Product layer
The application people actually use: interfaces, approval queues, and operational runbooks.
Frequently asked
Is standard application tracing enough?
It captures latency and errors but not the agent-specific questions: which tools were chosen, what evidence was used, whether an approval was respected. Those need to be recorded deliberately as part of the run rather than inferred from spans.
What is the single most useful signal early on?
Refusal rate. It is cheap to compute, it moves before anything else, and a change in it almost always points at a real cause — a schema shift, an expired credential, or a change in the inputs.
How does this relate to the audit trail?
The audit trail answers why an action was authorised, for compliance and dispute. Observability answers whether the system is healthy now. They share fields and serve different readers, so it is usually worth writing once and reading twice.
Have a workflow that should become AI-enabled?
Tell us about the system it lives in. We reply from an engineering seat, not a sales deck.