Audit an AI agent by reconstructing the observable chain from the original request and available evidence to the proposal, human decision, executor activity, and resulting artifact. Check each transition against the permissions and policy active at the time. Explain the outcome using recorded facts and decision reasons, while distinguishing missing evidence from a justified conclusion.
Start with the question the audit must answer
An audit is a structured investigation into whether a particular run behaved as authorised and whether the evidence supports its outcome. It is different from browsing a chat transcript or collecting a large volume of technical logs. Begin with the disputed event and the question that needs resolution.
Suppose an internal record now names the wrong operational owner. The audit question is whether the agent proposed the wrong target, the reviewer approved an unsupported change, the executor applied different content, or another process changed the record afterward. Each possibility requires different evidence and a different remedy.
Define the investigation scope before exporting data. Identify the run, affected resource, relevant period, and authorised investigators. Preserve access restrictions on source material and avoid copying entire conversations or unrelated records into a shared incident document simply because they are nearby.
Establish what actually happened at the destination
Begin with the resulting artifact or business record. Check its history and current state using the destination's authoritative interface where available. The agent's statement that an action succeeded is not proof that the action occurred, and a timeout does not prove that it failed.
Connect the observed destination change to the executor's operation reference. Look for the acknowledged result, the target identity, and any reconciliation performed after uncertainty. If another user or process changed the record later, separate that event from the agent workflow under review.
Preserve the distinction between attempt and effect. An executor may have attempted a forbidden operation that the destination rejected. That is different from a successful unauthorized write, though the attempted action may still expose a control defect. State which conclusion the evidence supports.
Trace the executed content back to approval
Under CodeDTX's Propose–Decide–Execute pattern, the agent creates an evidence-backed proposal, a named human decides, and a separate executor performs approved outward writes. Audit the connection between these stages, not merely whether each stage has some log entry.
Compare the executed target and payload with the version presented for review. Confirm that the decision was an approval, that the person held the required authority, and that the executor checked the relevant conditions before acting. A generic approval message does not establish authorization for different content.
Look for changes while the proposal waited. A revised recipient, updated source, or altered record state can invalidate an earlier assumption. Determine whether the workflow detected that change and requested another decision, or executed work whose review context no longer applied.
Examine the evidence available when the proposal was made
Read the proposal against the source material the workflow actually retrieved. A currently correct document is not evidence that the same content was available during the run. Use retained source references, revisions, or appropriately protected snapshots to establish the decision context.
Check whether the evidence supports each material claim. A citation may exist but refer to another record or omit an important qualification. Identify uncertainty that the proposal disclosed and uncertainty that it concealed. This helps distinguish a retrieval failure from an interpretation failure or a product presentation problem.
Review the access context as well. A proposal can be factually accurate while relying on information the requester was not entitled to see. Inspect the originating principal, tenant, connector scope, and returned content. Correct wording does not excuse a broken data boundary.
Explain why without inventing hidden reasoning
Use observable decision evidence: the request, selected sources, tool calls, written proposal rationale, reviewer reason, and execution checks. These records can show what the system relied on and which controls allowed a transition. They do not provide direct access to a model's private internal reasoning.
Do not ask a later model to reconstruct a definitive explanation from incomplete logs and then treat its narrative as historical fact. A model may help organise evidence or identify questions, but each audit conclusion should point to records an investigator can inspect.
Label uncertainty precisely. “The retained record does not show which source revision was used” is a useful finding. Claiming that the agent used today's revision because it seems plausible hides an evidence gap and can direct the team toward the wrong corrective action.
Test the suspected failure in a controlled environment
Once the evidence suggests a mechanism, create a synthetic or sanitised reproduction. For a wrong-owner change, use similar record names and conflicting ownership rules. Check whether the current configuration repeats the error and whether the authorization and approval boundaries still hold.
Preserve relevant version information. A model, prompt, retrieval rule, or connector may have changed since the incident. A failed reproduction with the current release does not disprove the earlier event, and replaying the same text does not guarantee an identical model response.
Test the proposed repair against the failure mechanism. If the problem was a mutable approved payload, rewriting the prompt is insufficient. The corrective action belongs in proposal version binding and executor validation. Add the relevant case to agent evaluations and the affected integration tests.
Close the audit with findings people can act on
Separate observed facts, supported conclusions, unresolved questions, and corrective work. For each defect, identify the control that failed, the owner responsible for changing it, and the evidence required to verify the repair. A general instruction to make the agent more careful is not an implementable outcome.
Consider whether the current workflow should remain paused or restricted while the finding is addressed. Record the decision and the condition for resuming, including who may authorise it. Preserve the original evidence and append the outcome rather than rewriting the incident history to reflect the repaired system.
The companion guide to audit trail contents describes what must be recorded in advance. An audit demonstrates whether those records are actually sufficient to connect an outward result to its evidence, decision, and permitted execution path.
Frequently asked questions
Is the conversation history enough to audit an agent?
Usually it is only part of the evidence. A conversation can show the request and visible answer while omitting retrieved records, permission checks, proposal revisions, approval authority, and the executor's destination result. Reconstruct the workflow using linked operational records and verify the resulting artifact rather than treating the transcript as a complete history.
Can an audit explain the model's internal reasoning?
An audit can explain the observable basis for a proposal and the controls surrounding it. It should use source evidence, tool activity, written rationale, human decision reasons, and execution records. Those artifacts do not reveal private internal reasoning, and a retrospective explanation generated by a model should not be presented as proof of what happened.
What if the logs are incomplete?
State which conclusions cannot be established and preserve the evidence that remains. Check authoritative destination history and other permitted records before deciding whether the gap can be resolved. Treat missing audit evidence as its own finding, with a retention or instrumentation repair, rather than filling the gap with an attractive but unsupported narrative.
Should the team replay the incident against production?
Use an isolated environment with synthetic or appropriately sanitised data to test the suspected mechanism. Replaying an outward action against live systems can create additional changes or expose sensitive information. Preserve the relevant configuration and expected boundary, and verify repairs through controlled tests before deciding whether the affected production capability may resume.
