Synthetic test data lets you check an AI agent against cases you design, with known right answers and no real customer records. It can prove the agent follows rules, handles edge cases and refuses unsafe requests. It cannot prove the agent copes with how real people actually write, so pair it with reviewed samples of real traffic before launch.
Why teams reach for synthetic data
Real records are the obvious test set, but they are often the hardest to use. They may hold personal or confidential data that should not sit in a test harness, they rarely come with an agreed right answer, and they change as the business changes. Synthetic data removes those problems because you write the case and the expected outcome together.
It also lets you test things that rarely happen. A refund request for an order that was split across two warehouses, a contract with two conflicting renewal clauses, a customer who writes in two languages in one message: real logs may hold a handful of these, or none. With synthetic cases you can create as many as you need, on purpose, before the agent meets one in production.
What synthetic data can prove
Synthetic cases are strong evidence when the question is "does the agent follow the rule we gave it?". Good uses include:
- Policy and permission rules. Requests the agent must refuse, approvals it must ask for and data it must not reveal. The expected outcome is clear, so a pass or fail means something.
- Edge cases in your own data. Missing fields, out-of-date records, duplicate customers and documents with conflicting versions. Seeding an old policy next to its replacement shows whether the agent picks the one in force, not just the closest match.
- Tool use. Whether the agent calls the right tool with the right arguments, and what it does when a tool returns an error or an empty result. This pairs well with sandbox testing of write actions.
- Regression after a change. A fixed synthetic set, run after every prompt, model or tool change, shows quickly whether something that worked has broken.
What it can't prove
Synthetic data is written by people, or by a model, who already know what the agent is supposed to do. That shapes it in ways real traffic is not.
- Real phrasing. Customers write vaguely, mix several requests into one, paste screenshots and use internal shorthand. Generated cases tend to be tidy and complete, so an agent can pass them and still struggle with the real thing.
- The mix of requests. A synthetic set reflects what you thought to test, not how often each kind of request really arrives. A high pass rate on rare edge cases says little about the common path.
- Unknown failure modes. You can only generate cases for problems you can imagine. The failures that hurt most after launch are often ones nobody predicted.
- Model blind spots. If the same model family generates the cases and runs the agent, both can share the same gaps. The test then agrees with the agent instead of challenging it.
How to build a synthetic set that earns trust
Start from the agent's written requirements, not from a generator prompt. List the rules, tools and data situations the agent must handle, then write or generate cases for each one, with the expected outcome stated before the agent runs.
Have a person who knows the domain review a sample of the generated cases. They will spot cases that are unrealistic, too easy or wrong about how the business works. Keep the cases under version control next to the agent's prompts and tool definitions, so a change to one can be traced to the test that covers it.
Mix how cases are made. Rule-based generators give repeatable data with exact expected values, which suits structured records. Model-generated text adds variety in wording, which suits conversations and documents. Hand-written cases cover the situations that matter most to the business. Scoring then follows the approach in what AI agent evals catch: check outcomes and tool calls, not just the final wording.
Pairing synthetic data with real traffic
Before launch, take a reviewed sample of real requests from the channel the agent will serve, with personal data removed following the same approach you use to keep personal data out of AI agent logs. Run the agent on it in a shadow mode, where its answers are recorded but not sent, and have reviewers mark the results.
Compare the two sets. If the agent passes the synthetic set but struggles on real samples, the synthetic cases are too clean, and the failures tell you which cases to add. After launch, turn reviewed failures from real use into new test cases, so the synthetic set keeps moving closer to what the agent actually sees. For retrieval-heavy agents, test the search step on its own as well, as described in RAG retrieval testing.
If you want help designing a test set for an agent you are building, talk to our team.
Frequently asked questions
Can synthetic data replace real data when testing an AI agent?
No. Synthetic data is good at checking rules, permissions, tool calls and edge cases you can describe in advance. It does not show how real users phrase requests or how often each kind of request arrives. Use it as the base of your test suite and add a reviewed, anonymised sample of real traffic before launch, then keep adding real failures after go-live.
Is it safe to use a language model to generate test cases?
Yes, if a person reviews a sample and the expected outcomes are set from your requirements rather than by the model. Model-written cases add useful variety in wording but can be unrealistic or quietly wrong. Where possible, use a different model family to generate cases than the one running the agent, so the test does not share the agent's blind spots.
How many synthetic test cases does an agent need?
There is no fixed number. Aim for at least one case for every rule, tool, permission and data situation in the agent's requirements, plus extra cases for the outcomes that would cost the business most if they went wrong. Grow the set over time by adding a case for each real failure found in review or production.
When should synthetic tests run?
Run the full synthetic set whenever the prompt, model, tools or connected data sources change, and before every release. A smaller, fast subset can run on every commit. Because the expected outcomes are fixed, a drop in results points straight at the change that caused it, which makes regressions much easier to find than with ad hoc manual checks.
