Testing a multilingual AI agent means running the same task-level evals in every language your users will write in, not only in English. Agents tend to slip in tool choice, argument extraction and multi-step consistency when the input language changes, so each language needs its own test cases, pass criteria and sign-off before it goes live.
Why English results do not carry over
Most agent test suites are written in English because the team writes in English. The agent then goes live for customers who write in Hindi, Tamil, Arabic, German or a mix of languages in the same message, and nobody has checked how it behaves there.
Research on multilingual agent benchmarks, such as the MAPS benchmark for agent performance and security, points the same way: agents inherit the uneven language behaviour of the models underneath them, and a non-English request alone can raise the chance of a wrong tool action. The weak point is usually not the reply text. It is the orchestration in between: deciding which tool to call, filling its arguments and keeping track of the task across several steps.
Where multilingual agents go wrong
Knowing the failure types tells you what your test cases need to cover.
Tool choice
Tool names and descriptions are usually written in English. When a request arrives in another language, the agent has to map it to the right English tool description, and close matches get confused. A request to cancel one item in an order can end up calling the tool that cancels the whole order.
Argument extraction
Dates, amounts, names and addresses follow local formats. A date written day first can be read month first. A number with a comma as the decimal separator can be read as a much larger number. Names in non-Latin scripts can be transliterated differently from the spelling in your customer record, so a lookup fails or finds the wrong person.
Consistency across steps
In a long task, the agent may read a document in one language, reason in English and reply in a third. Details get lost or changed between steps, and the final answer can contradict what the source actually said.
Guardrails and refusals
Safety filters and policy checks tuned on English text may miss the same request phrased in another language. Equally, they can block ordinary requests because a translated word looks like a risky one. Both need testing.
Code-mixed input
Many users mix languages in one message, or write their language in Latin script. These messages are common in real traffic and rare in test sets.
How to build the test set
Start from your existing agent evals and extend them language by language rather than writing a separate suite.
- Pick languages from real traffic. Use support logs or market plans to decide which languages and scripts matter, including code-mixed forms.
- Write cases natively, do not only translate. Machine-translated English tests miss local phrasing, formats and politeness forms. Have fluent speakers write or review a core set for each language.
- Test the actions, not just the words. For every case, check which tool was called, with which arguments, and what changed in the target system. A fluent reply on top of a wrong action is still a failure.
- Cover local formats on purpose. Include dates, currency amounts, phone numbers, addresses and names in each locale's own conventions.
- Run the same cases in a sandbox. Write actions should be exercised against test systems so you can compare results across languages safely.
Set pass criteria per language
Report results per language, not as one blended score. A blended score lets a strong English result hide a weak result in a smaller market. Agree with the product owner what pass means for each language and which failures block a launch, such as any wrong write action or any guardrail bypass.
Where a language does not reach that bar, you have options short of shipping it as is: route it to a human, ask the user to confirm extracted details before acting, restrict the agent to read-only tasks in that language, or keep it out of launch until it passes.
Design changes that reduce language risk
Some fixes belong in the agent's design rather than in more testing.
- Keep tool definitions clear and distinct, so similar tools are hard to confuse in any language.
- Use structured fields for dates, amounts and identifiers, and validate the agent's output against them before any write action.
- Normalise names and addresses before lookups, and confirm matches with the user when the script differs from the stored record.
- Log the detected input language with every request, so monitoring can show error rates by language after launch.
Where CodeDTX helps
CodeDTX engineers AI agents for enterprise systems that serve users in many languages, including the evaluation sets, guardrails and monitoring each language needs. If your agent has only been tested in English, talk to us about building multilingual tests before it reaches more of your users.
Frequently asked questions
Can we translate our English test cases instead of writing new ones?
Translation is a useful starting point, but it should not be the whole set. Machine-translated cases tend to use textbook phrasing and miss local date and number formats, informal wording and code-mixed messages that real users send. Use translation to cover the breadth of your existing scenarios, then have fluent speakers write or review a core set of cases in each language you plan to support.
Should the agent's tools and prompts be translated too?
Usually not. Most teams keep tool names, descriptions and system prompts in one language so they are easier to version and review. What matters more is that tool descriptions are clear and distinct, and that the agent is tested with requests in each language against those descriptions. If testing shows repeated confusion between two tools, rewrite or merge them before considering translated prompts.
How do we handle a language the agent is weak in?
You do not have to choose between shipping it unchanged and leaving it out. You can route that language to human staff, limit the agent to answering questions without taking actions, or make it confirm extracted details such as dates and amounts with the user before acting. Keep testing it with each model or prompt change, and widen what it can do once it passes your agreed bar.
Do we need to retest every language after a model upgrade?
Yes. A new model or model version can change behaviour in one language while leaving English results unchanged, so an English-only regression run will not catch it. Run the full multilingual eval set for every model change, compare results per language with the previous version, and release the upgrade in stages, watching error rates by language in production before moving all traffic.
