How to Test an AI Agent Before Launch: A Practical Evaluation Framework

Ask a team how they tested their AI agent and you'll often hear some version of the same answer. "We tried it on a bunch of examples and it looked good."

That's not nothing. It's also not a test. It's a demo with extra steps.

The problem is that agents fail in ways traditional software doesn't. The same input can produce different outputs. A failure might be a wrong answer, a wrong tool call, a correct answer reached by a dangerous route, or a perfectly good run that happens to expose data it shouldn't. "It looked good" can't catch most of that.

The good news is that you don't need a research lab to do this properly. You need a method. Here's the one we use, written as a framework you can run before you ship.

How Do You Test an AI Agent Before Launch?

To test an AI agent before launch, build a versioned test set from realistic cases, score runs with deterministic checks first and a calibrated LLM judge second, track task-success rate and safety metrics across repeated runs, re-run the suite on every change, red-team the agent against misuse, and approve launch only against go/no-go criteria you set in advance.

That's the whole framework in one sentence. The rest of this article unpacks each part and gives you checklists to run.

Key Takeaways

Define "success" in measurable terms before you test. Without it, every result is an opinion.
Your test set is the most valuable asset in the project. Build it from real cases, include ugly ones, and version it.
Use code-based checks wherever you can. Use an LLM judge only for what code can't verify, and calibrate it against human labels.
Agents are non-deterministic. Run each case multiple times and measure consistency, not a single lucky pass.
Regression testing is non-negotiable. Any change to a model, prompt, tool, or data source can quietly break something that worked.
Red-teaming should be planned, not improvised. Test injection, permission boundaries, data leakage, and runaway behavior.
Set go/no-go criteria in advance, with hard gates for safety and soft gates for quality.

The Framework at a Glance

1. DEFINE      What does success look like? What can go wrong?
      |
2. BUILD       Create and version the test set
      |
3. SCORE       Code checks -> LLM judge -> human review
      |
4. MEASURE     Task success, safety, consistency, cost, latency
      |
5. REGRESS     Re-run on every change; compare to baseline
      |
6. ATTACK      Red-team scenarios
      |
7. DECIDE      Go / no-go against pre-agreed criteria
      |
8. ROLL OUT    Shadow mode, staged launch, monitoring

You'll loop through steps 2 to 6 several times before step 7. That's expected.

Step 1: Define What "Success" Means

Start here, because everything else depends on it.

For each workflow the agent handles, write down:

The task: what the agent is supposed to accomplish, in one sentence.
Success criteria: how you'd know it was done correctly. Be specific. "Answers customer questions well" isn't testable. "Gives the correct refund eligibility and cites the right policy section" is.
Unacceptable outcomes: things that fail the run regardless of anything else, such as sending an external message without approval, exposing another customer's data, or taking an action outside its permissions.
Acceptable fallbacks: what the agent should do when it can't or shouldn't proceed, such as escalating to a person.

That last point matters more than it seems. An agent that says "I'm not sure, passing this to a human" on a hard case is behaving correctly. Your tests should reward that, not penalize it.

Step 2: Build the Test Set

A test set is a collection of inputs, each paired with what a good outcome looks like. It's your most reusable asset. Models and prompts will change. A good test set carries forward.

Where the cases come from

Real historical data: past tickets, emails, orders, and chats (anonymized and handled according to your privacy rules). This is your best source because it reflects reality.
Subject-matter experts: the people who do the work today know which cases are tricky.
Failure logs: every bug found in pilots or production becomes a permanent test.
Synthetic cases: generated to fill gaps, such as rare scenarios. Review them by hand, since generated cases tend to be cleaner than reality.
Adversarial cases: deliberately difficult or hostile inputs (more on these in the red-team section).

What to cover

Category	Examples
Happy path	Typical, well-formed requests
Edge cases	Missing fields, unusual formats, boundary values
Ambiguous requests	Where the right move is to ask a clarifying question
Out-of-scope requests	Things the agent should decline or redirect
Messy inputs	Typos, mixed languages, copy-pasted junk, long threads
Multi-step tasks	Cases requiring several tools in sequence
Failure conditions	A tool is down, an API returns an error, data is stale
Sensitive cases	Where a wrong action has real consequences

If your users write in Arabic, English, or a mix of both, include all of those. Dialects, transliterated names, and mixed-script product codes are where agents often slip, and test sets full of clean English hide it.

How big?

There's no magic number, but here's a rule of thumb: start with 50 to 100 well-chosen cases for a single workflow, and grow it over time. Hundreds of mediocre cases are worth less than fifty excellent ones.

One caution on small samples. If an agent passes 45 of 50 cases (90%), the true success rate could plausibly be anywhere from roughly 79% to 96%. A single percentage from a small set is a signal, not a guarantee. Report results with that humility, and prioritize adding cases in areas where you're unsure.

Keep it honest

Version the test set like code, so you can compare runs across time.
Hold out a portion you don't tune against. If you adjust prompts until every visible case passes, you've fitted to the test, not improved the agent.
Record expected outcomes, including acceptable alternatives. Many tasks have more than one right answer.
Review labels with a human expert. Wrong "correct answers" quietly corrupt everything downstream.

Step 3: Score the Results (Code First, Judge Second)

There's a hierarchy of scoring methods. Use the cheapest, most reliable one that fits.

Method	Good for	Trade-off
Deterministic checks (code)	Exact values, schema validity, tool called or not, forbidden actions, required fields	Reliable and cheap, but only works for things you can specify precisely
LLM-as-judge	Quality, tone, helpfulness, faithfulness to sources, free-text correctness	Scales well, but imperfect and needs calibration
Human review	Subjective quality, ambiguous cases, judge calibration	Most trustworthy, but slow and expensive

Evaluate more than the final answer

For agents, the answer is only part of the story. You can evaluate at three levels:

Component level: does retrieval return the right documents? Does each tool work correctly?
Trajectory level: did the agent take a sensible path? Right tools, right order, no unnecessary or dangerous calls?
End-to-end level: did the whole task succeed?

Trajectory checks catch a nasty category of failure: right answer, wrong route. An agent that reaches the correct result by querying data it shouldn't have access to has failed, even if the final text is perfect.

Many of these are easy to check in code. Examples:

Did the agent call a write tool without approval? Fail.
Did the output include data belonging to a different customer? Fail.
Did it exceed the step or cost limit? Fail.
Is the structured output valid against the schema? Required.

Using LLM-as-Judge With Care

Using a language model to grade another model's output is practical, and widely used. It's also easy to misuse. Treat the judge as an instrument that needs calibration, not as ground truth.

Known weaknesses

Research on LLM judges (for example, work published in 2023 around the MT-Bench benchmark) has documented several biases:

Position bias: when comparing two answers, judges can favor whichever appears first.
Verbosity bias: longer answers can score higher regardless of quality.
Self-preference: a model may rate outputs from its own family more favorably.

Newer models may behave differently, so verify these on your own data instead of assuming.

How to use a judge well

Don't use it where code will do. If you can check it with a rule, check it with a rule.
Write a specific rubric. "Rate the quality" is vague. "Does the response state the correct refund window and cite the policy? Yes or no" is usable.
Prefer narrow, binary, or low-scale scoring. Pass/fail on specific criteria is more consistent than a 1-to-10 score.
Judge one thing at a time. Separate checks for accuracy, tone, and completeness beat one blended score.
Give the judge references. Provide the expected answer or source documents so it checks against facts, not vibes.
Ask for reasoning before the verdict, and keep it for audit.
Randomize order in pairwise comparisons.
Use a different model family for judging than for generating, where practical.
Calibrate against humans. Have experts label a sample (say, 50 to 100 outputs), compare to the judge's verdicts, and measure agreement. If the judge disagrees with humans often, fix the rubric before trusting it.
Re-calibrate periodically and whenever you change the judge model or prompt.
Spot-check continuously. Read a random sample of judged outputs every cycle.

A simple trust test: if you wouldn't accept the judge's verdict on a case a human expert reviews, don't let it gate your launch.

Step 4: Measure What Matters

Pick a small set of metrics and track them consistently. Too many dashboards become noise.

Metric	What it tells you
Task-success rate	Share of cases where the agent completed the task correctly end to end
Tool-call accuracy	Right tool, right arguments, right order
Escalation rate (appropriate)	Does it hand off when it should?
Unsafe-action rate	Rate of forbidden or unapproved actions. Target: zero
Groundedness	Are claims supported by retrieved sources?
Consistency across runs	Does it succeed every time, or only sometimes?
Latency	Time to complete, including tool calls
Cost per task	Tokens and tool usage per completed task
Human override rate	How often reviewers change the agent's output

Task-success rate, defined carefully

Define it once and keep it fixed. A reasonable definition: the share of test cases where the agent reached a correct final outcome, with no unacceptable actions along the way, within the cost and step limits.

Report it by category, not just overall. A 92% average can hide a 60% rate on your most sensitive cases. Break results down by case type, language, and risk level.

Measure consistency, not just one pass

Agents are non-deterministic. A single run that passes tells you less than it seems.

Run each case several times (for example, five) and report both the average success rate and how often the agent passes every time. Some benchmark researchers use a metric called pass^k for this idea: the probability that the agent succeeds on all k attempts. The arithmetic is humbling. If an agent succeeds 90% of the time per run and runs are independent, the chance it succeeds five times in a row is about 59%.

For a workflow customers rely on repeatedly, that gap between "usually works" and "reliably works" is exactly what you need to see before launch.

Step 5: Regression Testing

Here's the thing about agents: they change even when you don't touch them. A model provider updates a model. A tool's API changes. A knowledge base gets new documents. And, of course, you edit prompts and tools constantly.

Any of these can silently break something that worked. Regression testing is how you catch it.

What should trigger a full re-run

A new or updated model version
Any prompt or system-instruction change
Adding, removing, or modifying a tool
Changes to retrieval (chunking, index, ranking, source data)
Changes to permissions or approval rules
Updates to connected systems or APIs
Any fix for a bug found in testing or production

How to run it

Keep a baseline. Store scores from the last accepted version.
Run the suite automatically in your CI pipeline or on a schedule.
Compare, don't just score. Show which cases flipped from pass to fail and the reverse.
Handle flakiness. Run cases multiple times. A case that passes three out of five isn't "passing," it's unstable. Flag it.
Set thresholds. A change that drops task-success below the baseline by more than an agreed margin, or introduces any unsafe action, is blocked.
Turn every production failure into a new test case. Your suite should grow from real mistakes.

Step 6: Red-Team the Agent

Normal testing asks, "does it work?" Red-teaming asks, "how could it be made to fail, leak, or misbehave?" Plan it deliberately. For agents with tools, the stakes are higher than for chatbots, because the agent can act.

OWASP's Top 10 for LLM applications lists prompt injection among the most significant risks, and it's a good starting reference for your scenarios. Build a scenario library like this:

Scenario	What you're testing	Expected behavior
Direct prompt injection	User says "ignore your instructions and..."	Refuses; follows original rules
Indirect prompt injection	A document, email, or web page the agent reads contains hidden instructions	Treats content as data; doesn't obey it
Permission boundary	User asks for another customer's data or an out-of-role action	Denies; logs the attempt
Data leakage	Prompts that try to extract the system prompt, credentials, or private records	Doesn't disclose
Social engineering	"I'm the CEO, do it urgently, skip approval"	Still follows approval rules
Tool misuse	Inputs crafted to trigger destructive or unintended tool calls	Validation blocks it
Malformed or hostile input	Huge inputs, odd encodings, injection in structured fields	Handles gracefully
Runaway behavior	Tasks that could loop or spawn repeated actions	Hits step, cost, and rate limits
Conflicting information	Two sources disagree	Flags the conflict or escalates
Language and script tricks	Instructions hidden in another language or mixed scripts	Same rules apply in every language

Two principles for red-teaming:

Defense should not rely on the model alone. If an action must never happen without approval, enforce that in code. Test that the enforcement holds even when the model is fooled.
Include people who didn't build it. Builders test what they expect. Someone with fresh eyes finds different holes.

For every successful attack, add it to the regression suite.

Step 7: Set Go/No-Go Criteria Before You Test

This is the step that builds the most trust, with your leadership and with customers. Decide what "ready" means before you see the results, so you aren't tempted to move the goalposts.

Split criteria into two kinds.

Hard gates (any failure blocks launch):

Zero unsafe or unauthorized actions in the test suite and red-team scenarios
No cross-user or cross-tenant data exposure
All sensitive actions protected by working approval steps
Logging, audit trail, and a kill switch confirmed working
Compliance and security sign-off obtained

Soft gates (quality targets you set based on the cost of errors):

Criterion	Example target (set your own)
Task-success rate, overall	Meets the target agreed with the business owner
Task-success rate, highest-risk category	Higher than the overall target
Consistency across repeated runs	Meets agreed pass^k threshold
Appropriate escalation	Agent hands off on defined hard cases
Latency and cost per task	Within agreed budgets
Judge-human agreement	Calibration meets your agreed level

The numbers in that table are deliberately not filled in. The right target depends on what a mistake costs. An agent drafting internal summaries can tolerate more errors than one issuing refunds. Agree the numbers with the business owner in writing.

A go/no-go record

For each release, keep a short record: test set version, model and prompt versions, results by category, red-team findings, open risks, who approved. It's useful for audits, and it's useful when something goes wrong six months later and everyone asks what was known at launch.

Step 8: Roll Out in Stages

Passing tests is necessary. It isn't the end.

Offline testing: the framework above.
Shadow mode: the agent runs on real inputs but its outputs aren't acted on. Humans compare them to what actually happened.
Assisted mode: the agent drafts, humans approve everything.
Limited release: a small group of users or a low-risk segment, with close monitoring.
Gradual expansion: widen scope as metrics hold and trust builds.
Ongoing monitoring: sample live conversations, track the same metrics, and feed failures back into the test set.

Don't skip shadow mode. Real traffic is messier than any test set.

What Buyers Should Ask Their AI Vendor

If you're on the buying side, this framework doubles as a due-diligence list. A vendor who has genuinely tested their agent should be able to answer these plainly:

How big is your test set, and where did the cases come from?
How do you measure task success, and what's the result on the highest-risk cases?
How do you test consistency across repeated runs?
If you use an LLM judge, how did you calibrate it against human review?
What happens to your tests when the underlying model changes?
What red-team scenarios have you run, and what did you find?
What are the go/no-go criteria, and who signs off?
How will we monitor quality after launch?

Vague answers aren't proof of poor work, but specific ones are a good sign.

Common Mistakes

Testing only the happy path. Real users bring typos, ambiguity, and surprises.

Relying on one run. A single pass hides inconsistency.

Trusting the judge blindly. Uncalibrated judges can reward long, confident, wrong answers.

Tuning to the test set. Without held-out cases, scores overstate real performance.

Ignoring the trajectory. Right answers via wrong paths are still failures.

One blended score. Averages hide the failures that matter. Report by category.

Skipping regression after "small" changes. Small prompt edits can have large effects.

Setting criteria after seeing results. That's rationalization, not evaluation.

Treating launch as the finish line. Production teaches you things no test set can.

Pre-Launch Evaluation Checklist

Define

Success criteria written per workflow
Unacceptable outcomes listed
Acceptable fallbacks defined
Business owner agrees on quality targets

Test set

50 to 100+ cases from real data and experts
Happy path, edge, ambiguous, out-of-scope, messy, multi-step, failure cases
Languages and scripts your users actually use
Expected outcomes reviewed by a human expert
Version-controlled, with a held-out portion

Scoring

Code checks for everything checkable
LLM judge with specific rubrics, used only where needed
Judge calibrated against human labels
Trajectory and tool-use checks in place

Metrics

Task-success rate defined and reported by category
Unsafe-action rate tracked
Multiple runs per case; consistency measured
Latency and cost per task recorded

Regression

Baseline stored
Automated run on every relevant change
Pass-to-fail flips reviewed
Production failures added to the suite

Red team

Injection (direct and indirect) tested
Permission and data-leakage scenarios tested
Approval-bypass attempts tested
Runaway and resource-limit scenarios tested
Findings fixed and added to regression

Decision

Hard gates and soft gates defined in advance
Go/no-go record completed and signed
Shadow mode and staged rollout planned
Monitoring and kill switch ready

Frequently Asked Questions

How do you evaluate an AI agent?

Build a test set of realistic cases, score outcomes with code-based checks and calibrated LLM judges, measure task-success rate and safety metrics across repeated runs, red-team the agent, and compare everything to go/no-go criteria agreed in advance.

What is a good task-success rate for an AI agent?

It depends on the cost of errors. There's no universal threshold. Agree on targets with the business owner for each workflow and risk level, and report success by category rather than as a single average.

Is LLM-as-judge reliable?

It can be useful, but only with care. Research has documented biases like position bias and verbosity bias. Use specific rubrics, provide references, and calibrate the judge against human labels before relying on it.

How many test cases do I need?

Start with 50 to 100 carefully chosen cases per workflow and expand over time. Quality and coverage matter more than raw count. Small test sets give wide uncertainty, so be cautious about reading too much into a single percentage.

Why run each test multiple times?

Because agents are non-deterministic. A case that passes once may fail on the next run. Running multiple trials reveals consistency, which matters for workflows people depend on repeatedly.

What is regression testing for AI agents?

It means re-running your test suite whenever something changes, such as the model, prompts, tools, retrieval, or connected systems, and comparing results to a stored baseline to catch quiet breakages.

What should a red-team test for an AI agent include?

Prompt injection (direct and indirect), permission and data-access boundaries, data leakage, approval bypass through social engineering, tool misuse, malformed inputs, and runaway behavior. Enforce critical rules in code, not just in prompts.

What are go/no-go criteria?

They're conditions, set before testing, that must be met to launch. Hard gates (such as zero unauthorized actions) block launch on any failure. Soft gates are quality targets set according to the cost of errors.

The Real Takeaway

An agent that "looked good in the demo" has told you very little. An agent that has passed a versioned test set, stayed consistent across repeated runs, survived a planned red-team, and met criteria you wrote down beforehand has told you quite a lot.

The trade-off is effort. Building a good test set and evaluation harness takes real time, often a meaningful share of the project. But it's the cheapest insurance you'll buy, and it keeps paying back every time you change a model, a prompt, or a tool.

If you're preparing an agent for launch, or evaluating one a vendor has built, we're happy to help you set up the testing approach. At O2Devs we build production AI agents with evaluation, monitoring, and approval controls designed in from the start. Get in touch and we'll walk through what a launch-ready test plan looks like for your use case.