AI agent evals: a correct answer can still fail

Your agent drafts a convincing status update. The wording is accurate, the tone is appropriate, and the customer’s name appears in the greeting.

The supporting incident belongs to another account.

That run should fail. A good text score cannot repair the wrong evidence or an access violation. Your AI agent evals need to make that outcome explicit.

The following example is fictional: an agent must draft an update for Cedar, issue 417. It may read approved records, but it may not send messages or change business data.

The task is small enough to inspect. That makes it useful for designing a test contract you can actually enforce in your evaluation harness.

TLDR

• Define the expected evidence, permitted calls, and forbidden effects before you run the test.
• Separate attempted access, enforcement denial, actual retrieval, and disclosure in the final answer.
• Compare versions against the same assertions, allowing valid alternative paths without relaxing the task’s boundaries.

What are AI agent evals?

AI agent evals test whether an agent performs a defined task under specified conditions. They can assess its answer, source evidence, tool calls, and resulting environment state. A useful evaluation states the acceptance criteria before execution and distinguishes measured behavior from the controls that restrict access or actions during the run.

Think of the evaluation as a contract between the task and the evidence. Your test setup supplies the conditions, runs the trial, and applies checks called graders.

A grader can be a deterministic check, a model judgment, or human review. The right choice depends on the claim you need the evidence to establish.

In Anthropic’s evaluation guide, Mikaela Grace and coauthors distinguish the interaction transcript from the outcome. A reported reservation and an actual database reservation are different evidence.

Apply the same discipline to a read-and-draft task. A sentence claiming that the correct account was checked isn’t proof that the lookup used it.

The answer looks right. The account is wrong.

Your evaluation should identify the first point where the run diverged from the task. Start with the requested account and follow the evidence through the call.

The fictional task requires Cedar’s issue 417 and its permitted linked product note. The agent instead asks for another account’s incident, and the test environment returns it.

The resulting draft contains plausible status information. It still violates the test contract because the answer rests on the wrong account’s evidence.

In short: acceptable wording cannot turn an unauthorized or mismatched source into valid task evidence.

Keep those observations separate. You may fix the lookup behavior and still have an enforcement gap in the environment. You may fix enforcement and still need to correct the agent’s attempted behavior.

Separate the attempted call from the actual exposure

Run a second version of the fixture where the wrong-account request is denied. No protected record returns, and the agent explains that it cannot verify the status.

The access boundary worked in that trial, so it would be inaccurate to report a completed leak.

The attempted lookup may nevertheless fail this case’s behavior rule. Your task permits access only to the approved resources; the agent requested something else.

This distinction lets you ask useful questions. Did the agent choose the wrong arguments? Did the enforcement layer reject them? Did any unauthorized data return? Did the response disclose it?

Don’t collapse those questions into one “safety score.” They identify different defects and different owners.

Use agent observability to understand the available trace evidence. A trace helps diagnose a run, but it is not a ground-truth account of private model reasoning.

Collect only the records and event details needed for evaluation. Unrestricted payload logging can create another disclosure problem while you investigate the first one.

Write the test contract before the next run

A useful agent evaluation framework begins with an explicit task and a checkable result. Write the case before modifying the prompt that produced the failure.

Otherwise, you can end up grading whatever the revised prompt happens to do well. The expected result should remain independent of that implementation.

Use controlled, fictional fixtures rather than copying unrestricted customer records into a test. Record the account scope, permitted source versions, tool schemas, and starting business state.

In short: the case defines both the work to complete and the boundaries that success cannot override.

The permitted-telemetry row matters. “Nothing changes” is too broad if your test setup legitimately writes test results. Name the business state that must remain untouched.

The safe testing lifecycle establishes the surrounding test environment. This case card adds the assertions for one task rather than replacing that broader process.

Verified source material is another prerequisite. AI knowledge management supports the source relationships and ownership that your expected answer depends on.

If the product note is outdated, a grader demanding its old answer will reward a mistake. Maintain source expectations as deliberately as the agent configuration.

Specify what must remain unchanged

Capture the outbox and relevant business-record state before the trial. Check them again afterward through a trusted test interface.

A trace line saying “draft only” doesn’t independently establish that no message was sent. Inspect the evidence that can actually answer the question.

Write hard-fail rules separately from quality dimensions. An unauthorized retrieval must not disappear inside a high average for tone, completeness, and helpfulness.

Also state which test you are running. This behavior case fails a forbidden attempted call. A separate containment test may deliberately inject that call and pass when enforcement denies it.

Both tests can be useful. Combining their goals without naming them makes the results contradictory.

Give the harness a known passing reference and a deliberately failing run. Your AI agent evals should detect defects in their own grading logic before judging a release candidate.

Which checks decide whether this case passes?

Assign each assertion to evidence it can inspect reliably. Use deterministic checks for exact boundaries and calibrated judgment for genuinely qualitative questions.

A grader should not guess permission from a friendly response. Nor should a source-ID match establish that every sentence is supported. This is where AI agent evals earn their keep: each check is tied to evidence it can actually inspect.

The checks work together, but they answer different questions.

Deterministic assertions for facts and boundaries

For this case, programmatic checks can compare requested identifiers, returned source identifiers, permitted arguments, and before/after business state.

Tool-use evaluation needs more than a list of tool names. The right lookup tool with the wrong account argument is still wrong for this task.

The official DeepEval Tool Correctness documentation describes configurable matching for tool inputs and outputs. Exact matching and order checks depend on configuration.

That doesn’t make a recorded tool call an independent state inspection. Add the state check where your assertion requires one.

In short: a grader is useful when its evidence supports the exact conclusion you record.

A partial trace should produce uncertainty where evidence is missing. It should not automatically produce a pass because the grader found no recorded violation.

Treat collection failures as test failures or invalid trials under your policy. Keep that result distinguishable from an agent behavior failure.

Calibrated judgment for the response

Use a language-model judge, often called LLM-as-judge, for dimensions such as clarity and how well the draft explains verified information.

Give it a rubric with examples and an insufficient-evidence option. Compare its decisions with human judgments and investigate meaningful disagreements.

Version the rubric and judge configuration. A higher score after changing the grader does not necessarily indicate a better agent.

DeepEval’s Argument Correctness metric uses model judgment by default. That differs from deterministic matching against expected tool parameters.

Its Task Completion metric can derive a completion judgment from recorded traces. That does not automatically verify a separate database or outbox.

Choose the mechanism according to the assertion. Do not treat similarly named metrics as interchangeable proof.

The agent sandboxing boundary remains a separate control. Your grader evaluates recorded behavior; the execution environment restricts what the agent can do while running.

A different tool path can still be correct

Require the constraints the task depends on, not every step of one successful demonstration. Exact trajectory matching can reject a valid solution.

The Cedar task might obtain its product note through an approved lookup. Another run might follow an already-authorized source reference from the issue record.

Both routes can satisfy the case if they retrieve the permitted evidence and preserve the no-send boundary. They don’t need identical call sequences.

Anthropic’s guide recommends accommodating valid alternative solutions when designing graders. A reference answer helps establish feasibility; it should not become an accidental requirement for one implementation.

Specify necessary ordering where it matters. If protected access requires an identity check first, preserve that dependency rather than allowing any permutation of calls.

An extra permitted read might be an efficiency concern. A wrong-account read is an access concern. Record them separately instead of assigning the same generic penalty.

You can add a local latency or cost budget when the task needs one. Explain how that budget relates to the workflow rather than importing a universal threshold.

Turn the failure into a regression case

Regression testing should preserve the failure you want the next version to catch. Keep each case’s assumptions explicit so it does not become a mysterious test nobody trusts.

Reset the environment between trials. Inherited caches, prior messages, or changed records can make a later run easier without improving the agent.

Use repeated trials appropriate to the task’s variability. Report how many trials you ran and what each outcome means, rather than presenting a single success as reliability.

Anthropic distinguishes finding at least one success across attempts from succeeding on every attempt. Those questions serve different purposes, especially when a failed attempt can cause harm.

A later successful retry does not erase an earlier unauthorized retrieval. Your summary needs to preserve the boundary failure even when the final customer response looks acceptable.

Compare the candidate with the same baseline conditions

Run the baseline and candidate against comparable fixtures, policies, tool interfaces, and grading rules. Record any difference you cannot control.

For the Cedar case, compare supported answers, correct holds, forbidden attempts, actual exposures, and unchanged business state. Keep grader uncertainty visible rather than excluding difficult cases silently.

A candidate can improve response quality while worsening access behavior. A single aggregate score could obscure that trade-off, so your AI agent evals should report hard failures alongside quality results.

Before blaming the agent, inspect a sample of failing traces and grader decisions. A broken fixture or outdated expected source can create a false regression.

Keep development cases separate from held-out evaluation where practical. Repeatedly tuning to the same failure case can teach the implementation to satisfy the test without addressing the broader class.

The continuous agent testing process owns that wider loop. Your contribution here is a maintained case with explicit assertions and an accountable reviewer.

Tooling helps when it exposes the evidence you need. It doesn’t make every useful assertion automatic.

DevRev positions Computer, by DevRev as an AI resolution platform. Its Agent Studio reference describes Playground sessions, bulk correctness and completeness tests, and session traces.

Use those documented checks for their stated purpose. The account-scope and external-state assertions in this case still need explicit implementation and verification in your evaluation setup.

After deployment, monitor the same failure class through approved telemetry. Follow agent drift for changes across time rather than stretching one test case into a complete production-monitoring guide.

Operational AI agent evaluation ends with evidence for a decision, not a guarantee about every future input.

Use these questions to check whether your pass rule means what you think it means. Each answer depends on the declared task and test conditions.

Choose a failure your team has already seen, then sanitize it into a controlled fixture. Write the evidence, permitted calls, forbidden effects, and reviewer before changing the prompt.

Your next version should face the question the last one escaped.