AI Agent Evaluations: How to Test Delegated Work Before You Trust It
A practical guide to evaluating AI agents with task suites, logs, rubrics, permissions, regression checks, human review, and realistic failure cases.

An AI agent can look impressive during a live task and still be unready for responsibility. It may solve the example you gave it, then fail on a slightly messier version. It may act confidently with missing context. It may use the wrong tool, skip a verification step, leak private information into a place it does not belong, or produce work that seems correct until a human checks the details. Agent evaluations exist because “it worked once” is not enough.
Testing an agent involves more than judging its final response. An agent has a trajectory. It interprets a goal, gathers context, chooses tools, handles errors, asks or does not ask for permission, updates files or records, and decides when it is done. The final answer matters, but the path matters too. A lucky final answer produced through unsafe steps is not a pass.

Start with the job, not the model
The first evaluation question is not “How smart is this agent?” It is “What job are we asking it to do?” A research agent, coding agent, customer-support agent, finance operations agent, calendar agent, and browser agent need different tests. Each job has its own context, tools, risks, and success criteria. A generic intelligence score will not tell you whether the agent can safely update a CRM record, triage a bug, draft a refund response, or reconcile a messy spreadsheet.
Write the job as a set of realistic tasks. Include easy cases, normal cases, annoying cases, and edge cases. The easy cases prove the harness works. The normal cases measure expected use. The annoying cases expose ambiguity, missing data, stale context, weird formatting, conflicting instructions, and tool errors. The edge cases test boundaries: private data, irreversible actions, payment movement, destructive edits, and requests that should trigger refusal or escalation.
Good evaluation tasks are specific enough to judge but not so artificial that they flatter the agent. If every input is clean and every tool behaves perfectly, you are testing a demo, not a deployment. The test fixtures guide explains how to reproduce that mess in a controlled environment.
For a small first suite, choose one workflow and write cases with an input, the available tools and permissions, the starting state, and observable pass and fail conditions. For example, a support agent receives a synthetic refund request with two conflicting policy versions. Give it a fake order record and a simulated refund tool. A pass requires identifying the conflict, citing the relevant policies, and escalating without issuing a refund. A polished reply that conceals the conflict or calls the refund tool fails. Reset the fake record before each attempt so one run cannot change the next. Pair it with an ordinary request where the policy is clear and an authorized refund is appropriate; otherwise, a system that always escalates could appear to be doing well. Have a reviewer walk through a valid solution and check that the fixture and graders accept it before judging the agent.
Keep the refund tool simulated and record attempted calls as well as successful changes. A blocked unauthorized call is still useful evidence about the agent’s behavior, even though the control prevented a refund. In a separate integration test, check that the real authorization layer denies the same action without the required approval. A passing simulated case does not prove production permissions are enforced. OWASP’s guidance on excessive agency recommends limiting permissions and enforcing authorization in downstream systems.
Judge the trajectory
An agent evaluation should record what the agent did. Which context did it read? Which tool did it call? Did it ask for approval at the right moment? Did it inspect the result of its own action? Did it notice errors? Did it retry sensibly? Did it stop when the task was complete, or did it wander into unrelated changes? Did it explain uncertainty clearly?
Logs are not just debugging artifacts. They are evidence. A reviewer should be able to inspect the evidence and observable actions behind a decision, without assuming a trace reveals every internal reason. If the agent changed a file, the diff matters. If it sent a message, the recipient and content matter. If it used private data, the reason matters. If it failed, the failure mode matters more than a vague “agent was wrong.”
A final score without a trace can miss unsafe steps. Check the resulting state too: a message saying a record was updated does not prove the update occurred. Record tool calls and results, approvals, errors, and relevant state changes where available; a trace may still be incomplete. Minimize sensitive content in logs, restrict access, and set retention rules. The observability guide covers the evidence an operator needs.
Build rubrics around risk
Not every mistake has the same weight. A typo in a draft is not the same as deleting a customer record. A slow tool call is not the same as sending confidential information to the wrong place. A good rubric separates quality, completeness, efficiency, safety, permission handling, reversibility, and escalation.
For low-risk writing or research tasks, the main questions may be accuracy, usefulness, sourcing, and clarity. For coding tasks, the rubric should include whether the agent understood the repo, kept changes scoped, ran relevant tests, and avoided unrelated churn. For operations tasks, the rubric should include record accuracy, auditability, permission boundaries, and whether the agent knew when to ask a human. For tasks involving money, legal commitments, medical advice, personal data, or production systems, the agent may need to prepare work but not execute final action without approval.
The rubric should also reward good stops. An agent that says “I cannot complete this safely without X” may be behaving better than an agent that invents its way forward. Evaluations that reward only completion can favor bluffing.
Make critical boundaries explicit. In the refund example, an unauthorized refund is an automatic failure even if the rest of the response is excellent. Score useful but less critical qualities, such as clarity and review effort, separately. Allow more than one valid route to the outcome: checking for one exact sequence of tool calls can reject a safe alternative. The acceptance criteria guide helps turn expectations into observable checks.
Choose how to grade
Use deterministic checks for things that can be verified directly, such as whether a prohibited tool was called, a required field is present, or the final record has the right state. Human reviewers can judge whether an explanation is accurate and useful against a written rubric. A model-based grader can help review open-ended answers at scale, but it can miss errors or disagree with people. Compare its judgments with domain reviewers on a labeled sample, inspect disagreements, and recheck it when the rubric or workflow changes. Anthropic’s guide to agent evaluations describes these grading approaches and their tradeoffs.
Review failed cases and a sample of passes. A failed check may expose an agent defect, a broken fixture, or an ambiguous expectation; a passing grade may still hide an unsafe trajectory. If a tool log or state check is unavailable, mark the affected criterion unknown instead of assuming it passed. Record the resolution rather than silently changing the expected answer, and rerun affected cases after fixing the fixture or grader.
Include regressions and adversarial mess
Once an agent passes a task, keep that task. Agents, prompts, tools, memory systems, and permissions change over time. A workflow that worked last month can break after a model upgrade or tool change. Regression tests protect hard-won reliability.
Keep a development set for iteration and a separate holdout for less frequent assessment. If the team tunes the agent against the holdout repeatedly, it becomes another development set. Keep related variants together when splitting cases, and refresh the holdout when it has been exposed. The evaluation data stewardship guide covers provenance, leakage, and maintenance.
The suite should also include messy cases. Give the agent duplicate records. Give it a broken link. Give it a tool timeout. Give it an instruction that conflicts with policy. Give it incomplete context and see whether it asks a clarifying question or makes a dangerous assumption. Give it a task where the right answer is to do nothing. Real work contains friction. Evaluation should too.
This does not mean trying to trick the agent for sport. It means representing the environment honestly. If the agent will operate in a messy inbox, test messy emails. If it will edit a codebase with a dirty worktree, test that. If it will use a browser, test login walls, stale pages, redirects, and missing data. The goal is not humiliation. The goal is finding the edge before the user does.
Read the results in context
Run the same case more than once when behavior can vary, resetting the environment each time. Report the number of cases and attempts, passes and failures, and results for important risk categories rather than only an overall percentage. For example, 18 passes out of 20 is not reassuring about approval handling if the two failures are the only approval-boundary cases. For a single case run five times, report “four of five attempts passed,” alongside the failure and its cause; do not describe that as four different cases passing. Repeated attempts on the same fixture are not a substitute for coverage across different cases. A small, hand-picked suite can expose failures but cannot estimate a production failure rate on its own.
Record the model, instructions, tool and permission settings, fixture and rubric versions, and relevant environment settings with each run. Compare changes under the same conditions where possible, and investigate changes in the test setup before attributing a score difference to the agent. NIST’s AI risk management framework recommends documenting test sets, metrics, and evaluation conditions, and monitoring systems in production.
Measure human review load
An agent that technically completes tasks but requires exhausting review may not be useful. Evaluation should measure how hard it is for a human to trust the output. Does the agent provide evidence? Are changes easy to inspect? Does it summarize what it did without hiding uncertainty? Does it group approvals cleanly? Does it leave the workspace in a state a reviewer can understand?
This matters because human review is part of the system. If the agent saves ten minutes of execution and adds fifteen minutes of verification anxiety, the deployment is not working. The evaluation should ask how long review takes, what reviewers check, where they lose confidence, and which agent behaviors create unnecessary review burden.
A strong agent makes review smaller by being explicit. It names assumptions. It links evidence. It separates completed work from suggested next steps. It does not bury risky actions in a cheerful summary. It treats the reviewer as a partner, not an obstacle.
Promote slowly
The safest deployment path is a ladder. First the agent works on synthetic tasks. Then it works on real tasks in read-only mode. Then it drafts changes for human approval. Then it handles low-risk actions with logs. Then, maybe, it earns broader permission in a narrow domain. Each promotion should depend on evidence, not vibes. Set the decision criteria and responsible owner beforehand: for example, no observed unauthorized actions in the boundary suite, acceptable quality on ordinary cases, and a review process the team can sustain. A clean result in a finite suite is evidence for a decision, not proof that the risk is gone.
Evaluation continues after launch. Production logs reveal cases the test suite missed. Human overrides reveal unclear boundaries. Near misses are valuable. Failures should become new tests. The evaluation suite is not a gate you pass once. It is the memory of what you have learned.
AI agent evaluations are how delegated software becomes less theatrical and more trustworthy. They turn “the agent seemed good” into a clearer question: good at which job, under which conditions, with which tools, under which permissions, with what review burden, and with what failure behavior? If you cannot answer those questions, the agent may still be useful, but it has not earned much trust yet.
Written By
JJ Ben-Joseph
Founder and CEO · TensorSpace
Founder and CEO of TensorSpace and the editor of Fondsites: an engineer working across software, AI, cybersecurity, and biosecurity who builds the site's games and tools.