Evaluating and Observing AI Agents in Production

A benchmark score, a judge score and a green dashboard are three different instruments, each with a documented gap between what it reads and what the business needs. METR asked four open-source maintainers to review 296 AI-generated patches. On METR's normalised scale, the grader's scores sat about 24 percentage points above the maintainers' merge rates ^[1]. On Terminal-Bench 2.0, infrastructure configuration alone moved results by 6 points ^[2].

Our thesis: evaluating an agent is a measurement programme, not a test you pass once, and the measurement needs validating too. Stack cheap deterministic checks under calibrated judges under human review, trace every run, and turn every production failure into a regression test.

Key figures

• ~24 pp

  grader pass rate above maintainer merge rate (METR, normalised)

  Source[1]
• 6 pp

  Terminal-Bench 2.0 swing from infrastructure setup alone

  Source[2]
• 50 vs 8 min

  Claude Sonnet 4.5 time horizon: grader vs maintainers (extrapolated)

  Source[1]

The METR study has limits. Only grader-passing patches went to review. It used one benchmark subset, one agent harness, five models (four of them Anthropic's), 3 of 12 repositories and static review without CI, and it makes no claim of a fundamental capability ceiling. The unnormalised gap is larger, at 34.9 points. The time-horizon estimates are extrapolated because the task-duration range is narrow ^[1]. A grader pass is not a merge decision, but the study does not show that benchmarks are useless.

Agents break the single-answer evaluation playbook

An agent takes many steps, calls tools and changes an environment, so a single output is the wrong unit of measurement. Anthropic's vocabulary helps: a task has defined inputs and success criteria, a trial is one attempt, the transcript is its full record, and the outcome is the final environment state, such as whether a reservation exists in the database. Evaluating an agent means evaluating the harness and the model together ^[3].

The metric that matters most for unattended work is pass^k, the probability that all k trials succeed. It falls as k rises; pass@k (at least one success in k attempts) rises. At 75% per-trial success, pass^3 is about 42% ^[3]. tau-bench made the same point in 2024: strong function-calling agents succeeded on fewer than 50% of tasks, and pass^8 was below 25% in retail ^[4]. tau2-Bench then reported significant drops when the user also acts on the shared environment ^[5].

Why one successful run proves little

NKK framework Illustrative

• pass^k at 75% per-trial success

Data for line chart, in % all k succeed

Period	pass^k at 75% per-trial success
k=1	75
k=2	56.3
k=3	42.2
k=4	31.6
k=5	23.7
k=6	17.8
k=7	13.3
k=8	10

In a survey of 86 practitioners plus 20 case studies, reliability was the top development challenge, about 74% relied mainly on human evaluation, and about 68% of production agents ran at most 10 steps before a human intervened ^[6]. The sample is not random, so read it as early practice.

What benchmarks can and cannot tell you

Benchmarks screen models; they do not forecast your deployment. Four failure sources recur.

Compressed scores. At release, the best model resolved 1.96% of SWE-bench issues ^[7]. The 2026 AI Index reports top SWE-bench Verified performance rising from about 60% to close to 100% of the human baseline between 2024 and 2025, yet still describes models as below that baseline. Its February 2026 leaderboard shows top models clustered between roughly 70% and 77% resolved ^[8].

Grader validity. An arXiv study of agentic benchmarks names SWE-bench Verified (insufficient test cases) and tau-bench (it counts empty responses as successful), and reports that such flaws can misestimate performance by up to 100% in relative terms ^[9]. METR's merge gap is a related problem ^[1]; that link is NKK's reading, not a claim either source makes.

Exploitability. UC Berkeley researchers report exploits that score at or near the maximum on SWE-bench Verified, WebArena and others without solving tasks, and call for isolating the agent from the evaluator ^[10]. That is a research-group blog post with an arXiv preprint; it does not state peer review. The Holistic Agent Leaderboard authors' LLM-aided log inspection found agents, in some cases, searching for the benchmark on HuggingFace instead of solving the task ^[11]. METR found an earlier version of o3 reward-hacked in 30.4% of RE-Bench runs (39 of 128), with hacks found in three of the RE-Bench task families. In a prompt test on one task, METR described the effect of instructions not to cheat as nearly negligible (20 runs per prompt) ^[12]. SWE-Bench Pro's authors describe it as contamination-resistant because its held-out and commercial sets are not public ^[13]; that is a design claim, not independent evidence.

Environment noise. Anthropic found infrastructure error rates fell from 5.8% to 0.5% as resource limits were lifted, and advises scepticism about leaderboard differences under 3 points until configurations are matched ^[2].

Time-horizon charts deserve the same care. METR says its measure is not the length of time an AI can work independently ^[14]. The 50% horizon is the human-expert task length at which METR predicts the model succeeds half the time, and measurements above 16 hours are unreliable on the current suite ^[15]. Error bars have been about a factor of 2 each way, and some reliability-critical tasks need 98% or higher success to be worth automating, so 50% is not a delegation threshold ^[14]. METR's Time Horizon 1.1 gives a post-2023 doubling time of about 131 days; full-period figures use a stitched trend that borrows earlier estimates ^[16]. Kapoor et al. argue for considering cost and accuracy jointly ^[17].

Reading the benchmarks you will be shown

Comparison

Criterion	What it measures	Documented limit	How to use it
SWE-bench Verified	Whether a patch passes tests on real GitHub issues	Top scores clustered in the 70s%; grader passes exceed maintainer merges; test cases insufficient	Regression smoke test only, never a deployment forecast
SWE-Bench Pro	Longer tasks across public, held-out and commercial repository sets	Authors claim contamination resistance; not independently verified here	Screening for harder coding work, with your own held-out tasks
tau-bench / tau2-Bench	Tool-agent-user dialogue with policy rules; reliability via pass^k	Empty responses counted as successes in tau-bench; drops under dual control	Use pass^k, and borrow the structure for your own domain simulations
METR time horizon	Human task duration at which a model succeeds 50% of the time	Not an autonomy duration; wide error bars; unreliable above 16 hours	Trend context for planning, not a go-live criterion

SWE-bench Verified

What it measures

Whether a patch passes tests on real GitHub issues

Documented limit

Top scores clustered in the 70s%; grader passes exceed maintainer merges; test cases insufficient

How to use it

Regression smoke test only, never a deployment forecast

SWE-Bench Pro

What it measures

Longer tasks across public, held-out and commercial repository sets

Documented limit

Authors claim contamination resistance; not independently verified here

How to use it

Screening for harder coding work, with your own held-out tasks

tau-bench / tau2-Bench

What it measures

Tool-agent-user dialogue with policy rules; reliability via pass^k

Documented limit

Empty responses counted as successes in tau-bench; drops under dual control

How to use it

Use pass^k, and borrow the structure for your own domain simulations

METR time horizon

What it measures

Human task duration at which a model succeeds 50% of the time

Documented limit

Not an autonomy duration; wide error bars; unreliable above 16 hours

How to use it

Trend context for planning, not a go-live criterion

LLM judges are useful instruments that need calibration

GPT-4-class judges reached over 80% agreement with human preferences on open-ended chat answers, the same level as agreement between humans ^[18], but that finding is about pairwise preference between chat replies, not about spotting defects in an agent's multi-step trajectory.

The evidence for the harder task is weaker. LLM-generated evaluators inherit the problems of the models they judge and require human validation ^[19]. Position bias is systematic: across 15 judges and more than 150,000 instances, it varied by judge and task and was strongly affected by the quality gap between candidates ^[20]. In AgentRewardBench, 12 LLM judges scored 1,302 expert-reviewed web-agent trajectories, and no single judge performed best across all benchmarks ^[21]. The Agent-as-a-Judge authors claim parity with a human baseline on DevAI, an AI-development-task benchmark, which is their own result on one benchmark ^[22].

One workshop paper, with a single author and a synthetic testbed, illustrates a subtler failure. A 14B judge shown only the request and final reply reached 34% to 76% recall on four fault types that leave the reply unchanged, but it flagged the clean runs just as often, so its paired discrimination (flag rate on fault-injected runs minus flag rate on the clean runs they came from) was zero. Told to check each step, it flagged every such fault and none of 100 clean runs ^[23]. Do not generalise the numbers; checking a classifier against clean runs is standard practice.

Evidence on judges missing defects in live production agents is thin. Our position: judges are scalable triage, calibrated per task against human labels, not a validated sole release gate.

Is your judge ready to gate a release?

Self-assessment

0 of 4 answered Reset

Human agreement

Is the judge compared with human labels on this task?

1. No comparison2. Labelled sample with agreement reported per criterion3. Rotating human sample, agreement tracked over time

False-alarm baseline

Do you know how often it flags clean runs?

1. Not measured2. Measured once3. Reported next to recall every release

Bias tests

Have you tested position and length effects?

1. No2. Swapped candidate order once3. Order and length tested per judge and per release

Visibility

What does the judge see?

1. Final reply only2. Reply plus tool results3. Every step

Answer all 4 dimensions to see your score and where to start.

Tracing is the substrate

Every layer of evaluation depends on traces. OpenAI defines a trace as the end-to-end record of model calls, tool calls, guardrails and handoffs for one run, and recommends starting with individual traces before datasets and eval runs ^[24]. Trace grading assigns structured scores or labels to a trace to show why an agent succeeded or failed, with more detail than black-box evals provide ^[25].

OpenTelemetry is standardising the shape of that data. Its GenAI conventions define an execute-tool span named execute_tool {gen_ai.tool.name} ^[26], operation names including create_agent, invoke_agent, invoke_workflow, plan and execute_tool ^[27], an evaluation-result event carrying a name and, where applicable, a score and label, with a recommended explanation, ideally parented to the span being evaluated ^[28], and duration metrics for agents and tools ^[29]. Two cautions apply. These pages carried the status Development on 2026-10-13, so pin versions and expect renaming ^[26]. And content is sensitive: instructions, user messages and outputs should not be captured by default, and tool arguments and results are opt-in attributes ^[26].

Span types in one agent run (simplified)

Process

1. 1
   Step 1: invoke_agent

   Root span; all other spans nest under it.

2. 2
   Step 2: plan

   Planning or task decomposition step.

3. 3
   Step 3: chat (model call)

   Inference span with token and latency metrics; message content is opt-in and may contain sensitive information.

4. 4
   Step 4: execute_tool

   One span per tool call, named with the tool; arguments are opt-in and may be sensitive.

5. 5
   Step 5: evaluation result event

   A judge or human score written back onto the evaluated span, so scores and traces live together.

Anthropic's 2025 postmortem of three infrastructure bugs shows why this matters: standard benchmarks and safety evaluations missed degradation users were reporting, while privacy controls limited when engineers could read user interactions. Its fix list included continuous quality evaluations on live production systems ^[30]. AWS's evaluation service distinguishes online evaluation on live traffic, with percentage-based session sampling, from on-demand modes ^[31].

Practice lags the tooling. LangChain ran a public, vendor-run survey with 1,340 responses (fieldwork 18 November to 2 December 2025). Just over half of organisations (52.4%) reported offline evals, but only 37.3% reported online evals, and the report implies that the share not evaluating fell from 29.5% overall to 22.8% for respondents with agents in production, though it does not label that base explicitly ^[32]. Its human-review and LLM-as-judge shares lack a stated base, so they are omitted. This is directional evidence only.

Labelling traces needs a vocabulary. The MAST taxonomy identifies 14 failure modes in three categories (system design, inter-agent misalignment, task verification), and its authors release a dataset of more than 1,600 annotated traces across 7 multi-agent frameworks ^[33]. Automated trace debugging was immature in 2025 testing: on TRAIL's 148 human-annotated traces, the best model reached 11% joint accuracy on error category and location, though the authors are associated with an evaluation vendor ^[34].

NKK model: the Agent Evidence Pyramid and the Trace-to-Test Loop

This is NKK's framework, built on three pieces of prior art: Anthropic's layered "Swiss cheese" view of evals, monitoring and review ^[3], Microsoft's split between system evaluation (end-to-end outcomes) and process evaluation (each step) ^[35], and OpenAI's progression from traces to datasets ^[24].

The Agent Evidence Pyramid has five layers of evidence, cheapest and most frequent at the base. Layers are ordered by when evidence is gathered (pre-merge to live) and by how much human judgement it holds; L3 and L4 apply the same kind of judge before release and live, and L5 also feeds calibration labels back down. To apply it, assign each claim you plan to make ("it books correctly") to the lowest layer that can falsify it, and escalate only when the cheaper layer has a documented blind spot.

NKK Agent Evidence Pyramid

NKK framework

1. L5 Human review and incident analysis

   Expert reading of sampled and flagged transcripts; failure labelling; the labels that calibrate L3 and L4.

   Weekly and per incident
2. L4 Online sampled scoring and guardrail telemetry

   Judges score a sample of live traces; guardrail trips, escalations and overrides tracked; scores written back as evaluation events.

   Continuous
3. L3 Trajectory and process judges

   Model-based graders for tool selection and step faithfulness, calibrated against human labels and clean-run false-alarm rates.

   Per release, on a sample
4. L2 Outcome graders with repeated trials

   Check environment state, not the reply; N trials per task; report pass^k, not a single run.

   Every merge, in CI
5. L1 Deterministic checks

   Schema validity, tool-argument validity, state assertions and policy rules in code.

   Every commit

What each layer catches and misses

NKK framework

Criterion	Catches	Blind spot	Typical owner
L5 Human review	Novel failures, unclear criteria, harms no metric names	Slow, costly, reviewers may disagree; privacy limits on reading transcripts	Domain owner with a trained reviewer pool
L4 Online scoring	Drift and degradation on real traffic	Shares judge biases with L3; sampling can miss rare failures	Platform or ML engineering
L3 Process judges	Wrong tool choice and unfaithful steps	Position bias; outcome-only views miss faults that leave the reply unchanged	Evaluation engineer
L2 Outcome graders	Wrong end state, unreliable behaviour across trials	Needs ground truth many business tasks lack; graders can be exploited or have too few tests	Agent team
L1 Deterministic checks	Malformed calls, policy violations, regressions	Brittle to valid variation; blind to meaning	Agent team

L5 Human review

Catches

Novel failures, unclear criteria, harms no metric names

Blind spot

Slow, costly, reviewers may disagree; privacy limits on reading transcripts

Typical owner

Domain owner with a trained reviewer pool

L4 Online scoring

Catches

Drift and degradation on real traffic

Blind spot

Shares judge biases with L3; sampling can miss rare failures

Typical owner

Platform or ML engineering

L3 Process judges

Catches

Wrong tool choice and unfaithful steps

Blind spot

Position bias; outcome-only views miss faults that leave the reply unchanged

Typical owner

Evaluation engineer

L2 Outcome graders

Catches

Wrong end state, unreliable behaviour across trials

Blind spot

Needs ground truth many business tasks lack; graders can be exploited or have too few tests

Typical owner

Agent team

L1 Deterministic checks

Catches

Malformed calls, policy violations, regressions

Blind spot

Brittle to valid variation; blind to meaning

Typical owner

Agent team

The pyramid breaks in three places. Judges at L3 and L4 share biases. L1 and L2 need ground truth many business tasks lack. Sampling rates and costs are use-case specific; NKK has not validated any default. It also says nothing about statistical power; report error bars ^[36]. As a usage rule, label any release claim that rests only on public benchmark scores or live-traffic judge scores as such in the release note ^[1][9].

The Trace-to-Test Loop keeps the pyramid honest by routing production failures back into the regression suite. This echoes the idea that high-scoring capability evals graduate into regression suites ^[3]. Run it weekly and track time from incident to regression test. It breaks when privacy rules block reading transcripts, when labelling capacity runs out, and when suites overfit to past failures.

NKK Trace-to-Test Loop

NKK framework

1. 1

   1 Capture

   Trace every run; content capture opt-in with redaction.

2. 2

   2 Sample

   Random, risk-weighted and low-score traces.

3. 3

   3 Label

   Humans tag failures with a MAST-style taxonomy.

4. 4

   4 Calibrate

   Compare judge with labels; report false alarms on clean runs.

5. 5

   5 Promote

   Turn each confirmed failure into a regression task with several trials.

6. 6

   6 Gate

   Release on pass^k and guardrail metrics.

7. 7

   7 Monitor

   Watch drift in production, then return to capture.

Guardrail, security and monitoring metrics

Guardrails, oversight and attack signals

Guardrails are measured systems too. OpenAI's Agents SDK places them at the input, output and tool-call level; a failure sets a tripwire that stops execution. Input guardrails run in parallel by default, so the agent may already have used tokens or tools before the tripwire fires ^[37]. NKK proposes recording trips by placement, false blocks, escalations, overrides and irreversible actions, each checked against a human-labelled sample.

Human oversight is itself a signal. In Claude Code, full auto-approve covered about 20% of sessions for new users and over 40% for experienced users, while interruptions rose from about 5% to 9% of turns; 0.8% of public API tool-call actions appeared irreversible ^[38]. That is vendor self-report on one product, with Claude-generated irreversibility labels. Anthropic concludes that effective oversight will require new post-deployment monitoring infrastructure.

Single-attempt tests understate risk, which argues for repeated-trial adversarial tests. In CAISI's AgentDojo-based hijacking evaluation, average attack success across five injection tasks was 57%, rising to 80% with 25 attempts per attack, against agents powered by the upgraded Claude 3.5 Sonnet ^[39]. In its red-teaming competition (13 frontier models, over 250,000 attempts), every target had at least one successful attack ^[40].

Monitoring and regulatory scope

NIST monitoring categories mapped to agent metrics

NKK framework

Criterion	Example agent metrics (NKK proposal)	Typical source of signal (NKK proposal)
Functionality	pass^k on a fixed regression set; tool-call success; cost per successful task	Repeated-trial eval runs
Operational	Latency by span; retries; tool error rate	Trace and metric data
Human factors	Escalation, override and interruption rates	Session logs and reviewer records
Security	Attack success under repeated attempts; tripwire rate by placement	Red-team runs and guardrail telemetry
Compliance	Log completeness; evidence retained for audit	Retained logs and audit sampling
Large-scale impacts	Irreversible-action share; incident count and severity	Tool-call classification and incident records

Functionality

Example agent metrics (NKK proposal)

pass^k on a fixed regression set; tool-call success; cost per successful task

Typical source of signal (NKK proposal)

Repeated-trial eval runs

Operational

Example agent metrics (NKK proposal)

Latency by span; retries; tool error rate

Typical source of signal (NKK proposal)

Trace and metric data

Human factors

Example agent metrics (NKK proposal)

Escalation, override and interruption rates

Typical source of signal (NKK proposal)

Session logs and reviewer records

Security

Example agent metrics (NKK proposal)

Attack success under repeated attempts; tripwire rate by placement

Typical source of signal (NKK proposal)

Red-team runs and guardrail telemetry

Compliance

Example agent metrics (NKK proposal)

Log completeness; evidence retained for audit

Typical source of signal (NKK proposal)

Retained logs and audit sampling

Large-scale impacts

Example agent metrics (NKK proposal)

Irreversible-action share; incident count and severity

Typical source of signal (NKK proposal)

Tool-call classification and incident records

NIST's report on monitoring deployed AI names fragmented logging, no trusted monitoring standards, and integrating automated with human-validated monitoring as open problems ^[41]. Three EU AI Act obligations apply where an agent is a high-risk system: automatic event logs (Article 12) ^[42], post-market monitoring (Article 72) ^[43], and human oversight with a stop capability (Article 14) ^[44]. Per the consolidated text as at 27 July 2026, as amended by the Digital Omnibus on AI, high-risk obligations apply from 2 December 2027 for Annex III systems. For Annex I systems the date is 2 August 2028 ^[45].

What to do next

Each step is tagged with the layer or loop stage it builds. Steps 1 to 3 are NKK's suggested first 30 days.

1. Define the outcome (L1, L2). Write success as an environment state your code can check, agreed with the business owner.
2. Run trials, not single runs (L2). Report pass^k for each release and cost per successful task.
3. Validate every judge (L3). Compare it with human labels on your task and report its false-alarm rate on clean runs before it gates anything.
4. Instrument with OpenTelemetry and pin versions (Capture). Keep content capture opt-in, with a redaction policy agreed with privacy.
5. Sample production (L4). Start with a small percentage of sessions plus every guardrail trip and escalation.
6. Wire the human loop (L5). Review weekly, label failures with a taxonomy, and promote each confirmed failure to a regression task.
7. Check regulatory scope. If the use case is high-risk, map logging, monitoring and oversight evidence to Articles 12, 72 and 14.

Conclusion

Every benchmark, judge and dashboard is an instrument with an error profile that is rarely reported. Treat evaluation as a continuing programme with its own error bars: in the words of one statistical treatment, evaluations are experiments ^[36]. Report pass^k, validate every judge against clean runs, and turn every incident into a test.