agent 评测Evaluating agents

agent 评测就是用数据集和评分器量化「改完是变好还是变坏」。 没有它，「优化提示词」只是手感；有了它，每次改动才有账可查。 评测分两个战场：上线前测回归，上线后盯真实流量。

Agent evaluation means quantifying "did the change make it better or worse" with datasets and graders. Without it, "optimizing the prompt" is just feel; with it, every change has an auditable ledger. Evaluation has two battlefields: regression testing before shipping, watching real traffic after.

LangSmith 官方文档把评测按运行时机分成两类。

离线评测：在开发时用整理好的数据集跑， 目的是比较版本、基准测试、抓回归。 官方一句话："Test before you ship"。

在线评测：对生产环境的真实交互实时评分， 做安全筛查、格式校验、质量启发式检测。 官方一句话："Monitor in production"。

两者的数据是通着的：线上看到的失败， 应该能沉淀成数据集里的一条用例。 LangSmith 官方原文："Add failing production traces to your dataset, create targeted evaluators, validate fixes with offline experiments, and redeploy." 这个回环是评测体系能长期活着的关键。

LangSmith's official docs split evaluation by when it runs.

Offline evaluation: run against curated datasets during development, to compare versions, benchmark, and catch regressions. Official one-liner: "Test before you ship".

Online evaluation: score real production interactions in real time, for safety screening, format validation, quality heuristics. Official one-liner: "Monitor in production".

The two share data: failures seen in production should become a test case in the dataset. LangSmith official text: "Add failing production traces to your dataset, create targeted evaluators, validate fixes with offline experiments, and redeploy." This loop is what keeps an evaluation system alive long-term.

官方列出四类评分方式：

代码规则最便宜也最可信——能算出来就别问模型。 LLM 当裁判放主观维度，但裁判自己也会错， 它的评分标准要写成可复用的指令，不能每次现编。

The official docs list four kinds of graders:

Code rules are the cheapest and most trustworthy — if it can be computed, don't ask the model. LLM-as-judge goes on subjective dimensions, but the judge itself errs; its rubric must be written as reusable instructions, not improvised each time.

官方给出的四步：

1. 建数据集：手工用例、历史 trace、或合成生成。 2. 定评分器：按上面四类选。 3. 跑实验：整个应用在数据集上跑一遍，可配置重复次数和并发。 4. 分析结果：比较实验，做基准、单测、回归、回测。

agent 的评测比单轮问答多一层：过程也要查。 最终答案对了，中间是不是调错了工具、绕了十圈，成本完全不同。 这也是 trace（轨迹记录）在 agent 评测里必不可少的原因。

The official four steps:

1. Build the dataset: handcrafted cases, historical traces, or synthetic generation. 2. Define graders: pick from the four kinds above. 3. Run experiments: run the whole application over the dataset, configurable repeats and concurrency. 4. Analyze results: compare experiments — benchmarks, unit tests, regressions, backtests.

Evaluating an agent adds one layer over single-turn QA: the process must be checked too. The final answer may be right, but if it called the wrong tools or looped ten times, the cost is completely different. This is why traces are indispensable in agent evaluation.

1. 数据集漂移：用例停留在三个月前的场景，线上早就没人那么问了。 2. 裁判偏见：LLM 偏好长答案、偏好自己熟悉的格式， 用 LLM 评 LLM 要定期和人工抽检对齐。 3. 过拟合数据集：为刷分优化，泛化反而降——留一份从没用来调过参数的数据。 4. 只看通过率：80% 到 85% 可能全挤在简单用例上，要看分类目、分难度的细分。

Reflexion 论文展示的是另一个方向：用反馈信号本身当评测。 它的 verbal reinforcement 把环境反馈转成语言反思存起来， 本质上是被动反馈驱动、自我执行的评测循环。 思路可借，但报告的数字（91% pass@1）是特定基准上的结果， 换任务不保证复现。

Anthropic 官方原文："optimizing single LLM calls with retrieval and in-context examples is usually enough"—— 先确认简单方案真的到顶了，再上复杂评测基建。

1. Dataset drift: cases frozen in three-month-old scenarios while production long ago stopped asking those. 2. Judge bias: LLMs prefer long answers and formats they know; LLM-grading-LLM needs periodic alignment against human spot checks. 3. Overfitting the dataset: optimizing for the score, generalization drops — keep a slice never used for tuning. 4. Watching only pass rate: 80% to 85% may all sit on easy cases; look at per-category, per-difficulty breakdowns.

The Reflexion paper points another direction: use the feedback signal itself as evaluation. Its verbal reinforcement turns environment feedback into verbal reflections that are stored — essentially a passive-feedback-driven, self-executing evaluation loop. The idea is borrowable, but the reported number (91% pass@1) is a result on a specific benchmark; transfer to other tasks is not guaranteed.

Anthropic official text: "optimizing single LLM calls with retrieval and in-context examples is usually enough" — first confirm the simple approach has truly hit its ceiling, then build complex evaluation infrastructure.