怎么搭一条最小评测流程How to build a minimal evaluation pipeline

最小评测流程就是一条能重复跑的对比通道。 它的产物不是分数，是「这次改动比上次好还是差」的判断依据。 本篇只搭五个部件：样本集、跑批、判据、记录、对比。 先让它跑起来，再谈平台化。

A minimal evaluation pipeline is one repeatable channel for comparison. Its output is not a score; it is the basis for saying whether this change beat the last one. This guide builds five parts only: sample set, batch run, graders, records and comparison. Get it running first; talk about a platform later.

• 你手上有一个能跑的 agent 或提示词，改过几次但说不清好坏。
• 团队里出现「我觉得这版更好」的争论，需要一份共同看的记录。
• 线上已经开始有失败样本可以回收。
• 没有预算建标注平台，也不想先接一整套观测系统。

不适用：任务本身没有可判定的输出（纯开放式头脑风暴）。 那种场景先重新定义任务，再来搭流程。

• You have a working agent or prompt, edited several times, and cannot say whether it improved.
• Your team argues "I think this version is better" and needs one record everyone reads.
• Production is already producing failure samples you can harvest.
• There is no budget for an annotation platform and no appetite for a full observability stack.

Not applicable: the task has no judgeable output (open-ended brainstorming). Redefine the task before building a pipeline for it.

1. 一个可重复调用的入口：同一段输入两次跑，路径不会随机改。
2. 至少 20 条真实样本（示例阈值，按你的任务复杂度上调）。
3. 每条样本写清「什么算通过」，写不出就先把这条删掉。
4. 一份能记下每次运行的地方：文件、表格或平台的实验记录都行。
5. 明确不测什么，写进说明，避免结果被过度解读。

1. A repeatable entry point: the same input twice must not take a randomly different path.
2. At least 20 real samples (an example threshold — raise it for harder tasks).
3. Each sample states what counts as passing; if you cannot write that, drop the sample.
4. Somewhere to record every run: files, a spreadsheet, or experiment records in a platform.
5. A written statement of what is not tested, so results are not over-read.

判据的顺序值得单独说。能用相等、包含、正则、JSON 字段判的，别交给模型。 promptfoo 官方文档把断言分成这些类型，另含相似度与模型评分两类。 模型判的部分要固定提示词与输出格式，否则同一批样本两次跑会漂。

LangSmith 官方文档把评测按运行时机分两类： 开发时在整理好的数据集上跑，生产里对真实交互实时评分。 官方原文（英文）："Run evaluations on curated datasets during development."

The ordering of graders deserves its own note. Anything equality, containment, regex or a JSON field can decide should not be asked of a model. The promptfoo documentation groups assertions into those types and adds similarity and model-graded checks. For model graders, pin the prompt and the output format, or the same samples drift between runs.

The LangSmith documentation splits evaluation by when it runs: curated datasets during development, live scoring of real production interactions. Official text: "Run evaluations on curated datasets during development."

1. 收样本：从最近的失败与常见的成功里各取一半。
2. 写判据：每条先写一条规则判据；实在写不出的标为「需人看」。
3. 定跑批：固定模型、版本、参数与工具权限，把它们记在运行元数据里。
4. 跑基线：先用当前版本跑一遍，得到一份可复现的起点。
5. 做改动：一次只改一个变量，否则差异无法归因。
6. 复盘：把不一致的样本人工看一遍，决定是判据错还是应用错。

第 4 步是很多人跳过的一步。没有基线，后面的数字没有参照物。 轨迹记录值得留：最终答案对了，中间绕了几步、调错没调错工具，成本差别很大。 OpenAI Agents SDK 官方文档（tracing 页）写明它捕获模型生成、工具调用与 agent 运行。

1. Collect samples: half from recent failures, half from typical successes.
2. Write graders: one rule grader per sample first; mark what truly resists rules as "needs a human".
3. Fix the run: pin model, version, parameters and tool permissions, and store them with the run metadata.
4. Measure a baseline: run the current version once so later numbers have a reference.
5. Change one variable at a time, otherwise the delta has no attribution.
6. Review disagreements: read the mismatching samples by hand and decide whether the grader or the application is wrong.

Step 4 is the one people skip. Without a baseline the later numbers have nothing to compare against. Keep traces: a correct final answer that took extra steps or called the wrong tool costs differently. The OpenAI Agents SDK tracing documentation states that traces capture LLM generations, tool calls and agent runs.

数字与情境均为示例，是虚构假设，不是任何产品的实测结果。

• 输入：18 条「按合同条款回答客户问题」的样本，每条带一个必须出现的条款编号。
• 处理：跑批用同一模型与同一检索配置；判据两条—— 规则判「输出里是否出现指定条款编号」，模型判「解释是否与条款一致」。
• 结果判断（示例）：规则判通过 15/18，模型判通过 11/18。 差异不在应用，而在判据：规则判只查编号，答案里编号正确但解释错也照样通过。
• 失败后下一步： 给规则判据补一条「不得出现禁止表述」，并把 3 条分歧样本人工复核； 复核确认 2 条是模型判据提示词含糊，改写判据描述后重跑同一批样本。
• 记录：基线 15/18、改动后 17/18，两个数字都标注样本集版本与判据版本。

一次通过率的变动，只有在样本集与判据都不变时才可比。 MLflow 官方文档也提醒：经典评测接口与生成式评测接口两套系统不互通， 官方原文（英文）："These two systems are not interoperable."

The figures and the scenario are illustrative — invented assumptions, not measured results for any product.

• Input: 18 samples of "answer the customer from the contract clause", each requiring one clause number to appear.
• Processing: one run uses the same model and the same retrieval configuration; two graders — a rule checking whether the required clause number appears, and a model judging whether the explanation matches the clause.
• Judging the result (example): the rule passes 15 of 18, the model passes 11 of 18. The gap is not the application but the grader: the rule only sees the number, so a right number with a wrong explanation passes.
• What to do after a failure: Add a forbidden-phrasing condition to the rule, and have a human review the 3 disagreeing samples; two of them trace back to a vague model-grader prompt — rewrite the criterion and rerun the same set.
• Records: baseline 15 of 18, after the change 17 of 18, both annotated with sample-set version and grader version.

A pass-rate delta is comparable only when the sample set and the graders stay fixed. The MLflow documentation also warns that its classic and generative evaluation interfaces are separate systems. Official text: "These two systems are not interoperable."

• [ ] 换一台机器或换一个时间重跑，同一样本集结果差异可解释
• [ ] 每条样本能说出通过条件与不通过条件
• [ ] 运行记录含模型版本、参数、工具权限与样本集版本
• [ ] 判据本身有人工复核样本，知道它错在哪
• [ ] 能一键只跑失败子集，用于快速验证修复

• [ ] Rerunning on another machine or at another time gives differences you can explain
• [ ] Every sample has both a pass condition and a fail condition written down
• [ ] Run records include model version, parameters, tool permissions and sample-set version
• [ ] The graders themselves have human-reviewed samples, so you know how they fail
• [ ] You can rerun only the failing subset with one command

• 分数忽高忽低：先查是不是没固定版本与参数，再查判据提示词是否被顺手改过。
• 全通过或全失败：判据大概率写错，挑一条样本人工看输出。
• 跑批中途报错（401 / 403）：凭据或权限与运行环境不一致，别把它当成模型质量问题。
• 批量请求被限流（429）：降并发、加退避；把重试次数记进运行元数据。
• 检索类任务分数异常：先看召回片段是否命中，再调生成端（这条顺序能省很多时间）。
• 引用类任务判不准：逐条把说法对应到证据，判据要检查对应关系而非只看有没有链接。
• 输出是 JSON 但字段缺失：解析成功不等于字段正确，判据要按 schema 校验。
• 样本不够：不要合成一堆相似问法充数，先把线上失败样本回收进集子。

• Scores swing up and down: check unpinned versions or parameters first, then whether a grader prompt was quietly edited.
• Everything passes or everything fails: the grader is probably wrong — read one sample's output by hand.
• The run errors halfway (401 / 403): credentials or permissions differ between environments; do not file this as a quality problem.
• Batch requests hit rate limits (429): lower concurrency, add backoff, and record retry counts in the run metadata.
• Retrieval scores look broken: inspect whether the recalled passages match before tuning generation — that order saves time.
• Cited answers are hard to judge: match each claim to its evidence, and have the grader check the mapping rather than the presence of a link.
• JSON parses but fields are missing: parsing is not schema validation; grade against the schema.
• Too few samples: do not pad with near-duplicate phrasings; harvest real production failures into the set instead.

1. 把这条流程接进发布检查，让不达标自动卡住（见对应指南）。
2. 给判据留一份人工抽检记录，定期与人工结论对齐。
3. 线上跑起来后，加一层对真实交互的持续评分，别只靠离线集。
4. 需要对外解释能力时，把样本集版本与判据一起写进说明。

1. Wire this pipeline into the release check so a failing run blocks the release (see the matching guide).
2. Keep a human spot-check log for the graders and align it with human conclusions periodically.
3. Once live traffic runs, add continuous scoring on real interactions instead of relying only on the offline set.
4. When explaining capability to outsiders, publish the sample-set version together with the graders.

2026-11-06 直抓了五处官方来源：LangSmith 评测文档、 promptfoo 入门页与断言类型页、MLflow 评测文档、OpenAI Agents SDK tracing 页。 本页只转述这些页面写明的部件、分类与限制，未运行任何评测产品。

以下未知：

• 各产品在你数据规模下的跑批耗时与并发上限——官方页未给出可比口径。
• 模型判据在具体任务上与人工结论的一致率——需要你自己的复核样本才能估。
• MLflow 两套评测接口在你所用版本里的迁移代价——未核对该版本发布说明。

On 2026-11-06 five official sources were fetched directly: the LangSmith evaluation page, the promptfoo intro and assertion pages, the MLflow evaluation page, and the OpenAI Agents SDK tracing page. This guide only reports the parts, categories and limits those pages state; no evaluation product was executed.

Unknowns:

• Batch runtime and concurrency ceilings at your data volume — the official pages give no comparable measurement basis.
• Agreement rate between model graders and human judgement for a specific task — this needs your own review samples.
• Migration cost between MLflow's two evaluation interfaces in the version you run — that release note was not checked.