← Safety

核验标记表示资料完整度，不是产品评分。下列未确认项可能影响选型；请按对应来源逐项确认。The confidence mark describes evidence completeness, not a product score. Unconfirmed items below may affect your choice; check them against the relevant sources.

模型与开放条件Model & openness

要有自己的模型接入：README 写明运行评测需要先设置 OPENAI_API_KEY，官方提示按 API 计费。 docs/run-evals.md：默认支持 OpenAI API 上的任意模型，任何实现 CompletionFn 协议的对象都可被 oaieval 调用， 仓库 evals/registry/completion_fns 提供现成完成函数。OpenAI 之外托管服务的一等支持：官方未说明。

Your own model access is required: the README states that running evals needs an OPENAI_API_KEY and warns about API costs. docs/run-evals.md: by default any model on the OpenAI API is supported, any implementation of the CompletionFn protocol can be run by oaieval, and ready completion functions live in evals/registry/completion_fns. First-class support for other hosted providers: not stated officially.

运行位置Where it runs

本机 Python 进程，README 标 "Minimum Required Version: Python 3.9"；pip install evals 装运行环境， 自建评测建议克隆仓库后 pip install -e .。主入口是终端命令 oaieval（单评测）与 oaievalset（评测集），可脚本化进 CI。 README 首行另把托管用法指向 OpenAI Dashboard 的 Evals 文档；那条云线本批次未取到官方正文。

A local Python process; the README states "Minimum Required Version: Python 3.9". pip install evals to run evals, clone + pip install -e . to author them. The entry points are the terminal commands oaieval (single eval) and oaievalset (eval set), so it scripts into CI. The README's first line points hosted usage to the OpenAI Dashboard Evals docs; that cloud line was not captured in this pass.

本地文件Local files

评测定义是仓库内 YAML（evals/registry/evals、evals/registry/eval_sets），实现代码在 evals/elsuite； 数据集走 Git-LFS 取到 evals/registry/data。默认日志写本机 tmp/evallogs 的 JSONL，--record_path 可改路径。 按 README，用 JSON 数据 + YAML 模板即可写评测，不必写代码。

Eval definitions are YAML in the repo (evals/registry/evals, evals/registry/eval_sets) with implementations under evals/elsuite; dataset files come through Git-LFS into evals/registry/data. Logs default to local JSONL in tmp/evallogs, overridable with --record_path. Per the FAQ, a template-based or model-graded eval needs JSON data plus YAML and no evaluation code.

关机后的任务Tasks after shutdown

本机长跑，不是托管队列。官方文档写明 oaievalset 把已完成的评测记在 /tmp/oaievalset/{model}.{eval_set}.progress.txt， 中断或崩溃后重跑可接着来；单个 eval 不能从中间续，只能重头跑。并发靠 EVALS_THREADS / EVALS_THREAD_TIMEOUT（默认 10 线程、40 秒超时）。 关机后由谁续跑：官方未说明。

A local long run, not a hosted queue. The official docs state oaievalset records finished evals in /tmp/oaievalset/{model}.{eval_set}.progress.txt, so a rerun picks up where it stopped; a single eval cannot be resumed mid-way and must restart. Concurrency is EVALS_THREADS / EVALS_THREAD_TIMEOUT (default 10 threads, 40-second timeout). What happens after the machine shuts down: not stated officially.

工具与扩展Tools & extensions

扩展点是 CompletionFn 协议、eval 模板与 model-graded YAML，README 称高级场景（提示链、工具调用 agent）走 Completion Function Protocol。 贡献面有限制：README 写明目前不接受带自定义代码的评测贡献，仍可提交 model-graded YAML。MCP 接入、护栏或拦截机制：官方未说明。

Extension points are the CompletionFn protocol, eval templates and model-graded YAML; the README routes prompt chains and tool-using agents to the Completion Function Protocol. Contribution is restricted: the README says evals with custom code are not accepted at present, while model-graded YAML files still are. MCP support, guardrails or interception mechanisms: not stated officially.

上下文与记忆Context & memory

没有跨会话记忆这一层——一次运行就是一轮，产物是本地 JSONL 日志与结尾报告。 多轮/长文本类评测走仓库里的 eval 模板（README 的 FAQ 提到模板选择会影响同一段数据的表现）。 上下文窗口截断规则与样本默认上限：本批次未取到官方正文。

There is no cross-session memory layer: one run is one round, and the artifacts are the local JSONL log plus the end-of-run report. Multi-turn or long-text workloads go through the eval templates in the repo (the FAQ shows the same dataset scored under different templates). Window truncation rules and default sample limits: no official text captured in this pass.

权限与限制Permissions & limits

代码 MIT；运行用自己的 API key，README 提供 private evals 路径以在不公开数据的情况下评测。 贡献条款要读原文：贡献进仓库的评测逻辑与数据按同 MIT 许可，且 "OpenAI reserves the right to use this data in future service improvements"， 提交还受平台 Usage Policies 约束。数据在 OpenAI 服务端的留存与训练用途：官方未说明。

The code is MIT; you run with your own API key, and the README describes private evals for testing your common LLM patterns without exposing the data publicly. The contribution terms need reading literally: contributed eval logic and data fall under the same MIT license, and "OpenAI reserves the right to use this data in future service improvements", with contributions subject to the platform Usage Policies. Server-side retention and training use: not stated officially.

适合什么任务Best for

适合：把公开基准与自写 YAML 评测放在本机或 CI 里批量跑、要可读的 registry 与本地 JSONL 日志、愿意实现 CompletionFn 接自有系统。 不适合：只要托管控制台与托管数据面（该走 Dashboard Evals）、强依赖非 OpenAI 模型且不想写适配层、要沙箱或工具审批这类运行时边界。

Fits: running public benchmarks plus your own YAML evals in batch on your machine or in CI, reading logs as files rather than a console, and implementing CompletionFn for your own systems. Not a fit: wanting only a hosted console and hosted data plane (that is the Dashboard line), depending heavily on non-OpenAI models without writing an adapter, or needing sandboxing and tool approval as runtime boundaries.

本页由This page is generated from the ai-handbook 数据层生成（CC BY 4.0）。data layer (CC BY 4.0). 方法论与坐标系定义见仓库内For the methodology and coordinate definitions, see METHODOLOGY.md.