← Safety

promptfoopromptfoo

promptfoo

开源Open source MITMIT 终端Terminal CICI evaleval 本地Local 多模型Multiple models partial lifecycle: active 核验Checked 2026-11-06

核验标记表示资料完整度，不是产品评分。下列未确认项可能影响选型；请按对应来源逐项确认。The confidence mark describes evidence completeness, not a product score. Unconfirmed items below may affect your choice; check them against the relevant sources.

未确认的信息Information not confirmed

主体归属：官方仓库元数据当日在册自述站点为 promptfoo.dev；公司主体未逐条核验
LICENSE 文件正文与逐字许可条款（当日抓取失败，仅有 API 元数据在册）
官方 changelog 页正文（两处路径当日均 404）
本地模型（OpenAI 兼容端点）接入的具体配置面
分布式执行与断点续跑能力
promptfoo 自身是否提供 MCP server
promptfoo.app 托管面的数据留存与训练条款
备案公示检索

Configuration surface for local models behind OpenAI-compatible endpoints
Distributed execution and checkpointed restarts
Whether an MCP server is provided
Data retention and training terms of the hosted promptfoo.app
Team, RBAC and audit capabilities on the hosted side
Behaviour of red-team plugins beyond what the official page states

English summaries and interface labels are available. Detailed source notes below are currently in Chinese.

promptfoo 是终端里的评测与红队工具。官方自我定位是「评测与红队 LLM 应用的开源 CLI 与库」，产物留在本机，模型自带 key。

适合与不适合When to use it, when not to

适合把评测当代码资产管理的团队：一份 YAML 定义多提示词、多模型、多用例，一条命令跑出矩阵对照，再让不达阈值的构建在 CI 里失败。不适合需要托管控制台做长期指标看板，或希望厂商代跑合规审计的场景。它另有红队面，但本站只记支持的攻击插件与漏洞类别，不记测出来的分数与通过率。

中文生态China availability 国产 / 国内可直接用 / 备案状态 · 快照domestic / direct access / CAC filing · snapshot 2026-11-06

出品属地Origin: 国外出品foreign-made　官方仓库与文档域名当日在册（promptfoo.dev 与 github.com/promptfoo）；项目主体与注册地未逐条核验，按 foreign 记。the official repository and docs domains are on record today (promptfoo.dev and github.com/promptfoo); the entity and place of incorporation were not item-verified, recorded as foreign.
国内使用Direct access: 大陆可直连direct OK　www.promptfoo.dev 的 7 个文档页当日直抓均 200；github.com 的仓库与 releases HTML 页当日抓取失败（改由官方 GitHub API 在册），npmjs.com 返回 403。seven docs pages on www.promptfoo.dev fetched 200 today; github.com repository and releases HTML pages failed to fetch (taken from the official GitHub API instead), and npmjs.com returned 403.
备案状态CAC filing: 备案未核验filing unknown：本次未检索备案公示名单；开源 CLI 本体是否落入备案口径未核验，记入未知项清单。the filing registry was not searched today; whether an open-source CLI falls under filing scope is unverified — listed under unknowns.

备案信息以网信办《生成式人工智能服务已备案信息》公示名单为准（按批次滚动更新，本站只抄编号不解读）；「大陆可直连」是本站从大陆网络直连实测的结果，不是厂商承诺。GitHub 类站点当日实测有波动，口径写在依据里。Filing status is quoted from the CAC public registry (updated batch by batch; this site copies the number and does not interpret it). "Direct access" is this site's own reachability measurement from a mainland network, not a vendor promise. GitHub-hosted sites fluctuated on the measurement day — the note says so.

固定坐标系Fixed coordinate system

8 个维度dimensions, 与同分区其他对象逐项可比each comparable item by item against others in the same section.

模型与开放条件Model & openness: 全靠自带凭据：官方 intro 当日在册支持 OpenAI、Anthropic、Azure、Google、HuggingFace 与 Llama 这类开源模型，口径写明 BYOK 与 API keys。本地模型走 OpenAI 兼容端点的具体配置面本次未逐条核验。 评测器不内置厂商模型，judge 调用落在你配置的账户上。
Bring-your-own keys throughout: the intro page was on record the same day as supporting OpenAI, Anthropic, Azure, Google, Hugging Face and open models such as Llama, stated as BYOK with API keys. The exact configuration surface for local models behind OpenAI-compatible endpoints was not verified item by item. The evaluator ships no vendor model; judge calls run against the account you configure.
运行位置Where it runs: 终端为主：npx promptfoo init 起手、eval 跑评测、view 起本机结果页（当日在册），配置是 promptfooconfig.yaml，本地运行不需要登录（当日在册）。另有 promptfoo.app 托管查看入口，同页在册称分享链接对本组织私有；托管面细节未核验。
Terminal-first: npx promptfoo init to start, eval to run, view to serve a local results page (all on record the same day), with configuration in promptfooconfig.yaml and local runs stated as requiring no login. A hosted viewer at promptfoo.app also appears on that page, which states shared links are private to your organisation; hosted-side details were not verified.
本地文件Local files: 读写都在本机：配置、用例与结果产物留在本地目录，viewer 读本地文件渲染（当日在册）；测试数据可来自 CSV 或 JSON 与自定义函数（当日在册）。平台不读本机代码库以外的隐式上下文。
Reads and writes stay on your machine: configuration, test cases and result artefacts live in local directories and the viewer renders local files (on record); test data may come from CSV or JSON, or from custom functions. The platform does not pick up implicit context beyond what you point it at.
关机后的任务Tasks after shutdown: 评测是本机进程内跑完的批任务，官方在册有缓存、并发与热重载；CI 里按阈值让构建失败（当日在册「Fail the build when quality thresholds aren't met」）。关机续跑不适用；分布式执行与断点续跑面未在册，记为未知。
Evaluation runs as a local batch process; caching, concurrency and hot reload are on record, as is failing the build when quality thresholds are not met in CI. Resuming after shutdown does not apply. Distributed execution and checkpointed restarts are not on record and are treated as unknown.
工具与扩展Tools & extensions: 作为被调用方，它的扩展面是 provider 配置、断言类型与自定义函数，文档索引当日在册含 CI 与代码扫描（GitHub Action）集成页；结果可发到 PR 评论（当日在册）。 promptfoo 自身是否提供 MCP server 未核验到官方声明，记为未知。
Its extension surface, as the called side, is provider configuration, assertion types and custom functions; the documentation index was on record the same day including CI and code-scanning (GitHub Action) integration pages, and results can be posted to PR comments. Whether promptfoo itself provides an MCP server is not stated in what was captured, so it is recorded as unknown.
上下文与记忆Context & memory: 评测上下文完全由配置文件定义：多提示词 × 多模型 × 多用例的矩阵对照（当日在册 matrix views），期望值可用 CSV 的 __expected 列或自定义函数给（当日在册）。跨会话记忆不是这类工具的功能面。
Evaluation context is entirely defined by the configuration file: a matrix across prompts x models x test cases (matrix views on record); expectations can come from a CSV __expected column or custom functions. Cross-session memory is not part of what this class of tool does.
权限与限制Permissions & limits: MIT 许可（当日在册元数据）；本机执行加自带 key，意味着模型调用费用与数据出口都在你自己账户（BYOK 口径当日在册）；官方 CI 页建议把 key 存成加密 secrets（当日在册）。 promptfoo.app 托管面的数据留存与训练条款正文本次未抓到，记为未知。
MIT licensed (metadata on record). Local execution plus your own keys means model spend and egress sit in your account (BYOK on record); the official CI page recommends storing keys as encrypted secrets. Retention and training terms for the promptfoo.app hosted side were not captured, so they remain unknown.
适合什么任务Best for: 适合把评测写进仓库、用一次命令跑多提示词与多模型对照、并在 CI 里按阈值挡住合并的团队。不适合要托管控制台做长期指标看板、要厂商侧代跑合规审计的场景；本站只记支持的断言类型与红队能力面，不记测出的分数，也不排名。
Fits teams that want evaluation in the repository, one command to compare many prompts against many models, and a CI threshold that blocks merges. Does not fit anyone needing a hosted console as a long-term metrics dashboard, or vendor-run compliance auditing. This site records the assertion types and red-teaming surface only, never scores produced by the tool, and draws no ranking.

头号误解Top misconception

把「自带评测能力」当成自带模型——相似度与 rubric 类断言都调你配置的服务，key 与用量在你账上（当日在册）
把红队页当成安全评分——官方在册的是攻击插件与漏洞类别面，本站不采通过率也不排名
把本机 viewer 当成 SaaS——promptfoo.app 是可选面，评测与产物默认留在本机（当日在册）

证据来源Evidence sources

最近核验记录Latest recorded check 2026-11-06，内容更新日content updated on 2026-11-06。单项复核范围见正文或来源说明，不代表所有字段在同一天重新核验。See the text or source notes for item-specific scope; this does not mean every field was rechecked on that date.

类型Type	名称Name	链接Link
docs	promptfoo · 文档 intro（自述 open-source CLI and library for evaluating and red-teaming LLM apps；断言含 determinism 与 similarity 与 JSON validity 与 LLM-rubric；BYOK 与多厂商 provider，2026-11-06 直抓 200）promptfoo · docs intro (self-described open-source CLI and library for evaluating and red-teaming LLM apps; assertions include determinism, similarity and JSON validity and LLM-rubric; BYOK across many providers; 200 direct on 2026-11-06)	https://www.promptfoo.dev/docs/intro/
docs	promptfoo · 断言与期望值文档（确定性类 equals 与 contains 与 regex 与 json 与 sql 与 latency 与 cost；judge 类 llm-rubric 与 factuality 与 similarity；CSV __expected 与自定义函数，2026-11-06 直抓 200）promptfoo · assertions and expectations docs (equals/contains/regex/json/sql/latency/cost; llm-rubric/factuality/similarity; CSV __expected and custom functions; 200 direct on 2026-11-06)	https://www.promptfoo.dev/docs/configuration/expected-outputs/
docs	promptfoo · 上手页（npx promptfoo init 与 eval 与 view、promptfooconfig.yaml、本地运行不需登录、OPENAI_API_KEY 配置，2026-11-06 直抓 200）promptfoo · getting started (npx promptfoo init / eval / view, promptfooconfig.yaml, no login needed locally, OPENAI_API_KEY; 200 direct on 2026-11-06)	https://www.promptfoo.dev/docs/getting-started/
docs	promptfoo · 本机结果页（viewer 命令与结果图表；另在册 promptfoo.app 分享链接对本组织私有，2026-11-06 直抓 200）promptfoo · local viewer (viewer command and result charts; promptfoo.app shared links private to your organisation; 200 direct on 2026-11-06)	https://www.promptfoo.dev/docs/usage/web-ui/
docs	promptfoo · CI 集成页（密钥存加密 secrets、阈值不达则构建失败、结果发 PR 评论，2026-11-06 直抓 200）promptfoo · CI integration (keys as encrypted secrets, fail the build below thresholds, results to PR comments; 200 direct on 2026-11-06)	https://www.promptfoo.dev/docs/integrations/ci-cd/
docs	promptfoo · 红队页（自动生成攻击与插件面、覆盖越狱类漏洞检查；不含本站口径的分数结论，2026-11-06 直抓 200）promptfoo · red-teaming page (generated attacks and plugin surface, jailbreak-class checks; no score conclusions adopted by this site; 200 direct on 2026-11-06)	https://www.promptfoo.dev/docs/red-team/
docs	promptfoo · 文档索引（当日在册含 caching、red-team configuration、code scanning 与 GitHub Action 等条目）promptfoo · documentation index (caching, red-team configuration, code scanning and GitHub Action entries on record the same day)	https://www.promptfoo.dev/llms.txt
repo	promptfoo/promptfoo · 仓库元数据（GitHub API 当日直抓在册：MIT、最近推送 2026-11-06、官方站 promptfoo.dev）promptfoo/promptfoo · repository metadata (GitHub API on record 2026-11-06: MIT, last push 2026-11-06, official site promptfoo.dev)	https://api.github.com/repos/promptfoo/promptfoo
changelog	promptfoo/promptfoo · Releases（经官方 GitHub API 当日在册，最新 0.125.1 发布于 2026-10-25；HTML releases 页正文本次抓取失败）promptfoo/promptfoo · releases (via the official GitHub API on record: 0.125.1 published 2026-10-25; the HTML releases page itself failed to fetch)	https://api.github.com/repos/promptfoo/promptfoo/releases/latest

官方口径与未核到项Official record and unverified items

本站尚未完成实测，结论只覆盖能力面与许可面。

实测建议（三条，都是官方可读页给不出的东西）：

阈值门禁的稳定性：同一份配置连跑 20 次，judge 类断言的波动幅度有多大
本机产物体积与缓存命中：用例数上千时，缓存与并发参数的实际收益
托管面的边界：promptfoo.app 的数据留存、成员权限与导出方式

This entry has not completed measured runs: no pass rates, no attack success rates, no latency or cost figures. Everything above is organised from the official pages fetched on 2026-11-06, and the gaps are listed separately rather than filled by inference.

变更记录说明

本对象没有可直接抓到的 changelog 页（/changelog 与 /docs/changelog 当日均 404），变更记录以官方仓库 Releases 为准；当日经 GitHub API 在册最新为 0.125.1（2026-10-25），仓库最近推送日期为 2026-11-06，因此 lifecycle 记 active。