工具箱：Agent 评估、护栏、客服验收、故障清单Toolkit: agent ops checklists

模板内的比例、样本数、阈值和权限策略是示例。请按任务、授权与风险确定实际条件；填完表格不等于已经通过验收。

四个块用于上线前评审与事故复盘；括号里替换成你的项目值。

1. Agent 评估 rubric

| 维度         | 权重 | 通过线 | 证据来源       |
|--------------|------|--------|----------------|
| 任务完成率   | 30%  | ≥【X】% | 100 条离线集   |
| 引用准确率   | 20%  | ≥【X】% | 抽 20 条回原文 |
| 该拒答时拒答 | 20%  | ≥【X】% | 禁区问题集     |
| 工具调用正确 | 15%  | ≥【X】% | 调用日志       |
| 平均轮次     | 10%  | ≤【N】轮 | 会话日志       |
| 延迟         | 5%   | ≤【N】秒 | 监控           |

示例评分可用每维 1–5 分，实际门槛须事先约定；权限或安全失败应单独判定，不能由加权总分抵消。

2. 四层护栏清单

输入层
- [ ] PII 识别与脱敏
- [ ] 提示注入检测
- [ ] 超长输入截断
- [ ] 文件类型白名单

知识层
- [ ] 只答库内内容
- [ ] 每条回答引用原文句
- [ ] 过期条目标记并下线
- [ ] 冲突来源同时保留

工具层
- [ ] 工具白名单
- [ ] 参数 schema 校验
- [ ] 写操作人工确认
- [ ] 调用频率限制

输出层
- [ ] 敏感承诺过滤
- [ ] 转人工触发条件
- [ ] 全链路日志留痕
- [ ] 一键回滚开关

3. 客服 Agent 验收单

- [ ] 意图清单与禁区清单已评审
- [ ] 知识库只含已发布内容，带生效日期
- [ ] 写操作在已授权范围内执行，敏感操作按风险另设确认
- [ ] 离线集 100 条，覆盖高频与禁区
- [ ] 记录灰度期错误承诺，并预先约定停止与回滚条件
- [ ] 转人工条件与责任人写清
- [ ] 日志保留周期符合合规
- [ ] 回滚开关演练过一次

4. 故障复盘清单

| 问题 | 记录 |
|------|------|
| 发生时间 / 发现时间 | |
| 影响范围（用户/订单/数据） | |
| 触发路径（输入/知识/工具/输出） | |
| 当时模型与版本 | |
| 是否有护栏拦截 | |
| 回滚时间与方式 | |
| 根因（可复现证据） | |
| 修复与回归测试 | |
| 同类风险排查结果 | |

把证据、假设与确认的根因分开；未确认的部分仍可写“可能”，但不要当作结论。

Use these four blocks for pre-release review and incident analysis. Replace project fields. The weights, sample counts and thresholds are illustrative; define actual criteria before evaluation. Authorization or safety failures should not be offset by a weighted average.

1. Evaluation rubric

Dimension | Example weight | Threshold | Evidence
Task completion | 30% | [X]% | Fixed task set (example: 100)
Citation accuracy | 20% | [X]% | Source checks (example: 20)
Required refusals | 20% | [X]% | Out-of-scope questions
Tool correctness | 15% | [X]% | Invocation logs
Average turns | 10% | [N] | Session logs
Latency | 5% | [N] seconds | Monitoring

Example scoring: 1–5 per dimension, with an agreed minimum.
Specify non-negotiable permission/safety conditions separately.

2. Four layers of controls

Input: identify/redact personal data; check injection; handle long inputs;
restrict file types.
Knowledge: answer within the permitted corpus; cite supporting passages;
retire outdated records; preserve conflicts.
Tools: restrict names; validate schemas; require appropriate approval for writes;
limit call frequency.
Output: check sensitive commitments; define escalation; retain suitable traces;
provide rollback.

3. Service-agent acceptance

- [ ] Intents and excluded requests reviewed
- [ ] Approved knowledge has effective dates
- [ ] Write authorization is enforced
- [ ] Task set covers frequent and prohibited requests (100 is illustrative)
- [ ] Incorrect commitments monitored with defined stop conditions
- [ ] Escalation conditions and owners recorded
- [ ] Log retention checked against applicable requirements
- [ ] Rollback exercised

4. Incident record

Record | Evidence
Occurrence/discovery time |
Affected users/orders/data |
Input/knowledge/tool/output path |
Model/version |
Controls that fired |
Rollback time/method |
Reproducible cause |
Fix/regression checks |
Related risk review |

Separate evidence, hypotheses and confirmed causes.

• 评估只测「能答」：不测拒答，上线后错误承诺最多。
• 护栏只写在文档：没有指标与日志，出事无法定位。
• 验收单不复核：一次通过就永久通过，知识库更新后失效。
• 复盘写情绪：没有可复现证据，修复只能靠猜。

• Test appropriate refusals as well as answerability.
• Controls need observable behavior and records, not documentation alone.
• Repeat relevant checks after knowledge or configuration changes.
• Incident explanations need reproducible evidence; keep unconfirmed hypotheses labeled.