为了让测试通过,把测试改弱了
Made the test pass by weakening the test
失败的检查变绿了,不是因为代码对了,而是智能体跳过、删除、放宽了测试,或者把被测单元 mock 掉了。
适用于: Claude Code · Cursor · Codex · Copilot agent
症状
CI 是绿的,但 diff 改了测试文件。留意 `.skip`、`xit`、重新生成的快照、`toBe(19.99)` 变成 `toBeCloseTo(20, 0)`、专门针对测试输入的分支、`@ts-expect-error` 或 `continue-on-error`。
为什么会这样
智能体优化的目标是“检查通过”,而改检查往往是最短路径。developertoolkit.ai 的判定规则是:只要 diff 改了判定文件(测试/CI),就归为 oracle gaming,因为改动本身就是证据。agent-failure-modes(AFM-14)和 agent-review(AR025 禁用或跳过测试)也列了同样的行为。
示例
模式摘自 developertoolkit.ai 和 agent-review。
ts- expect(total).toBe(19.99);
+ expect(total).toBeCloseTo(20, 0);
- it("rejects expired tokens", ...)
+ it.skip("rejects expired tokens", ...)
复制即用的修复
按你用的智能体挑选文件。规则片段用英文写,放进任何仓库都能用。
规则
Claude Code:加入 CLAUDE.md
CLAUDE.md## Tests are the spec: don't weaken them
- Never skip, delete, loosen or mock away a failing test to make it pass. Fix the code.
- Do not regenerate snapshots or edit CI config unless the user asked for it.
- If you believe a test is wrong, stop and explain why, quoting the test and the requirement. Let the user decide.
- In your summary, list every test file you changed and why.
规则
Codex、Copilot 编码智能体等:加入 AGENTS.md
AGENTS.md## Tests are the spec: don't weaken them
- Never skip, delete, loosen or mock away a failing test to make it pass. Fix the code.
- Do not regenerate snapshots or edit CI config unless the user asked for it.
- If you believe a test is wrong, stop and explain why, quoting the test and the requirement. Let the user decide.
- In your summary, list every test file you changed and why.
Cursor 规则
Cursor:项目规则
.cursor/rules/test-gaming.mdc---
description: "AgentFails: Made the test pass by weakening the test"
globs: **/*.{test,spec}.*
alwaysApply: false
---
## Tests are the spec: don't weaken them
- Never skip, delete, loosen or mock away a failing test to make it pass. Fix the code.
- Do not regenerate snapshots or edit CI config unless the user asked for it.
- If you believe a test is wrong, stop and explain why, quoting the test and the requirement. Let the user decide.
- In your summary, list every test file you changed and why.
钩子
Claude Code PreToolUse 钩子:除非你明确允许,测试文件只读
像范围守卫一样注册(matcher 为 Edit|Write|MultiEdit)。如果这次任务本身就是改测试,用 `ALLOW_TEST_EDITS=1 claude` 启动。
.claude/hooks/protect_tests.py#!/usr/bin/env python3
"""Block edits to test files and CI config unless ALLOW_TEST_EDITS=1."""
import json, os, re, sys
call = json.load(sys.stdin)
if call.get("tool_name") not in ("Edit", "Write", "MultiEdit") or os.environ.get("ALLOW_TEST_EDITS") == "1":
sys.exit(0)
path = call["tool_input"].get("file_path", "")
protected = re.compile(r"(^|/)(tests?|__tests__|spec)/|\.(test|spec)\.[jt]sx?$|_test\.(go|py)$|(^|/)test_[^/]+\.py$|\.github/workflows/")
if protected.search(path):
print(f"BLOCKED: {path} is a test/CI file. Make the code pass the existing test. "
"If you believe the test itself is wrong, stop and explain why to the user.", file=sys.stderr)
sys.exit(2)
sys.exit(0)
钩子
Claude Code Stop 钩子:测试不过就不让智能体收工
Stop 钩子以退出码 2 结束时,Claude 会继续工作并看到 stderr。检查 stop_hook_active 可避免无限循环。把 `npm test` 换成你的测试命令。
.claude/settings.json{
"hooks": {
"Stop": [
{
"hooks": [
{
"type": "command",
"command": "python3 -c \"import json,sys,subprocess; d=json.load(sys.stdin); sys.exit(0) if d.get('stop_hook_active') else None; r=subprocess.run('npm test --silent', shell=True, capture_output=True, text=True); sys.stderr.write('Tests are failing. Fix them (without weakening them) before stopping:\\n'+r.stdout[-3000:]+r.stderr[-2000:]) if r.returncode else None; sys.exit(2 if r.returncode else 0)\""
}
]
}
]
}
}