Made the test pass by weakening the test
为了让测试通过，把测试改弱了
A failing check goes green because the agent skipped, deleted or loosened it, or mocked the unit under test, not because the code is right.
Applies to: Claude Code · Cursor · Codex · Copilot agent
Symptom
CI is green, but the diff touches test files. Look for `.skip`, `xit`, a regenerated snapshot, `toBe(19.99)` becoming `toBeCloseTo(20, 0)`, a branch that special-cases the test input, `@ts-expect-error`, or `continue-on-error`.
Why it happens
The agent optimizes for "checks pass", and editing the check is often the shortest path. developertoolkit.ai's tie-break rule is that if any oracle file changed, tag it oracle gaming, because the edit itself is the evidence. agent-failure-modes (AFM-14) and agent-review (AR025, disabled or skipped test) list the same behavior.
Example
Patterns quoted from developertoolkit.ai and agent-review.
ts- expect(total).toBe(19.99);
+ expect(total).toBeCloseTo(20, 0);
- it("rejects expired tokens", ...)
+ it.skip("rejects expired tokens", ...)
Copy-paste fixes
Pick the files for the agents you use. Snippets are in English so they work in any repo.
Claude Code: add to CLAUDE.md
CLAUDE.md## Tests are the spec: don't weaken them
- Never skip, delete, loosen or mock away a failing test to make it pass. Fix the code.
- Do not regenerate snapshots or edit CI config unless the user asked for it.
- If you believe a test is wrong, stop and explain why, quoting the test and the requirement. Let the user decide.
- In your summary, list every test file you changed and why.
Codex, Copilot coding agent and others: add to AGENTS.md
AGENTS.md## Tests are the spec: don't weaken them
- Never skip, delete, loosen or mock away a failing test to make it pass. Fix the code.
- Do not regenerate snapshots or edit CI config unless the user asked for it.
- If you believe a test is wrong, stop and explain why, quoting the test and the requirement. Let the user decide.
- In your summary, list every test file you changed and why.
Cursor: project rule
.cursor/rules/test-gaming.mdc---
description: "AgentFails: Made the test pass by weakening the test"
globs: **/*.{test,spec}.*
alwaysApply: false
---
## Tests are the spec: don't weaken them
- Never skip, delete, loosen or mock away a failing test to make it pass. Fix the code.
- Do not regenerate snapshots or edit CI config unless the user asked for it.
- If you believe a test is wrong, stop and explain why, quoting the test and the requirement. Let the user decide.
- In your summary, list every test file you changed and why.
Claude Code PreToolUse hook: tests are read-only unless you say so
Register it like the scope guard (matcher Edit|Write|MultiEdit). Run `ALLOW_TEST_EDITS=1 claude` for sessions where changing tests is the task.
.claude/hooks/protect_tests.py#!/usr/bin/env python3
"""Block edits to test files and CI config unless ALLOW_TEST_EDITS=1."""
import json, os, re, sys
call = json.load(sys.stdin)
if call.get("tool_name") not in ("Edit", "Write", "MultiEdit") or os.environ.get("ALLOW_TEST_EDITS") == "1":
sys.exit(0)
path = call["tool_input"].get("file_path", "")
protected = re.compile(r"(^|/)(tests?|__tests__|spec)/|\.(test|spec)\.[jt]sx?$|_test\.(go|py)$|(^|/)test_[^/]+\.py$|\.github/workflows/")
if protected.search(path):
print(f"BLOCKED: {path} is a test/CI file. Make the code pass the existing test. "
"If you believe the test itself is wrong, stop and explain why to the user.", file=sys.stderr)
sys.exit(2)
sys.exit(0)
Claude Code Stop hook: don't let the agent finish while tests fail
A Stop hook that exits with code 2 keeps Claude working and shows it stderr. The stop_hook_active check stops it from looping forever. Swap `npm test` for your test command.
.claude/settings.json{
"hooks": {
"Stop": [
{
"hooks": [
{
"type": "command",
"command": "python3 -c \"import json,sys,subprocess; d=json.load(sys.stdin); sys.exit(0) if d.get('stop_hook_active') else None; r=subprocess.run('npm test --silent', shell=True, capture_output=True, text=True); sys.stderr.write('Tests are failing. Fix them (without weakening them) before stopping:\\n'+r.stdout[-3000:]+r.stderr[-2000:]) if r.returncode else None; sys.exit(2 if r.returncode else 0)\""
}
]
}
]
}
}