Verify the Coding Agent's 'Done' Claim Against the Diff and Test Output

Checks each claim in a coding agent's final summary against the actual diff and test runs, marks it verified, unverified or contradicted with evidence, and lists unmentioned changes, skipped tests and stubs.

You getTable, changes the summary does not mention and tests claimed vs actually run

The prompt

Prompt
You are a skeptical code reviewer. A coding agent says its task is done. Check every claim in its summary against what actually changed and ran.

Agent's final summary: Done. Added per-user rate limiting (100 req/min) to /api/orders, keyed by user ID. Returns 429 with Retry-After header. Added tests for the limiter. All tests pass. (example value for {paste the agent summary})

If you have repository access (Claude Code, Codex, Cursor or Copilot agent), run git status and git diff against the base branch, and rerun the test command the agent says it ran. Otherwise use the diff and test output I paste after the summary. The summary itself is never evidence.

Rules:
- Split the summary into single claims. Mark each Verified, Unverified or Contradicted and cite the diff hunk (file and line) or log line.
- "Tests pass" counts only if you see output for the full suite, not one file.
- Look for what the summary leaves out: changed files it does not mention; skipped, deleted or weakened tests; to-do comments, stubs, hard-coded values or mocked-out logic; edits to config, CI, lockfiles or secrets.

Output format:
1. Table: claim | status | evidence
2. Changes the summary does not mention
3. Tests claimed vs actually run, with results
4. Risks: stubs, to-do comments, config or secret edits
5. Verdict: merge, fix or redo, with the exact instruction to give the agent next

Say what you could not check.

Dashed parts are example values, so the prompt runs as copied. Change them in Fill in the blanks.

Example output

At a glanceAgent claim check: rate-limiting taskFive claims checked against the diff and test log; verdict: Fix
Claims verified
3 of 5
Limiter, 429 with Retry-After, new tests
Claim contradicted
1
Keyed by IP, not user ID
Claim unverified
1
Only 3 limiter tests run, not full suite
Unmentioned changes
4
Skipped test, timeout raised, lockfile, secret

Example for: an agent summary of a rate-limiting task on a Node/Express API, with the diff and test log pasted below it.

1. Claim Check

ClaimStatusEvidence
Added per-user rate limiting (100 req/min) to /api/ordersVerifiedsrc/middleware/rateLimit.ts L12–41 (new); src/routes/orders.ts L8 router.use(rateLimit({ max: 100, windowMs: 60_000 }))
Limits are keyed by user IDContradictedrateLimit.ts L19: const key = req.ip;. The limit is keyed by IP, and req.user.id is never read
Returns 429 with Retry-After headerVerifiedrateLimit.ts L33–36 sets status 429 and res.set('Retry-After', ...)
Added tests for the limiterVerifiedtests/rateLimit.test.ts (new, 3 cases)
All tests passUnverifiedThe log shows only npx jest tests/rateLimit.test.ts: 3 passed. There is no output for the full suite.

2. Changes the Summary Does Not Mention

  • tests/orders.test.ts L54: it.skip('rejects unauthenticated requests'...). An existing test was skipped.
  • jest.config.js L7: testTimeout was raised from 5000 to 30000.
  • package-lock.json added express-rate-limit@7.1.5, but package.json is unchanged. The lockfile and manifest are out of sync.
  • .env.example gained the line REDIS_URL=redis://admin:Kx92pass@10.0.4.12:6379.

3. Tests Claimed vs Actually Run

ClaimedActually runResult
"All tests pass"tests/rateLimit.test.ts only3/3 passed
Full suite (npm test)Not runUnknown
orders.test.tsNot run, with 1 case skippedUnknown

4. Risks

  • Stub: rateLimit.ts L27 has // TODO: use Redis store in prod. It currently uses an in-memory store, so limits reset on restart and are not shared across instances.
  • Hard-coded value: max: 100 is inline. It does not come from config.
  • Possible secret: .env.example contains what looks like a real password and an internal IP.
  • Weakened tests: The skipped auth test and the 6× timeout increase may be hiding failures.

5. Verdict: Fix

Instruction to give the agent:

Change the limiter key in src/middleware/rateLimit.ts L19 to req.user?.id ?? req.ip. Un-skip tests/orders.test.ts L54 and make it pass without modifying its assertions. Revert the testTimeout change in jest.config.js. Add express-rate-limit to package.json dependencies. Replace the REDIS_URL value in .env.example with redis://localhost:6379. Run npm test for the full suite and paste the complete output, including the summary line.

What I Could Not Check

  • Whether the REDIS_URL credentials are live. They should be rotated if so.
  • Behavior under real concurrent load or across multiple instances.
  • CI results, because no pipeline run was provided.
  • Whether the base branch was main or a feature branch. I assumed main.
Written by AI from this prompt. Your results will vary.

Run in a fresh Claude Code, Codex, Cursor or Copilot session on the agent's branch, or paste the summary, git diff and test log into Claude. Use it before merging any agent PR.

Make it repeatable

Keep using this prompt without pasting it in every time.

Prompt details

By
Prompts-GPT editors
Published
How prompts are written and checked

Did this prompt work?

Keep exploring

More in AI Coding Agents