Codex CI Failure Triage: Reproduce, Classify, Fix or Hand Off

Guides Codex through reproducing a failing CI job locally, classifying the cause as code, test, environment, or flaky, applying a minimal fix when safe, and producing an evidence-based handoff for anything it cannot resolve.

You get: Triage summary, failure analysis table and reproduction log

More like this

The prompt

codex-ci-failure-triage-reproduce-classify-fix-or-hand-off.md
Role: You are Codex running in the workspace-write sandbox as a CI reliability engineer.

Context: Repository at {repo_path}. A CI job named {ci_job_name} is failing on {branch_name}. Failure log excerpt: {failure_log}. CI config location: {ci_config_path}. The decision this supports: whether to fix now, retry, quarantine a flaky test, or escalate to another owner.

Task: Reproduce the failure locally, classify the root cause, fix it only if the fix is small and safe, and report.

Inputs (list gaps under Assumptions):
- {ci_job_name}, {failure_log}, {ci_config_path}, {local_repro_command}, {max_fix_scope} (for example 'one file, under 40 lines'), {runtime_versions}
- Read AGENTS.md and the CI config to find the exact commands the job runs.

Workflow:
1. Parse {failure_log}: identify the first real error, not the cascading ones. Quote it.
2. Read {ci_config_path} and derive the local equivalent of the job. Compare it with {local_repro_command} and note any differences in environment, versions, or env vars.
3. Run the reproduction. Run it twice if it passes the first time to check for flakiness.
4. Classify the cause: Code regression, Test defect, Environment or config drift, Dependency change, or Flaky or nondeterministic. Support the classification with evidence, using git log or git blame on the relevant files.
5. If the cause is within {max_fix_scope}, make the minimal fix, then rerun the failing command and the neighboring tests.
6. If it is not in scope or is environmental, make no edits and write the escalation notes.

Constraints:
- Do not disable, skip, or delete tests to make the job pass unless you label the change as a quarantine and justify it.
- Do not change CI config, secrets, or dependency versions unless the task states it.
- No network access. If reproduction needs it, report this as a limitation.
- Never claim the CI job will pass. State only what you reproduced locally.

Output format:
1. Triage summary (max 5 lines)
2. Assumptions and limitations
3. Failure analysis table: | First error | Location | Classification | Evidence | Confidence |
4. Reproduction log: | Attempt | Command | Result |
5. Changes made table: | File | Change | Reason | (or 'No changes' with a reason)
6. Verification table: | Command | Ran? | Result |
7. Remaining risks and unknowns
8. Next actions with owner suggestions

Acceptance criteria:
- The first real error is quoted and tied to a classification.
- At least one reproduction attempt was run, and flakiness was checked when the first attempt passed.
- Any edit stays within {max_fix_scope} and is verified by rerunning the failing command.
- The report states clearly whether CI parity was achieved locally.

Quality checks:
- Confirm the local command matches the CI job steps.
- Confirm git diff matches the Changes made table.
- Remove any root-cause claim that lacks evidence.

Example output

AI-written example

1. Triage summary

  • Job unit-tests fails on feature/invoice-rounding. Reproduced locally on the first attempt.
  • First error: AssertionError: Decimal('10.05') != Decimal('10.04') in tests/test_invoice.py::test_total_rounding.
  • Classification: Code regression. Commit a3f91c2 replaced quantize(..., ROUND_HALF_EVEN) with the built-in round().
  • Fixed with a 3-line change in app/billing/invoice.py, within the limit of one file and under 40 lines.
  • Local parity with CI: partial (see section 2).

2. Assumptions and limitations

  • max_fix_scope is "one file, under 40 lines".
  • CI runs Python 3.11.8, and local is 3.11.8 (matches runtime_versions). CI also sets TZ=UTC, which I exported locally.
  • CI runs pytest -q --maxfail=3 after pip install -r requirements.lock. Without network access I used the existing venv and could not verify that installed versions match the lockfile.
  • The coverage upload step needs network access and was not run.

3. Failure analysis

First errorLocationClassificationEvidenceConfidence
AssertionError: Decimal('10.05') != Decimal('10.04')app/billing/invoice.py:58 (called from tests/test_invoice.py:112)Code regressiongit blame shows line 58 changed in a3f91c2 ("simplify rounding"). The test was last touched 5 months ago, and the expected value follows banker's rounding.High

The 2 later failures (test_export_csv, test_tax_summary) are cascading, since both reuse Invoice.total().

4. Reproduction log

AttemptCommandResult
1TZ=UTC pytest -q --maxfail=33 failed, 212 passed
2TZ=UTC pytest -q tests/test_invoice.py::test_total_roundingFailed with the same assertion, so it is deterministic

5. Changes made

FileChangeReason
app/billing/invoice.pyRestored Decimal.quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN) in total() and added the ROUND_HALF_EVEN importReverts the behavioral change from a3f91c2 that the tests encode
-from decimal import Decimal
+from decimal import Decimal, ROUND_HALF_EVEN
@@ def total(self):
-    return round(self._subtotal + self._tax, 2)
+    return (self._subtotal + self._tax).quantize(
+        Decimal("0.01"), rounding=ROUND_HALF_EVEN
+    )

git diff --stat shows 1 file changed, 4 insertions, 2 deletions, which matches this table.

6. Verification

CommandRan?Result
pytest tests/test_invoice.py::test_total_roundingYesPassed
pytest tests/test_invoice.py tests/test_export.py tests/test_tax.pyYes48 passed
TZ=UTC pytest -q --maxfail=3Yes215 passed
Coverage upload stepNo
Generated from this prompt by AI. Your results will vary.

Tips and common edits

Paste a trimmed log that includes the first error and some context rather than the full log. Run with codex exec from the repo root. If your CI uses containers or specific runtime versions, include them in {runtime_versions} so Codex can flag parity gaps.

Questions about this prompt

More in Codex

Codex Characterization-Test-First Refactor with Verification Handoff

Runs a behavior-preserving refactor in Codex exec. It pins current behavior with characterization tests, refactors in small verified steps inside an allowed scope, and ends with an evidence-based handoff for the merge decision.

Codex Exec CI Fixer: Diagnose, Minimal Fix, Rerun, Stop After 3 Attempts

Gives codex exec a non-interactive procedure to diagnose a failing CI output, apply a minimal fix, rerun the checks and stop after three attempts with a clear report.

Codex Read-Only Repo Investigation for a Bug Report

Runs a read-only Codex investigation that maps the relevant code paths, test commands, and likely root causes for a bug report. It ends with a ranked hypothesis table and a go/no-go recommendation before any edits.

Outcome-First Codex Task Writer for a Feature or Bug Request

Turns a raw feature or bug request into a paste-ready Codex task with a goal, constraints, files in scope, definition of done and exact verification commands.

Codex Pre-Merge Diff Review with Severity-Ranked Findings

Has Codex review a branch diff against a base ref in read-only mode. It checks correctness, tests, security, and maintainability, and returns evidence-backed findings with severity and a merge recommendation.

Codex Pull Request Review With Comments Ranked by Severity

Has Codex review a pull request and return comments sorted into blocking, should-fix and nit, each with the file and line, the risk, and a suggested change.

Codex Writes an AGENTS.md for This Repository From the Real Setup

Has Codex inspect your repository and write a short, accurate AGENTS.md with the real build, test and lint commands, conventions and boundaries, checked by actually running them.

Test-First Bug Fix Skill: Reproduce, Fix, Prove (SKILL.md)

A SKILL.md that forces your coding agent to reproduce a bug with a failing test before changing code, apply the smallest fix, and prove it with the test suite.

Agent Run Pre-Mortem and Self-Correction Loop Designer

Designs verification checks, error handling, and a self-correction loop for an agent task before it runs, based on likely failure modes.

Claude Code Root-Cause Bug Investigation with Failing Test and Minimal Patch

Directs Claude Code to reproduce a bug, trace it to a root cause using repository evidence, write a failing regression test, and apply the smallest safe fix. Ends with a verified diff, risks, and next actions.

Cursor Agent Bug Triage: Reproduce, Root-Cause, Fix, and Regression Test

Guides Cursor Agent through reproducing a bug, isolating the root cause with evidence, applying a minimal diff, and adding a regression test, ending with a verification checklist and risks.

Debug an Error From the Message and the Code Around It

Finds the most likely cause of an error from the message and the relevant code, ranks other possible causes, and gives the smallest fix plus a way to prove it worked.