Spot formatting regressions across model updates

Audit outputs from upgraded AI models against your strict formatting rules. Use it to catch broken JSON, lost headers, or leaking chain-of-thought tokens.

You getStatus, regression Summary and defect Log

Prompt

Prompt
Act as a quality assurance engineer for LLM pipelines. Review the provided target formatting specification and the new model output to spot all regressions.

Rules:
- Compare line by line: check markdown headers, bullet styles, delimiter tokens, JSON validity, casing, and trailing text.
- Flag any added conversational filler, missing sections, unexpected prefixes, or schema violations.
- Quote the exact failing text snippet alongside the expected format rule.
- Do not assume missing keys are optional unless specified.

Output format:
- Status: PASS or FAIL
- Regression Summary: brief paragraph stating overall compatibility.
- Defect Log: a markdown table with columns Rule Checked, Observed Output, Severity (High/Medium/Low), and Recommended Prompt Fix.

Target specification and model output are below:

Example output

Example for: testing a strict JSON event extractor after a model upgrade

  • Status: FAIL
  • Regression Summary: The new model output violates the strict raw JSON requirement by wrapping the payload in markdown code fencing and adding conversational intro text before the opening brace.
  • Defect Log:
Rule CheckedObserved OutputSeverityRecommended Prompt Fix
Pure raw JSON output"Here is the parsed data: ```json..."HighAdd explicit instruction: 'Start output directly with the opening brace and end with the closing brace. Do not use markdown backticks.'
Strict timestamp format (ISO 8601)"timestamp": "2024-05-12 14:30"MediumSpecify regex pattern: 'Format timestamp strictly as YYYY-MM-DDTHH:MM:SSZ.'
No trailing commentary"Let me know if you need any adjustments!"HighAppend: 'Output must contain zero conversational tokens or conversational sign-offs.'
Written by AI from this prompt. Your results will vary.

In ChatGPT, Claude, or Gemini, paste your expected output format or schema first, followed by the raw text generated by the updated model. Ask for modified prompt instructions to patch high-severity failures.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in AI Tools & Prompt Engineering
AI Tools & Prompt Engineering

Build a state sheet for roleplay characters

Create a persistent persona state sheet that prevents roleplay models from breaking character. Use it for training simulations, scenario games, or creative writing.