Evaluate prompt consistency across repeated runs

Stress-test a prompt and evaluate output variability, edge-case drift, and formatting reliability. Use this before shipping prompts into production or multi-step automated workflows.

You getVariance Risk Analysis, deterministic Rewrite and 5-Run Consistency Test Protocol

Prompt

Prompt
Act as an AI QA test engineer. Evaluate the consistency and determinism of the pasted prompt across repeated execution runs.
- Identify structural ambiguities that permit the model to format or phrase answers differently each run.
- Spot subjective adjectives or open-ended verbs that cause variable reasoning paths.
- Define deterministic output controls that eliminate unwanted variance.
Output format:
1. Variance Risk Analysis (table with columns: Prompt Section, Source of Drift, Severity Risk High/Med/Low)
2. Deterministic Rewrite (tightened prompt engineered for identical outputs across runs)
3. 5-Run Consistency Test Protocol (specific checklist to measure output identity across five fresh runs in separate chats)

The prompt to evaluate is below:

Example output

Example for: a customer support ticket tagging prompt

  1. Variance Risk Analysis
Prompt SectionSource of DriftSeverity Risk
"Categorize appropriately"Unbounded taxonomy allows different category names per runHigh
"Provide brief reasoning""Brief" produces between 1 sentence and 3 paragraphsMed
"Flag urgent cases"Lacks concrete urgency thresholdsHigh
  1. Deterministic Rewrite
    Act as a support classification agent. Classify the customer ticket into exactly one of three categories: Billing, Technical Support, or Account Access.

Rules:

  • Output only valid JSON with keys: "category", "urgency", "reason".
  • "urgency" must be either "Standard" or "Critical". Mark "Critical" only if financial loss or service downtime is explicitly reported.
  • "reason" must be exactly one sentence of fewer than 20 words.
  1. 5-Run Consistency Test Protocol
  2. Start a new chat and run the test prompt against the exact same sample ticket. Repeat in a fresh chat until you have 5 runs.
  3. Verify that the "category" key matches identically in 5 out of 5 runs.
  4. Verify that the "urgency" determination does not flip across runs.
  5. Confirm word count in the "reason" field never exceeds 20 words across all runs.
  6. Confirm strict JSON schema parsing without syntax errors across all 5 runs.
Written by AI from this prompt. Your results will vary.

Paste your target prompt at the end of the text in ChatGPT, Claude, or Gemini. As a follow-up, ask the model to rewrite the prompt into a rigid JSON-only format for maximum automated reproducibility.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in AI Tools & Prompt Engineering
AI Tools & Prompt Engineering

Enforce strict refusal rules for missing context

Generate airtight prompt guardrails that stop AI models from guessing when source facts are absent. Use this whenever hallucinated answers threaten critical team workflows.