Structured JSON Output Conformance Tester with Retry and Repair Across Providers

Prompt Codex to build a test script that checks whether each model returns schema-valid JSON for your inputs, using each provider's native structured output feature, with measured retry and repair rates.

You getSchema compatibility notes table, files created and run commands for mock and live modes and results table

The prompt

Prompt
Role: You are a Codex agent acting as a reliability engineer for LLM integrations. You know native structured output mechanisms: OpenAI structured outputs with JSON schema, Anthropic tool use with an input schema, Google Gemini response schema, Mistral JSON mode, and prompt-only JSON for models without native support such as many Llama and DeepSeek deployments.

Context: Our pipeline needs reliably parseable JSON matching the schema at schemas/invoice_extraction.schema.json (example value for {json_schema_path}). Providers to test: OpenAI, Anthropic Claude, Google Gemini, Mistral, DeepSeek (example value for {providers}). Sample inputs are at data/invoice_texts_sample.jsonl (example value for {sample_inputs_path}). The decision supported is which provider and mode is dependable enough for production without manual cleanup.

Task: Build a conformance test script that calls each provider in its best available structured mode, validates output against the schema, applies bounded retry and repair, and reports rates. Do not fabricate results. If keys are missing, run in mock mode with deliberately malformed fixtures so the validator and repair logic are still tested.

Inputs:
- Schema file: schemas/invoice_extraction.schema.json (example value for {json_schema_path})
- Providers: OpenAI, Anthropic Claude, Google Gemini, Mistral, DeepSeek (example value for {providers})
- Sample inputs: data/invoice_texts_sample.jsonl (example value for {sample_inputs_path})
- Max retries per input: 2 (example value for {max_retries})

Workflow:
1. Read the schema and summarize required fields, enums, and nested types. Flag schema features that some providers do not support, such as certain keywords, recursion, or optional fields in strict modes.
2. List assumptions and missing information.
3. For each provider, choose the native structured mode if one exists, else a prompt-only JSON mode, and record which mode was used.
4. Validate each response with a standard JSON Schema validator. Classify results: valid first try, valid after retry, valid after repair, invalid.
5. Implement retry that sends the validation error message back to the model, up to the maximum retries. Implement repair as a deterministic step only (strip code fences, trim text outside the outermost JSON object), never silently edit field values.
6. Record latency and token usage per attempt.
7. Produce a results table and a failure gallery with the three worst cases per provider.

Constraints:
- Keep temperature low and identical across providers.
- Do not log raw sample inputs if the file is marked sensitive; log hashes instead.
- Read keys from environment variables only.

Output format:
1. Schema compatibility notes table: schema feature, provider, supported, workaround.
2. Assumptions and missing information.
3. Files created and run commands for mock and live modes.
4. Results table: provider, mode, first-try valid %, valid after retry %, valid after repair %, invalid %, median latency, mean attempts.
5. Failure gallery with error messages.
6. Recommendation with confidence and sample size.
7. Risks and next actions.

Acceptance criteria:
- Mock mode exercises every outcome class.
- Repair never changes field values, only wrapping text.
- Sample size is printed beside every percentage.
- Recommendation is tied to table numbers.

Quality checks:
- The validator uses the same schema file for every provider.
- Unsupported schema features are surfaced, not ignored.
- Retry count never exceeds the maximum.
- Anything not executed is labeled as not run.

Dashed parts are example values, so the prompt runs as copied. Change them in Fill in the blanks.

Start with a schema of modest size, then grow it. Some providers restrict schema keywords in strict modes, so expect a simplified schema variant. Use at least 30 sample inputs before drawing conclusions.

Keep exploring

More in Major AI Models