Cross-Model Prompt Evaluation Harness with Rubric Scoring and Cost Report

Direct Codex to build a script that runs one prompt suite across multiple models, scores outputs against a rubric, and produces a CSV and Markdown report that helps you choose a model.

You getDataset summary table, files created with one-line purposes and report template with sections

The prompt

Prompt
Role: You are a Codex agent acting as an ML evaluation engineer. You know how to compare OpenAI, Anthropic Claude, Google Gemini, xAI Grok, Meta Llama, DeepSeek, Mistral, Cohere, and Amazon Nova outputs fairly.

Context: We must decide which model to use for this task: classify inbound sales emails by intent and draft a two-sentence reply (example value for {task_description}). Models under test: GPT-4.1, Claude Sonnet, Gemini 2.5 Flash, Mistral Large, Amazon Nova Pro (example value for {models_under_test}). Test cases live at data/sales_emails_eval.jsonl (example value for {dataset_path}). The decision supported is: which model gives acceptable quality at the lowest cost and latency within a spend cap of 25 (example value for {budget_limit_usd}) USD.

Task: Build a runnable evaluation harness, a dry-run mode, and a report generator. Do not invent results. If no API keys are available, run only the dry-run mode with stubbed responses and clearly label the report as a dry run.

Inputs:
- Task description: classify inbound sales emails by intent and draft a two-sentence reply (example value for {task_description})
- Models under test: GPT-4.1, Claude Sonnet, Gemini 2.5 Flash, Mistral Large, Amazon Nova Pro (example value for {models_under_test})
- Dataset path: data/sales_emails_eval.jsonl (example value for {dataset_path})
- Rubric criteria: intent accuracy, tone fit, factual restraint, format compliance (example value for {rubric_criteria})
- Budget cap in USD: 25 (example value for {budget_limit_usd})

Workflow:
1. Inspect the dataset file format and report the number of cases and any malformed rows.
2. List assumptions about pricing, token counting, and scoring. Pricing must be loaded from an editable config file with a date field, not hardcoded.
3. Implement a runner that sends identical prompts to each model with the same temperature setting and records output, latency, input tokens, output tokens, and errors.
4. Implement a scorer with two parts: deterministic checks where possible (format, length, required fields) and an optional LLM-as-judge pass using a model that is NOT in the set being judged, with the judge prompt stored in a file. Flag the self-preference bias risk.
5. Estimate cost before running and abort if the estimate exceeds the budget cap.
6. Save raw results as JSONL and aggregated results as CSV.
7. Generate a Markdown report.

Constraints:
- Randomize model call order per case to reduce ordering effects.
- Cache responses by hash of model and prompt so reruns do not spend twice.
- Never print API keys. Read them from environment variables.
- Do not claim statistical significance without stating sample size.

Output format:
1. Dataset summary table.
2. Assumptions and missing information.
3. Files created with one-line purposes.
4. Report template with sections: Summary, Score by criterion table (model by criterion), Cost and latency table, Failure examples, Recommendation with confidence level, Risks, Next actions.
5. Exact commands to run dry-run and live modes.

Acceptance criteria:
- Dry run completes without network access.
- Aggregated CSV has one row per model per criterion.
- The report states the sample size and the cost estimate versus actual.
- The recommendation cites table values, not opinions.

Quality checks:
- Judge model is excluded from the candidates it scores.
- Pricing config has an as-of date and a note to verify it with each vendor.
- Failed calls are counted, not silently dropped.
- Limitations of the rubric are listed in the report.

Dashed parts are example values, so the prompt runs as copied. Change them in Fill in the blanks.

Keep your dataset to 20 to 50 realistic cases for a first pass. Review the judge prompt manually and spot-check at least five judged outputs before trusting scores. Update the pricing config before each run.

Keep exploring

More in Major AI Models