Model Routing Config and Router Script Based on Cost, Latency, and Data Sensitivity

Use Codex to produce a rules-based model router with YAML config, fallback chains, and tests, so each workload goes to a model that fits budget, latency, context needs, and data-handling rules.

You getRouting config file contents summary, decision table and code files changed

The prompt

Prompt
Role: You are a Codex agent acting as a platform engineer who designs LLM routing layers across OpenAI, Anthropic Claude, Google Gemini, xAI Grok, Meta Llama, DeepSeek, Mistral, Cohere, and Amazon Nova.

Context: Our workloads are: ticket triage, contract clause extraction, internal code review comments, marketing copy drafts (example value for {workload_types}). Candidate models: GPT-4.1 mini, Claude Sonnet, Gemini 2.5 Pro, Llama 3.3 70B self-hosted, Mistral Small (example value for {candidate_models}). Latency target per request: 2500 (example value for {latency_target_ms}) ms at p95. Monthly budget: 3000 (example value for {monthly_budget_usd}) USD. Data sensitivity rules: customer PII and contracts may only go to self-hosted Llama or vendors with a signed DPA (example value for {data_sensitivity}). The decision to support is which model each workload should use by default and what the fallback is.

Task: Build a deterministic router module, a YAML routing config, and tests. Use rules first; do not add a model-based classifier unless you justify it. Do not state model capabilities or prices as verified facts. Put them in a config file marked needs verification, with fields for source and as-of date.

Inputs:
- Workload types: ticket triage, contract clause extraction, internal code review comments, marketing copy drafts (example value for {workload_types})
- Candidate models: GPT-4.1 mini, Claude Sonnet, Gemini 2.5 Pro, Llama 3.3 70B self-hosted, Mistral Small (example value for {candidate_models})
- Latency target: 2500 (example value for {latency_target_ms})
- Monthly budget: 3000 (example value for {monthly_budget_usd})
- Data sensitivity rules: customer PII and contracts may only go to self-hosted Llama or vendors with a signed DPA (example value for {data_sensitivity})

Workflow:
1. Ask for or list missing information, such as expected request volume, average tokens per request, and region requirements. Proceed with labeled assumptions.
2. Create a model capability table in config: context window, supports tools, supports structured output, supports vision, approved for sensitive data (true, false, unknown), price per million input and output tokens, as-of date.
3. Define routing rules in priority order: hard filters first (sensitivity, required features, context length), then scoring by estimated cost and latency fit.
4. Implement a function that takes a request descriptor and returns a primary model, a fallback chain, and a reason string.
5. Implement a monthly cost projection function using volume assumptions and print a table per workload.
6. Write tests covering: a sensitive request that must avoid unapproved models, an oversized context request, a budget-exceeded scenario, and an unknown capability value.
7. Run the tests and report real results.

Constraints:
- Treat unknown as not approved for sensitive data.
- The router must be pure and testable, with no network calls.
- Every routing decision must return a human-readable reason.

Output format:
1. Assumptions and missing information.
2. Routing config file contents summary.
3. Decision table: workload, primary model, fallback chain, reason, projected monthly cost.
4. Code files changed.
5. Test results.
6. Risks, including stale pricing, capability drift, and vendor data-policy changes.
7. Next actions checklist with owners left blank.

Acceptance criteria:
- Same input always returns the same routing decision.
- Sensitive workloads never route to unknown or unapproved models.
- Projected cost per workload and total are shown against the budget.
- All tests pass or failures are explained.

Quality checks:
- No capability or price is presented without an as-of date and a verify note.
- Fallback chains contain no duplicates.
- Reason strings reference the rule that fired.
- Assumptions are listed apart from verified repo facts.

Dashed parts are example values, so the prompt runs as copied. Change them in Fill in the blanks.

Fill capability and price fields from each vendor's current documentation and data-processing terms before relying on the output. Have security or legal review the sensitivity approvals. Re-run the cost projection when volumes change.

Keep exploring

More in Major AI Models