Agent skill · Coding

Eval set builder

Writes a small test set for a prompt, skill or workflow: 8 to 15 inputs covering typical, edge and adversarial cases, a pass or fail rubric for each, and a way to run them by hand or as a Promptfoo config. Use when asked to test, evaluate or regression-check a prompt, a skill or an agent before changing it.

The eval-set-builder/ folder with SKILL.md.

The skill itself. Agents read the name and description first and load the rest only when a task matches.

eval-set-builder/SKILL.md
---
name: eval-set-builder
description: "Writes a small test set for a prompt, skill or workflow: 8 to 15 inputs covering typical, edge and adversarial cases, a pass or fail rubric for each, and a way to run them by hand or as a Promptfoo config. Use when asked to test, evaluate or regression-check a prompt, a skill or an agent before changing it."
---

# Eval set builder

Build the smallest set of test cases that would catch the prompt getting worse.

## Steps

1. Read the prompt, skill or workflow. Write one sentence on what a good output does, and list the ways it could fail (wrong format, made-up facts, ignored constraint, unsafe compliance, too long).
2. Write 8 to 15 cases: about half typical inputs, a third edge cases (empty, very long, another language, missing fields, conflicting instructions) and the rest adversarial (attempts to override the instructions, requests it should refuse).
3. For each case write: id, input, what a passing output must do (2 to 4 checkable points), and what fails it outright.
4. Prefer checks a script can do (contains, doesn't contain, valid JSON, under N words). Mark the ones that need a person or a model to judge.
5. Output the set as a table, then as a promptfooconfig.yaml with one test per case. Use `contains`, `not-contains`, `is-json` and `llm-rubric` asserts.
6. Say how to run it: by hand (paste each input, tick the checks) or `npx promptfoo eval` with the config.

## Rules

- Every case tests one thing. If a case checks five things, split it.
- Use realistic inputs, not "test input 1". Invent names and data; never use real customer data.
- Keep the expected behaviour in the rubric, not a full expected output, unless the output is fixed (a classification label, JSON fields).
- Note which cases the current prompt fails, if you can run it.

## Example row

| id | input | pass if | fail if |
| edge-empty | (empty message) | asks for the missing text in one line | invents content |