Evaluate models on your own tasks
Build a small test set, score outputs against a rubric, and compare revisions fairly.
Define a pass before you test
Pick a job you understand, such as extracting a task and due date from meeting notes. Write down observable criteria. “Sounds helpful” is hard to score consistently. “Keeps the correct date and uses null when missing” is easier to score.
| Input | Required behavior |
|---|---|
| “Send the agenda by Friday.” | Keep Friday attached to sending the agenda. |
| “Send the agenda when you can.” | Return a missing date. |
| “We met Monday. Send the agenda Friday.” | Choose the deadline Friday. |
| “We discussed the agenda.” | Return no action item. |
Run the same tests on each candidate
- Keep the prompt and examples fixed. Record model revision, runtime, context, and temperature for each run.
- Score each answer for format, factual accuracy, and required behavior. Record latency and memory separately.
- Label failure types, such as invented date, incorrect format, or false action item. Use these labels to decide what to change.
- Keep a second set of examples you have never used to adjust the prompt. Re-run those held-out examples before switching models.
Try it
Use the four rows above to compare two prompts or two small models. Report exact counts alongside any percentages. Add a multilingual example if your workflow needs it and an ambiguous example where abstaining is appropriate.
Repeat the comparison when you change model, quantization, prompt template, or runtime. Public benchmark scores can help you pick candidates. Your test set answers whether the system meets your own requirements.
Check your understanding
Why set aside examples before tuning the prompt?
Held-out examples show how the prompt works on inputs it was not adjusted to fit.
Sources
Progress is saved in this browser.