AI to Production · tool 4 of 9
Evaluating AI Quality
Evaluating AI Quality: replaces “it looks good” with evidence.
Best for: Before and after every change · Time: Days to set up
Domain: Across all domains · Focus area: Executing
How it works
An evaluation set is a collection of real examples with agreed good answers. Run it on every change of model, prompt or data. It’s to AI what regression tests are to software.
- Use real cases, including hard ones.
- Have experts score them.
- Track results across versions.
How to apply it
- Collect 50–200 real examples.
- Agree a scoring rubric.
- Run a baseline.
- Block releases that get worse.
In practice
A prompt change improves average scores, but the set shows refund cases got worse. It doesn’t ship.
Pro tip
Add every production failure to the evaluation set. It gets stronger each time something goes wrong.
Use it
- Template: AI Evaluation Set (blank), filled example
- Playbook: Our AI pilot is stuck
- Case study: Air Canada Chatbot
Quick check
When should you re-run the evaluation?
Sources
Last reviewed 2026-11-02 against the PMBOK® Guide, 8th edition · Practitioner review pending · How this guide is made