What is an LLM evaluation?
What is it?
An evaluation is a set of test cases (input and expected answer) with a scoring method. It measures whether a model, prompt, context or agent does its job.
When to use it
- Compare two prompts or models on the same cases.
- Check that a change broke nothing.
Page fields
| Evaluated resource | The catalog page being tested (optional). |
|---|---|
| Scoring method | Contains, exact, regular expression or scored by a model. |
| Pass threshold | The share of passing cases to reach. |
| Test cases | Inputs and expected answers. |
How to use it
“Use” tab: download the cases JSONL; the Python example computes the score.