Reliability · Agent eval suites
Build an eval suite that catches regressions before your users do
Stop shipping prompt changes as unmeasured bets.
You probably have this problem if
- Someone changed a prompt last week and nobody can say whether it helped
- Quality discussions are arguments about anecdotes rather than numbers
- You found out about a regression from a customer, not from CI
- Your "tests" are a notebook one engineer runs by hand before a release
- You are afraid to upgrade the model because you cannot tell what would break
What you end up with
A suite your team owns that fails the build when agent quality drops.
- Usually part of
- Agent Production Readiness Audit
- $4,000 – $6,000
- Phases
- 5 steps, detailed below with effort per step
- Domains I have done this in
- HR analyticsRecruitmentWorkflow automation
Why this is the thing to fix
An agent has no compiler. Nothing tells you that a two-word prompt change broke tool selection on 8% of inputs — you find out when a customer complains, three weeks later, with no way to attribute it to the change that caused it.
This is the gap that makes every other improvement unsafe. You cannot route models to cut cost if you cannot prove quality held. You cannot upgrade to a newer model. You cannot refactor a prompt with confidence. An eval suite is not a testing nicety; it is the thing that makes the rest of the work possible.
It is also the single most common thing missing. Most teams I look at have zero automated evals on agent output, while having thorough unit tests on the glue code around it.
The first 30 eval cases, by category
Teams stall on "how many cases do we need?" Thirty real ones, distributed like this, catches most of what matters. Sample from actual production traffic — team-authored cases systematically miss the malformed middle of the distribution.
| Category | Cases | What it catches |
|---|---|---|
| Happy path | 6 | Core behaviour. The baseline that must never regress. |
| Malformed input | 5 | Typos, fragments, no punctuation, wrong language. The most common real traffic and the least tested. |
| Ambiguous requests | 4 | Where the agent should ask rather than guess. Catches confident wrong answers. |
| Must-refuse | 4 | Out of scope, unsafe, or not authorised. What it declines is as much a requirement as what it does. |
| Adversarial / injection | 3 | Prompt injection, instruction override, data exfiltration attempts. |
| Long conversation | 3 | 30+ turns. Instruction drift only appears at length, and almost nobody tests at length. |
| Tool-call edge cases | 3 | Missing records, permission denied, empty results, timeouts. |
| Known past failures | 2+ | Every production incident becomes a permanent case. This category only grows. |
Grow the last row forever and stop growing the others. A suite that only ever gets happy-path cases added is a suite that stops finding bugs.
How to score each case
The scoring method matters more than the case count. LLM-as-judge is overused — it is slow, costs money on every CI run, and is itself non-deterministic. Use the cheapest method that can actually detect the failure.
| Method | Use for | Cost per run |
|---|---|---|
| Exact / schema assertion | Structured output, tool name, tool arguments, refusal flags | Free, instant, deterministic |
| Deterministic check in code | Numbers, dates, IDs, citation present, PII absent | Free, instant |
| Retrieval recall@k | Whether the right document was fetched — measured separately from the answer | Free once labelled |
| LLM-as-judge | Tone, helpfulness, faithfulness to source — genuinely subjective qualities only | Real money on every CI run; use a cheap model and pin the version |
| Human review | Calibrating the judge, and a small sample each release | Expensive; budget a fixed sample size |
Rule of thumb: if a check can be written as an assertion, it must be. Reach for a judge only when the property is genuinely subjective.
How I actually do it
With effort per step, so you can judge whether to hire me or hand this to someone on your team.
- 10.5 day
Sample real traffic
Pull a stratified sample of real inputs — including the confusing ones — and anonymise where needed. If there are no logs to sample, that is finding number one and we fix it first.
- 21 day, with your team
Label expected outcomes
Not expected wording — expected decisions, tool calls, and refusals. This is the part that needs someone who knows the domain, so we do it together.
- 31.5 days
Build the harness
Runs the suite against your agent, scores by the cheapest adequate method per case, and emits a per-category report rather than one aggregate number.
- 40.5 day
Wire into CI with a threshold
A pass threshold that blocks merges, and a separate softer threshold that warns. An eval suite nobody is forced to look at decays within a month.
- 51 day
Close the loop from production
One-click capture from a trace to a new eval case. If adding a case is manual it stops happening the first week the team is busy.
What you keep
- 30+ labelled eval cases drawn from your real traffic
- A scoring harness with per-category reporting, not one aggregate score
- CI integration with a pass threshold that blocks merges
- A capture path so production failures become permanent cases
- A short written guide so your team can extend it without me
Failure modes this prevents
From the failure taxonomy — each links to the causes and fixes in full.
high · very common
It works on our test prompts and fails on real users
The team's test inputs are clean, well-formed, and written by people who know how the system works.
medium · common
The same question gives different answers each time
Some variation is inherent to sampling.
medium · common
The agent ignores its instructions once the conversation gets long
Early turns follow the rules; by turn twenty the agent has drifted — wrong tone, abandoned constraints, forgotten refusals.
Questions
How many eval cases do we actually need?
Thirty real cases, distributed across the categories in the table above, catches most of what matters. Case count is the wrong metric to optimise — coverage across failure categories is what finds bugs.
Three hundred happy-path cases will find less than thirty well-chosen ones that include malformed input, ambiguity, refusals and long conversations.
Should we use LLM-as-judge for evaluation?
Only for genuinely subjective properties like tone or helpfulness. It is slow, costs money on every CI run, and is itself non-deterministic — which means your test suite has flaky tests by construction.
Anything that can be an assertion should be an assertion. In practice about 70% of useful checks are deterministic: schema validity, tool name, argument correctness, citation presence, PII absence, refusal behaviour.
Can we build an eval suite without production traffic yet?
Yes, but expect it to be weaker. Hand-authored cases systematically miss the inputs you would never think to write, which is exactly where agents fail.
Start with the categories above, ship behind a limited rollout, and treat the first two weeks of real traffic as your real dataset. Prioritise the logging that makes that possible before launch, not after.
How long does this take to build?
About 4–5 days for a first suite that pays for itself immediately, assuming some logs exist to sample from. That is the estimate I quote and the sequence is in the method above.
If there is no logging at all, add 2–3 days for the tracing needed to sample real traffic — see the observability engagement.
Want this done, or just want a second opinion on it?
The method above is genuinely what I do — if your team can run it themselves, run it. If you would rather it were done in a week by someone who has done it before, that is what the agent production readiness audit is for.