Decision support
Fourteen questions to ask your AI team before launch
With the answer that should reassure you, and the answer that should worry you.
The question
We are about to ship an agent to customers. What should I be asking, and what answers should worry me?
The short answer
Ask about four things, in this order: can you prove it still works after a change, can you explain a failure after the fact, is spend bounded, and does someone own it. The single most useful question is "if you changed a prompt today, what would tell you something broke?" — if the answer is "a customer would tell us", you are not ready to launch, regardless of how well the demo goes.
Written for
- Engineering leaders running a go-live review
- Founders shipping their first customer-facing agent
- Product leaders accountable for an AI feature
Why the answer is that
You are unlikely to be able to evaluate an agent's implementation directly, and you do not need to. What you can evaluate is whether the team has the machinery to know when it is wrong — and that is a set of questions with unambiguous good and bad answers.
The pattern to listen for is confidence without measurement. "It works really well" is not an answer to "how do you know". Teams with real evals answer with numbers and caveats. Teams without them answer with anecdotes, and often genuinely believe the anecdotes, because nothing has contradicted them yet.
Use these in a review rather than sending them as a document. The quality of the hesitation tells you as much as the answer does.
The fourteen questions
Grouped by what they protect. Read the middle column aloud if you want the conversation to go faster.
| Ask | Reassuring answer | Worrying answer |
|---|---|---|
| If you changed a prompt today, what would tell you something broke? | An eval suite runs in CI and blocks the merge below a threshold | "We'd test it manually" or "a customer would tell us" |
| Where did our test cases come from? | Sampled from real production traffic, including malformed inputs | "We wrote them" — team-authored cases miss what real users do |
| What is it supposed to refuse to do, and do we test that? | A refusal set exists and is part of the suite | A pause, then a description of what it is supposed to do |
| Can you show me a single bad run end to end? | Pulls up a trace with every step, prompt, tool call and token count | Greps application logs, or cannot reproduce it |
| What are the ways this fails, by name? | Named categories with rates attached | Individual anecdotes, described case by case |
| What does one conversation cost us? | A number, attributable per feature | The monthly provider invoice total |
| What stops a runaway loop over a weekend? | Per-request token caps, per-user limits, a tested global breaker | "We have billing alerts" — those fire after the money is gone |
| What is our p95 latency, not the average? | Per operation, with an alert threshold | An average, or "it feels fast" |
| What happens when a tool returns something unexpected? | Schema validation rejects it; the rejection is logged | "The model handles it" |
| Could this agent access data it should not? | Identity from the session, enforced in code, tested adversarially | "The prompt tells it not to" |
| Has anyone tried to break it on purpose? | Prompt injection and boundary cases are permanent test cases | "Not yet" |
| Which model version are we on, exactly? | A pinned version, logged per request | A floating alias — behaviour can change with no deploy |
| If a customer disputes an output next quarter, can we reconstruct it? | Trace retention covers it, including prompt version | "We'd look at the logs" |
| Who owns whether this works? | A named person with time allocated and metrics they report | "The team does" — which means nobody does |
You do not need every reassuring answer to launch. You do need to know which ones you are missing, and to have decided that deliberately rather than discovered it later.
The four you should not launch without
If you are short on time, these are the ones where a bad answer means real damage rather than a bad experience.
- 1Tool inputs are validated before execution — this is the failure mode that writes to the wrong record or emails the wrong customer, rather than merely giving a poor answer
- 2There is a hard ceiling on spend per request and per user, tested — not a billing alert, which notifies you after the money is gone
- 3Some kind of eval suite exists, even a small one, so a change can be shown not to have broken things
- 4One person is accountable for whether it works, with time allocated to it
Everything else on the longer list can be added after launch. These four are cheap, take days rather than weeks, and are the difference between a bad week and an incident with a customer's name on it.
Get the answer for your own system in four minutes
The free scorecard produces a score out of 100, a breakdown across reliability, cost and observability, and your gaps ranked by production risk with a fix and effort estimate for each. It prints to a PDF you can take into a planning meeting — which is usually more persuasive than an argument.
Run the scorecardIf you are the engineer who found this page: Written to be forwarded. If you are the engineer who found this, send the relevant one up — it is framed for the person who has to approve the work.
If you decide to do the work
The engagements this decision leads to, each with the full method and effort estimates — whether your team runs it or I do.
Questions
What should I ask my team before launching an AI agent?
Start with: "if you changed a prompt today, what would tell you something broke?" It is the single most diagnostic question, because the answer reveals whether any measurement exists at all. "A customer would tell us" means you are not ready.
Then cover the other three areas: can you reproduce a failure after the fact, is spend bounded by something that fails closed, and is one named person accountable for quality.
How do I evaluate AI work when I am not an AI engineer?
You do not evaluate the implementation, you evaluate whether the machinery to detect being wrong exists. That is entirely assessable without deep AI knowledge — you are asking about tests, tracing, cost ceilings and ownership, which are the same things you would ask about any production service.
Listen for confidence without measurement. "It works really well" is not an answer to "how do you know".
Is it ever reasonable to launch without an eval suite?
Yes — for a limited internal rollout, or a small pilot where you can read most outputs by hand and the cost of being wrong is low.
What is not reasonable is launching to customers without one and calling that a considered decision. If you are doing it, do it knowingly, with a date by which the suite exists, and treat the first two weeks of real traffic as the dataset you build it from.
Our demo is impressive. Why is that not enough?
Because a demo is a sample of inputs chosen by people who know how the system works. Real inputs are terse, misspelled, contradictory, multilingual and occasionally adversarial, and the gap between those two distributions is where agents fail.
An impressive demo tells you the capability exists. It tells you nothing about the failure rate on the inputs you have not imagined.
Want a second opinion on the decision?
Describe your situation and I will tell you what I would do — including when that is "nothing" or "handle it internally". No charge for that, and it is genuinely how a lot of these conversations end.