Decision support
What an AI agent audit should cover — and what to refuse to pay for
How to scope, price and sanity-check a reliability engagement before you sign anything.
The question
We are considering paying someone to audit our agent. What should we actually be getting?
The short answer
A useful audit produces four artefacts you keep: a failure taxonomy specific to your agent, a cost and latency profile, an eval suite wired to your real traffic, and a prioritised fix list with effort estimates. If a proposal does not name deliverables you own afterwards, you are buying a slide deck. Expect fixed scope and a fixed price — hourly billing on a diagnostic engagement transfers all the risk to you.
Written for
- VP Engineering or CTO approving the spend
- Founders buying technical help for the first time
- Engineering managers writing the internal business case
Why the answer is that
The failure mode of buying technical consulting is paying for observation. Someone spends two weeks reading your code and hands you a document describing problems you already suspected, with no mechanism to detect them recurring. It reads well, it is accurate, and nothing about your system has changed.
The test for whether an audit is worth buying is simple: what runs in your CI on the Monday after it ends? If the answer is nothing, you bought a report. The artefacts matter more than the analysis, because the artefacts are what keep working once the consultant leaves.
The second test is whether the work is scoped or open-ended. A diagnostic engagement has a knowable shape — there are only so many places an agent can be failing — so it can be quoted at a fixed price. An hourly quote on a two-week diagnostic means the person quoting has not decided what they are doing, and you are absorbing that uncertainty.
In scope, out of scope, and padding
Use this to read a proposal. The middle column is where most engagements quietly lose value.
| Item | Verdict | Why |
|---|---|---|
| Failure taxonomy specific to your agent | Insist on it | Named categories are what make failures countable. Without them you cannot prioritise or measure improvement. |
| Eval suite wired to your real traffic | Insist on it | The one deliverable that keeps paying after the engagement. If it is not included, ask why. |
| Cost and latency profile per operation | Insist on it | Usually finds recoverable spend that offsets a meaningful share of the fee. |
| Prioritised fix list with effort estimates | Insist on it | A list without effort estimates cannot be turned into a sprint plan, which means it will not be actioned. |
| Recorded walkthrough with your engineers | Insist on it | Knowledge transfer, and it protects you when the person who attended leaves. |
| Implementation of the fixes | Separate engagement | Reasonable to buy, but do not let it be bundled vaguely into a diagnostic — the scope and price should be quoted after findings, not before. |
| Model selection or benchmarking advice | Usually padding | Cheap to produce, rarely your actual problem, and it will be stale in three months. |
| Prompt rewriting as the headline deliverable | Red flag | Prompt tuning without an eval suite is unmeasurable by definition. If this is the main offer, the engagement cannot prove it worked. |
| A maturity model or capability matrix | Padding | Generates a slide, changes nothing. If it appears without the artefacts above, it is filling space. |
| Access to production credentials | Refuse | An agent can be audited from code, traces and anonymised samples. Anyone insisting on production write access should explain precisely why. |
A good heuristic: for every item in a proposal, ask "what do we still have in three months?" Anything that answers "a document" should be a small part of the price, not the substance of it.
How price should track scope
Not market rates — a logic for checking whether a quote is coherent. What should move a price is the amount of system and the amount of missing groundwork, nothing else.
| Situation | What that implies | Reasonable shape |
|---|---|---|
| One agent, traces already exist, clear question | Days of work, not weeks. Most of the time goes into evals. | Lower end of a fixed range, 1 week |
| One agent, no tracing at all | Instrumentation has to happen before anything can be measured. | Add several days; expect tracing as a named deliverable |
| Several agents or a multi-agent system | More surfaces, more tool boundaries, more failure paths. | Upper end of a fixed range, 2 weeks |
| Regulated data or a compliance deadline | Isolation and audit-trail work, and evidence you can show a reviewer. | Scoped separately — the deliverable is evidence, not just fixes |
| Quote varies with your company size | You are being priced on ability to pay, not on scope. | Ask what changes in the work. If nothing does, that is your answer. |
| Quote is hourly with an open end | The scope has not been decided; you carry the overrun risk. | Ask for a fixed price on a fixed scope instead |
For reference, my own audit is $4,000–6,000 for one to two weeks, and the range moves on scope only — not on where you are, how much you raised, or how much you appear able to pay.
Get the answer for your own system in four minutes
The free scorecard produces a score out of 100, a breakdown across reliability, cost and observability, and your gaps ranked by production risk with a fix and effort estimate for each. It prints to a PDF you can take into a planning meeting — which is usually more persuasive than an argument.
Run the scorecardIf you are the engineer who found this page: Written to be forwarded. If you are the engineer who found this, send the relevant one up — it is framed for the person who has to approve the work.
If you decide to do the work
The engagements this decision leads to, each with the full method and effort estimates — whether your team runs it or I do.
Questions
What should an AI agent audit cost?
It should be a fixed price tied to a fixed scope, and the number should move only with the amount of system being examined and how much groundwork is missing. My own is $4,000–6,000 for one to two weeks.
What should make you pause is a quote that changes with your company's size or funding, or an hourly rate with no ceiling on a diagnostic engagement — that means the scope has not been decided and you are absorbing the uncertainty.
How do I know if the audit was worth it?
Ask what runs in your CI the Monday after it ends. If the answer is an eval suite that fails the build when agent quality drops, you bought something durable. If the answer is nothing, you bought a report.
The secondary test is whether the fix list has effort estimates. A list of problems without estimates cannot be converted into a sprint plan, which means in practice it will not be actioned.
Can our own team do this instead of hiring someone?
Often yes, and you should check before spending anything. Run the free scorecard — it takes four minutes and produces a ranked gap list with a fix and an effort estimate for each.
If the gaps are clear and someone on your team has the time, do it internally. Hire outside help when the gaps are clear but nobody has the time, or when you want it done by someone who has done it before and will not spend three weeks learning.
Should we give a consultant access to production?
No, and be wary of anyone who requires it. An agent can be audited from source code, existing traces or logs, and anonymised samples of real inputs.
Read access to the relevant repositories and whatever observability exists is enough. If someone insists on production credentials, ask them to name the specific finding they cannot reach without them.
Want a second opinion on the decision?
Describe your situation and I will tell you what I would do — including when that is "nothing" or "handle it internally". No charge for that, and it is genuinely how a lot of these conversations end.