Is your AI agent production-ready, or just working today?

No regression safety net on agent behaviour

Every prompt tweak is an unmeasured bet. You will ship a regression, hear about it from a customer, and have no way to tell which change caused it.

Fix: Freeze 30–50 real inputs with expected outcomes, score them with a mix of assertions and LLM-as-judge, and gate merges on the aggregate.

Effort: 3–5 days for a first suite that pays for itself immediately · Risk if unaddressed: critical

Evals do not reflect real user behaviour

Your suite passes while real users hit failures nobody imagined. Confidence goes up, quality does not.

Fix: Add a one-click 'capture this run as an eval case' path from your trace viewer, and make triaging new failures a weekly ritual.

Effort: 2–3 days once tracing exists · Risk if unaddressed: high

Prompt changes are untraceable

Debugging becomes archaeology. You cannot reproduce a reported failure, so you cannot confirm a fix.

Fix: Treat prompts as versioned artefacts in git, stamp the version onto every trace, and make rollback a deploy-free operation.

Effort: 1–2 days · Risk if unaddressed: high

Hallucinated tool arguments reach real systems

This is the failure mode that causes actual damage rather than just a bad answer — wrong record updated, wrong customer emailed, wrong row deleted.

Fix: Validate every tool input against a strict schema, re-check authorisation server-side rather than trusting the agent, and require confirmation for irreversible actions.

Effort: 2–4 days, and the highest-value work on this list · Risk if unaddressed: critical

Unit economics are invisible

You cannot price your product, forecast spend, or tell which feature is quietly consuming the budget. Cost surprises arrive monthly.

Fix: Log token counts and computed cost on every call with feature and tenant tags, then chart cost per conversation as a first-class product metric.

Effort: 2–3 days · Risk if unaddressed: high

No circuit breaker on spend

One looping agent, one adversarial user, or one bad deploy can produce a five-figure bill over a weekend. Alerts notify you after the money is gone.

Fix: Enforce max token caps per request, rate limits per user per day, and a global kill switch that degrades to a cheaper model or a queue rather than spending without bound.

Effort: 2–3 days · Risk if unaddressed: critical

Paying premium rates for trivial work

Usually the single largest recoverable line item — often 40–70% of spend on sub-tasks a much cheaper model handles at equal quality.

Fix: Classify tasks by required capability, route accordingly, add prompt caching on stable system context, and verify quality per tier with your eval suite.

Effort: 3–5 days, typically the fastest measurable ROI · Risk if unaddressed: medium

Tail latency is unmeasured

The average looks acceptable while a meaningful slice of users waits 40 seconds and abandons. Retries and multi-step chains live entirely in that tail.

Fix: Instrument per-step timing, alert on p95 by operation, and set an explicit latency budget that streaming or partial results must satisfy.

Effort: 2 days · Risk if unaddressed: medium

No per-run trace

Every investigation starts from zero. Mean time to diagnosis is measured in days, and some failures are simply never explained.

Fix: Adopt step-level tracing (OpenTelemetry or a purpose-built LLM observability tool), propagate one trace ID across the whole run, and make traces searchable by user and outcome.

Effort: 3–5 days · Risk if unaddressed: critical

Failures cannot be counted, so they cannot be prioritised

Without categories there are no rates. The team fixes whatever was complained about loudest rather than what happens most.

Fix: Name your failure modes, tag every incident against them, and review the distribution monthly. Fix by frequency times severity.

Effort: 1–2 days to define, ongoing to maintain · Risk if unaddressed: high

Guardrails rely on the model following instructions

Prompt-based rules are advisory, not enforcement. One jailbreak or one unusual input and unreviewed content — or someone else's data — reaches a user.

Fix: Enforce checks in code outside the model: PII detection, policy classifiers, and grounding verification against retrieved sources. Log every block as a signal.

Effort: 3–5 days depending on regulatory surface · Risk if unaddressed: high

Agent quality is nobody's job

Reliability work loses to feature work every sprint, because features have an owner arguing for them and reliability does not.

Fix: Name an owner, give them explicit time, and have them report a small number of reliability metrics on the same cadence as product metrics.

Effort: An organisational decision, not an engineering one · Risk if unaddressed: high