The agent calls the right tool with invented arguments
The model picks a correct tool but fabricates its inputs — a plausible-looking customer ID, an out-of-range date, an enum value that was never defined. This is the failure mode most likely to cause real-world damage rather than merely a bad answer, because the call succeeds against a real system.
Also described as: agent hallucinates tool arguments · LLM passes wrong parameters to function call · agent invents IDs that do not exist
Root causes — check in this order
1. No schema validation between model output and execution
Model output is text. If nothing validates it before it becomes a function call, any string reaches your system.
How to check: Trace one tool call end to end. Count the validation steps between the model's response and the side effect. If the answer is zero, this is your problem.
2. Tool descriptions are ambiguous about identifier format
If the description says "the customer ID" without stating the shape, the model infers a plausible one from context — often from an unrelated example in the conversation.
How to check: Re-read your tool schemas as if you were a new engineer. Any field whose format is not obvious will eventually be guessed.
3. Authorisation is trusted from the agent rather than re-checked
An agent asked to fetch "my invoices" may pass a different tenant's ID. If the tool trusts the argument, isolation is broken by a hallucination.
How to check: Ask whether a tool call could reference another tenant's data and still succeed. Test it deliberately.
Fixes that hold
Validate every tool input against a strict schema
1–2 daysReject rather than coerce. A hard failure the agent can retry against is safer than a silently corrected value that produces a confidently wrong result.
Re-derive identity and authorisation server-side
1–3 daysNever accept tenant or user identifiers from model output. Take them from the authenticated session and ignore what the agent supplied.
Require confirmation or dry-run for irreversible actions
2–4 daysDeletes, payments, and outbound messages should either return a preview for confirmation or run against a staging path first.
Log every rejected call as a quality signal
Half a dayRejection rate per tool is one of the most useful agent health metrics, and almost nobody tracks it.
Often appears alongside: Output is cut off mid-way and nothing reports an error · The same question gives different answers each time
The agent loops, repeating the same step until something kills it
The agent calls a tool, dislikes the result, and calls it again with near-identical arguments — indefinitely. Every iteration is billed. Discovered either via a timeout or via an invoice.
Also described as: AI agent infinite loop · agent repeats the same tool call · agent never terminates
Root causes — check in this order
1. No maximum step count per run
Agent loops are while-loops with a language model as the condition. Without a hard bound, a model that never reports satisfaction never exits.
How to check: Find the max-iterations value in your orchestration code. If there is not one, this will happen to you.
2. A tool returns an error the model cannot act on
An opaque "500 Internal Server Error" gives the model nothing to change, so it retries the identical call, reasoning that it may work this time.
How to check: Look at what your tools return on failure. Errors should say what was wrong and what to do differently.
3. No progress detection between steps
Nothing notices that step 14 is identical to step 11. Repetition is the clearest possible signal of a stuck agent and it usually goes unmonitored.
How to check: Hash each tool call and its arguments. Alert when the same hash repeats within a run.
Fixes that hold
Hard step ceiling per run, plus a wall-clock timeout
Half a dayBoth, not either. A ceiling bounds cost, a timeout bounds user-visible latency, and they fail in different situations.
Return actionable errors from tools
1–2 days"Customer 4471 not found — search by email instead" gives the model a different action. "Error 500" does not.
Detect repetition and break out deliberately
1 dayOn the second identical call, inject a message naming the repetition and requiring a different approach. On the third, stop and escalate.
Cap spend per run, not just per month
1 dayA per-run token budget turns an unbounded incident into a bounded, logged, and alertable one.
Often appears alongside: Spend jumped several times over with no corresponding traffic increase · Average latency looks fine but some users wait forty seconds
Spend jumped several times over with no corresponding traffic increase
Month-over-month spend multiplies while usage looks flat. Almost always caused by per-call token growth rather than more calls — and invisible if you only monitor request counts.
Also described as: LLM costs suddenly increased · unexpected OpenAI bill · agent token usage spike
Root causes — check in this order
1. Conversation history grows without bound
Resending the whole transcript each turn makes per-turn cost grow linearly with turn count. A 40-turn conversation can cost twenty times a 5-turn one.
How to check: Chart input tokens against turn index. A rising line is your answer.
2. Retrieval started returning more or larger chunks
A re-indexed corpus or a changed top-k silently multiplies input tokens on every single call.
How to check: Compare average retrieved-context length before and after your last indexing change.
3. Tool definitions grew
Every tool schema is billed on every call. Adding eight verbose tools raises the floor cost of all traffic, including calls that use none of them.
How to check: Count tokens in your serialised tool definitions. Compare to average input tokens per call.
4. A silent fallback to a more expensive model
Rate limits or errors on a cheap model can fall through to an expensive one. Behaviour looks correct; unit cost changes by an order of magnitude.
How to check: Group spend by model ID. Any traffic on a model you did not intend to use is the finding.
Fixes that hold
Track cost per conversation as a product metric
2–3 daysNot monthly total — per unit of work, on the same dashboard as your other product metrics, so drift is visible in days rather than at invoice time.
Bound history explicitly
2–4 daysSliding window, summarisation of older turns, or both. Choose deliberately; unbounded is not a choice, it is an omission.
Cache stable prompt prefixes
1–2 daysSystem prompts and tool definitions are ideal candidates. A cache read costs about a tenth of base input — but only pays off once content is genuinely re-read.
Alert on cost per conversation, not spend
1 dayA budget alert fires after the money is gone. A unit-cost alert fires while it is still a bug.
Often appears alongside: The agent loops, repeating the same step until something kills it · The agent ignores its instructions once the conversation gets long
It works on our test prompts and fails on real users
The team's test inputs are clean, well-formed, and written by people who know how the system works. Real inputs are terse, misspelled, multilingual, contradictory, or adversarial. The gap between those two distributions is where agents fail.
Also described as: agent works in testing but not production · LLM fails on real user input · demo works but production does not
Root causes — check in this order
1. Eval cases are all team-authored
You cannot imagine the inputs you would never write. Hand-authored suites systematically omit the malformed middle of the distribution.
How to check: Sample 50 real production inputs at random. How many resemble anything in your test set?
2. No eval suite at all — testing is manual
Manual testing checks whether it works today, not whether it still works after the next change.
How to check: If a prompt changed right now, what would tell you something broke, other than a customer?
3. Happy-path-only coverage
Empty inputs, contradictory requests, prompt injection attempts, and questions outside scope are all normal traffic and rarely tested.
How to check: Does your suite contain a single case the agent is supposed to refuse?
Fixes that hold
Build the suite from real traffic
3–5 daysSample real inputs across the distribution, including the confusing ones. Thirty real cases beat three hundred invented ones.
Make production failures flow back automatically
2–3 daysOne click from a trace to a new eval case. If capturing a failure is manual, it will not happen once the team is busy.
Test refusals and edge cases as first-class requirements
2 daysWhat the agent must decline is as much a requirement as what it must do, and needs the same coverage.
Often appears alongside: The same question gives different answers each time · The answer is wrong because retrieval returned the wrong context
The agent ignores its instructions once the conversation gets long
Early turns follow the rules; by turn twenty the agent has drifted — wrong tone, abandoned constraints, forgotten refusals. Instructions compete with an ever-growing transcript for the model's attention.
Also described as: agent forgets system prompt · LLM loses instructions in long context · agent drifts over many turns
Root causes — check in this order
1. Instructions appear once, at the very start
As the transcript grows, a single early instruction becomes a smaller and more distant fraction of the input.
How to check: Reproduce with a 30-turn conversation. If compliance decays with turn count, this is it.
2. Naive truncation drops the instructions themselves
A sliding window that keeps the last N messages will eventually slide the system prompt out entirely.
How to check: Print the exact final payload at turn 40 and confirm your constraints are still in it.
3. Summarisation loses constraints while keeping content
Summarisers preserve narrative and discard rules, because rules read as boilerplate.
How to check: Inspect a generated summary. Are the operating constraints still present?
Fixes that hold
Re-assert critical constraints late in the payload
Half a dayKeep hard rules structurally pinned near the end of the input rather than only at the beginning.
Separate durable state from conversational history
3–5 daysConstraints, entities, and decisions belong in a structured state object rebuilt each turn — not left to survive inside a transcript.
Enforce the important rules in code
2–4 daysIf a rule genuinely must hold, verify the output rather than trusting the instruction. Prompts are advisory; code is enforcement.
Add long-conversation cases to your evals
1–2 daysMost suites test three-turn exchanges. Drift only appears at length, so test at length.
Often appears alongside: Spend jumped several times over with no corresponding traffic increase · The same question gives different answers each time
The answer is wrong because retrieval returned the wrong context
The model is behaving correctly given what it was handed — and what it was handed was wrong. Attributing this to hallucination sends teams to tune prompts when the defect is in retrieval.
Also described as: RAG returns irrelevant chunks · vector search bad results · agent answers from wrong document
Root causes — check in this order
1. Chunking split the answer across boundaries
Fixed-size chunking cuts tables, lists, and procedures in half. Neither half retrieves well and neither answers the question.
How to check: Search for a question you know the answer to. Read the retrieved chunks. Is the answer actually in them?
2. Semantic similarity is not relevance
"How do I cancel?" is embedding-similar to a page about cancellation fees. Similar topic, wrong answer.
How to check: Measure retrieval separately from generation. Report recall@k against known-good documents.
3. Stale or duplicated index
Outdated documents retrieve just as confidently as current ones, and near-duplicates crowd out the single correct source.
How to check: Check the newest document in your index against the newest in the source system.
4. No grounding check on the output
Nothing verifies that the answer is actually supported by the retrieved text, so an unsupported claim ships looking identical to a supported one.
How to check: Sample answers and verify each claim against its cited chunk.
Fixes that hold
Measure retrieval as its own component
3–5 daysBuild a labelled set of question-to-document pairs and track recall@k. Retrieval quality is invisible when only end-to-end answers are scored.
Chunk along document structure, not byte counts
2–4 daysSplit on headings, sections, and table boundaries. Keep a parent reference so a matched chunk can expand to its full context.
Add hybrid search and reranking
3–5 daysKeyword search catches exact identifiers that embeddings miss; a reranker fixes ordering. Together they typically move recall more than any prompt change.
Verify grounding before returning the answer
2–4 daysCheck that each claim is supported by retrieved text. Say "I could not find this" rather than answering unsupported — the single largest trust win available.
Often appears alongside: It works on our test prompts and fails on real users · The agent ignores its instructions once the conversation gets long
Average latency looks fine but some users wait forty seconds
Multi-step agents have long tails by construction: each step adds latency, and retries multiply it. Monitoring averages hides exactly the runs where users abandon.
Also described as: agent slow for some users · LLM p95 latency high · multi-step agent takes too long
Root causes — check in this order
1. Step count varies widely per request
A two-step run and a fourteen-step run average out to something no user experiences.
How to check: Plot the distribution of steps per run. If it has a long right tail, so does your latency.
2. Retries are serial and invisible
Three retries with backoff can add tens of seconds. If retries are not traced, the time appears to vanish into the model call.
How to check: Count retries per run and attribute time to them explicitly.
3. Sequential tool calls that could run concurrently
Independent lookups executed one after another add up linearly for no reason.
How to check: Look for consecutive tool calls with no data dependency between them.
4. Nothing streams, so the user sees nothing
Even correct latency feels broken with no feedback. Perceived and actual latency are different problems.
How to check: Time to first visible token. If it equals total time, you are not streaming.
Fixes that hold
Alert on p95 and p99 per operation
1–2 daysPer operation, not globally — a slow rare path is invisible in an aggregate.
Parallelise independent tool calls
2–3 daysOften the largest single latency win, and it changes nothing about output quality.
Stream, and show step progress
2–4 daysNaming the current step converts dead waiting into visible progress. Cheap, and it moves abandonment more than most real speedups.
Set a latency budget with a defined degradation
2 daysDecide in advance what happens at the limit: a partial answer, a cheaper model, or a queued handoff. Do not let the timeout decide.
Often appears alongside: The agent loops, repeating the same step until something kills it · Spend jumped several times over with no corresponding traffic increase
The same question gives different answers each time
Some variation is inherent to sampling. The problem is when variation crosses from wording into substance — different numbers, different decisions, different tool calls — and you cannot tell which kind you have.
Also described as: LLM inconsistent responses · agent not reproducible · different answer same prompt
Root causes — check in this order
1. Variation is never measured
Without running the same input repeatedly, nobody knows whether variance is cosmetic or material.
How to check: Run twenty identical inputs. Diff the outputs. Are the decisions stable even where the wording is not?
2. Free-text output where structure was needed
Prose invites rephrasing. A constrained schema removes the room for substantive drift.
How to check: Any output another system consumes should be structured, not parsed out of prose.
3. Non-deterministic retrieval upstream
Approximate vector search can return different neighbours across calls, so the model sees different context for an identical question.
How to check: Log retrieved document IDs. Are they stable for a repeated query?
4. Silent model version drift
A floating model alias changes underneath you. Behaviour shifts with no deploy on your side.
How to check: Pin explicit model versions and log the exact version served on every call.
Fixes that hold
Constrain outputs to a schema
2–3 daysStructured output eliminates most substantive variance for anything programmatic. Reserve prose for text a human reads.
Score consistency in your evals
2 daysRun each case several times and assert agreement on the decision, not on the wording. Consistency is a measurable property.
Pin model versions explicitly and log them
Half a dayNever depend on a floating alias in production. Upgrade deliberately, behind your eval suite.
Often appears alongside: It works on our test prompts and fails on real users · The answer is wrong because retrieval returned the wrong context
Output is cut off mid-way and nothing reports an error
The model hits its output limit and stops. The response is well-formed enough to look complete, so downstream code accepts it — and a truncated list, a half-written record, or invalid JSON propagates as if it were valid.
Also described as: LLM response truncated · agent output incomplete · max tokens hit silently
Root causes — check in this order
1. The stop reason is never inspected
Every provider reports why generation ended. Code that ignores that field cannot distinguish a finished answer from a severed one.
How to check: Search your codebase for the stop-reason field. If it is never read, you have this bug now.
2. Output limit too low for worst-case responses
Limits get set from typical output length. The long tail exceeds them, and the tail is where the important answers are.
How to check: Chart output token counts. How much traffic sits at exactly the ceiling? That is all truncated.
3. Structured output parsed leniently
Forgiving parsers salvage truncated JSON into a valid-looking object with missing fields, turning a loud failure into a silent one.
How to check: Feed deliberately truncated JSON to your parser. Does it throw, or return something plausible?
Fixes that hold
Treat a length-based stop reason as an error
Half a dayNever return a length-truncated response as success. Retry with a higher limit, or split the task.
Validate structured output strictly
1–2 daysRequired fields must be required. Reject on missing, do not default — a defaulted field is a wrong answer with no trace.
Alert on the rate of length-capped responses
Half a dayA rising rate means your limits no longer fit your traffic. It is a leading indicator, and almost never monitored.
Often appears alongside: The agent calls the right tool with invented arguments · The same question gives different answers each time
Nothing matched that. If your failure is not on this list I would genuinely like to hear about it — it probably belongs here.