Decisions you own, not code you write

Short answer

A useful audit produces four artefacts you keep: a failure taxonomy specific to your agent, a cost and latency profile, an eval suite wired to your real traffic, and a prioritised fix list with effort estimates. If a proposal does not name deliverables you own afterwards, you are buying a slide deck.

Short answer

Ask about four things, in this order: can you prove it still works after a change, can you explain a failure after the fact, is spend bounded, and does someone own it. The single most useful question is "if you changed a prompt today, what would tell you something broke?" — if the answer is "a customer would tell us", you are not ready to launch, regardless of how well the demo goes..

Short answer

Buy, unless you have a hard data-residency requirement or an existing observability platform with a team who owns it — in which case extend that. Building from scratch is almost never right: the initial version takes days, and the maintenance you did not budget for is what actually costs you.

Short answer

Five: cost per conversation, eval pass rate, failure rate by named category, p95 latency per operation, and containment or escalation rate. Those five answer every question an executive asks — is it working, is it getting better, what does it cost, and is it getting worse — and each one is a leading indicator rather than a post-mortem.

Short answer

Handle it internally when the gaps are clear and someone has the time — most reliability work is a few days per gap and needs no specialist. Bring in outside help when the gaps are clear but capacity is not, when you need it done by someone who has done it before, or when you need independent evidence for a customer or regulator.

Short answer

It can be, and the controls are mostly ones you already know. Three things are genuinely different.

Short answer

One difference matters and the rest is marketing: an agent takes actions with side effects in your systems, a chatbot returns text to a human. When output is text, a wrong answer costs you a confused user and a support ticket.

Short answer

That you can reconstruct any individual decision, and that a human is accountable for the ones that matter. No control framework has an AI section — SOC 2 asks about change management, access control and monitoring exactly as it does for any system.

Short answer

Default to an API. Self-hosting replaces a variable cost you can turn off with a fixed cost you cannot, and it moves model operations onto your team — quantisation, batching, KV-cache management, capacity planning, upgrades.

Short answer

Classify by two properties of the action, not by how sure the model claims to be: can it be undone, and how far does it reach. Anything irreversible and wide gets a human gate.

Short answer

The comparison usually presented is monthly token cost against monthly salary, and agents win it enormously — which is why it is the wrong comparison. Three lines are missing.

Short answer

Connecting is straightforward — if a system has an API, an agent can call it, and if it does not, the same wrapper you would write for any other client works here too. The difficulty is elsewhere: your internal APIs were almost certainly designed on the assumption that the caller knows what it is doing.