Case study · Reliability & Cost & latency
Multi-agent systems without the spiral
Two production agent systems on MCP and CrewAI — a project-management assistant and an autonomous lead-generation pipeline. What multi-agent buys you, what it costs, and the specific cases where a single agent with good tools wins.
- In production, not a demo
- MCP + CrewAI
- My role
- Technical Lead — architecture, agent design, delivery
- Stack
- PythonFastAPIModel Context ProtocolCrewAILangChainAzure OpenAIPostgreSQLDocker
What broke
A researcher agent and a qualifier agent disagreed about whether a company was worth pursuing, so the orchestrator asked both again. And again. Nineteen billed model calls to decide one lead was a bad fit.
I have built two multi-agent systems that ran in production rather than in a demo: a project-management assistant that let people manage tasks and boards conversationally, and an autonomous lead-generation pipeline that researched prospects, enriched company data, qualified leads, and drafted personalised outreach.
Both worked. Both taught me that the interesting engineering in multi-agent systems is almost entirely about bounding them.
The honest framing first
Multi-agent architectures are oversold. The genuine wins are narrow:
- Separation of context. A researcher agent that only sees research context produces better output than one prompt trying to hold research, qualification and writing at once.
- Different tools per role, so no single agent gets a bloated tool list billed on every call.
- Independent, parallelisable work — enriching forty companies concurrently.
Everything else people claim for multi-agent — “emergent collaboration”, “self-correcting teams” — I have not seen hold up under cost scrutiny. Two agents debating is two agents billing. If a single agent with a well-designed tool set can do the job, it will be cheaper, faster, and far easier to debug.
Architecture
Lead generation pipeline, with the bounds made explicit
The interesting parts are the step ceiling, the structured handoffs, and the fact that agents cannot call each other directly.
Select a stage to see the decision made there.
01 Trigger & budgetFastAPI
Every run starts with an explicit budget: a maximum step count and a maximum token spend. The run is a bounded resource from the first line, not something that gets bounded later when the bill arrives.
Tradeoff: Occasionally a genuinely complex prospect hits the ceiling and returns partial results. Partial results with a clear reason beat unbounded spend.
02 OrchestratorCrewAI, role-based
A single component owns sequencing. Agents never invoke each other — they return structured output to the orchestrator, which decides what runs next. That keeps the control flow in ordinary code where it can be read and tested.
Tradeoff: Less "autonomous" than agents delegating freely. It is also the difference between a system you can reason about and one you can only observe.
03 Research agentNarrow tool set
Gathers public company information. Sees only the tools it needs — search and fetch — so its per-call tool overhead stays small and its failure surface stays narrow.
Tradeoff: Duplicated some capability across agents rather than sharing one fat tool registry. Cheaper per call and much easier to attribute failures.
04 EnrichmentDeterministic code
Firmographic lookups, domain resolution, and dedupe are plain code against APIs. No model involved, because none is needed.
Tradeoff: Unglamorous. Also the most reliable and cheapest stage in the pipeline, and a reminder that the best agent design decision is often to not use an agent.
05 Qualifier agentStructured output only
Scores fit against defined criteria and returns a strict schema — score, reasons, confidence. Never prose. Downstream code branches on the schema rather than parsing sentences.
What went wrong: Constrained output loses nuance a human reader might value. It also eliminated an entire class of inconsistency, because there is no room to rephrase a decision.
06 Outreach draftingHuman in the loop
Drafts personalised outreach. Nothing sends automatically — a person reviews and approves. That was a product decision as much as an engineering one.
Tradeoff: Caps throughput at human review capacity. Given that the failure mode is emailing a real prospect something wrong, that cap is a feature.
07 Run tracePer-run, step level
Every step, prompt, tool call and token count is recorded against one run ID, so any decision can be reconstructed after the fact.
Tradeoff: Storage and some latency per step. Without it, diagnosing a bad qualification is guesswork — and bad qualifications are the thing you most need to explain to a sales team.
What broke
The qualifier agent originally returned prose with a recommendation embedded in it. The orchestrator’s logic checked whether the recommendation was confident enough to proceed — and when it could not tell, it re-ran the research and qualification steps to gather more information.
On an ambiguous prospect, that produced a loop. The researcher found slightly different information, the qualifier expressed slightly different hedged confidence, and the orchestrator asked again. Nineteen billed model calls to conclude that one company was a bad fit — a conclusion a human reaches in about four seconds.
Three things fixed it, and all three are now defaults for me:
- Structured output with an explicit confidence field. The orchestrator branches on a number, not on its reading of a sentence. Ambiguity became a value the code could handle rather than a condition it could not detect.
- A step ceiling per run. Bounded before it is needed, not after.
- Repetition detection. Hash each tool call with its arguments; if the same hash recurs within a run, stop and escalate rather than hoping the next attempt differs.
I have since written this failure up as a named category — the runaway loop — because I have now seen it in several other codebases with the same root cause.
What MCP actually changed
For the project-management assistant, I used Model Context Protocol to expose tools — task manager, board manager, status handler — as modular servers rather than functions wired into the agent.
The real benefit was not the protocol itself. It was that the tool boundary became a place where validation, authorisation and logging naturally lived. When tools are ordinary in-process functions, it is tempting to pass model output straight through. When they are a separate server with a schema, validating input is the obvious thing to do.
The cost: another deployment surface, and the tool schemas count against input tokens on every call. For a system with many tools, trimming those schemas was a measurable cost win.
What I would tell you before you build one
Start with a single agent and good tools. Add a second agent only when you can name the specific context separation or parallelism it buys you. When you do:
- Agents return structured data to an orchestrator; they do not call each other.
- Every run has a step ceiling and a token budget before it is ever launched.
- Every agent gets only the tools its role needs.
- One trace ID spans the whole run, or you will not be able to explain anything.
Next case study
Throughput discipline at telecom scale
A configuration-driven telecom data platform processing 200M configuration and 320M performance parameters per 15-minute cycle on Golang, Kafka, Kubernetes and PostgreSQL. The project that shaped how I think about bounded resources — and why I trust it more than any AI credential I have.
Read itHave a system with a similar shape?
Tell me what it does and what worries you. I will tell you what I would look at first — that part is free.