Multi-Agent Orchestration

Agent failure handling gives one worker a recovery plan. Multi-agent orchestration asks a different question: when should the system stop giving one agent every responsibility and split the job into typed nodes, explicit edges, persistent state, and approval points?

Multi-agent orchestration splits a hard workflow into specialized steps with explicit dependencies. This article shows when a graph makes control flow clearer than asking one model call to plan, retrieve, decide, and act alone. A graph becomes safer only when it also limits capabilities, validates merged evidence, and binds approvals to exact side effects.

Imagine a single warehouse worker who has to find your package in a sprawling fulfillment center, check the carrier tracking, verify the refund policy, and write the customer email, all while ten other orders are backing up. Even a strong worker will miss details, mix up labels, and forget which shelf they already checked. Real warehouses solve this with teams: one person picks, another packs, a third updates tracking, and a floor manager makes sure each handoff happens at the right moment.

Modern AI systems face the same bottleneck. A single LLM call can research, analyze, and write, but as tasks get complex it starts to forget details (context window saturation) and blur roles. The fix is multi-agent orchestration: specialized agents for specialized jobs, connected by an explicit workflow graph. Instead of one prompt that tries to do everything, you build a small pipeline where a triage agent classifies the request, an order-lookup agent queries the database, and a response formatter composes the reply.

This article builds on ideas you have already practiced: structured outputs, ReAct loops, human approval gates, memory, and failure recovery. If those feel fuzzy, review the prerequisite lessons first. Here, we take the next step: wiring multiple specialized agents into a graph so they collaborate on a single job without hiding state in a transcript.

AutoGen^[1] and MetaGPT^[2] were early demonstrations that structured roles and explicit coordination improve multi-step execution over ad-hoc multi-agent chat. The AutoGen paper reported gains across coding, math, and question answering, while MetaGPT showed that explicit standard operating procedures produce more coherent artifacts than open-ended chat. Those results moved multi-agent work from a prompt trick to a systems design problem.

The honest case: when multi-agent helps and when it hurts

Before you reach for a graph of agents, take the cost seriously. First-party engineering reports now make the boundary clearer: parallel exploration can pay off, while coupled decisions and unbounded writes make multi-agent systems harder to control.

Cognition, the team behind Devin^[3], published a widely read argument titled "Don't Build Multi-Agents."^[4] Their position: when you split work across parallel subagents, each subagent acts on its own slice of context. Because every action carries implicit decisions, those decisions conflict when the subagents cannot see each other's full traces. The result is a system that looks parallel but produces inconsistent, hard-to-reconcile output. Their recommended default is a single agent with strong context engineering, and where you must split work, share the complete trace rather than isolated messages.

In April 2026, Cognition updated that position with a narrower pattern they said was working in practice: multiple agents may contribute intelligence, but writes stay single-threaded.^[5] That refinement is central to production design. Parallel research, retrieval, and proposal branches do not justify parallel payment, address-change, or label-generation writes.

Anthropic, almost the same week, published "How we built our multi-agent research system"^[6] describing the opposite choice for their Research feature: a lead agent that spawns 3 to 5 subagents, each exploring a different part of the question in its own context window. Their internal research eval showed this setup outperformed a single Opus 4 agent by 90.2%. The catch they state plainly: multi-agent systems use about 15 times more tokens than a chat, so the task value has to justify the spend. They also name where multi-agent is a poor fit: domains where all agents must share the same context or where there are heavy dependencies between agents. They cite coding as exactly that kind of poor fit, because subtasks are tightly coupled and agents struggle to coordinate in real time.

Read together, Cognition's warning and refinement plus Anthropic's research system describe the same trade-off from different task shapes:

The practical rule for 2026: reach for multi-agent only when subtasks are genuinely separable and parallel, and when the task value justifies the token and failure-surface cost. Otherwise a single agent with disciplined context engineering is cheaper and easier to debug. The rest of this article assumes you have cleared that bar and now need to wire the graph correctly.

Here is a small admission check for that decision. Parallel branches win only when work is independent, valuable enough to fund extra execution, and able to merge through a clear contract.

Why linear agent chains break

In the simplest case, an agent pipeline is a straight line: the user asks a question, agent A looks it up, agent B analyzes it, and agent C replies. For a narrow task like "What is the return policy?", a chain works fine. But real customer inquiries are messy. A shopper might ask, "Where is my order, and can I change the delivery address?" That touches inventory, shipping rules, and address validation at the same time. A rigid assembly line forces you to process these in sequence even when they could run in parallel.

The problem with linear chains

A naive pipeline processes steps sequentially, handing output directly from one task to the next. The visual below compares that rigid path with a DAG route that can branch by intent and merge results before the final answer.

Linear chains break the moment you need:

• Conditional routing: A refund request needs the policy agent, not the shipping agent.
• Parallel execution: Checking stock, carrier status, and warehouse location can happen at the same time.
• Cycles with conditions: If the tracking number returns nothing, go back and search by customer email instead.
• Dynamic workflow modification: The agent itself decides what happens next based on what it just learned.

What a DAG gives you

A Directed Acyclic Graph (DAG) models your agent workflow as nodes (agents or functions) and edges (dependencies). Think of it as a flowchart that defines routing rules: "If the Triage Agent says 'shipping question', go to the Shipping Analyst. If it says 'refund request', go to the Policy Agent." A DAG gives you branching and parallelism without back-edges.

Once you add retry loops, review loops, or approval loops, you are no longer dealing with a strict DAG. You're dealing with a state machine or a more general graph. LangGraph supports both. In the DAG route above, the triage node can fan out to specialists and every successful branch still converges on the formatter without cycling.

Key property

The control flow is explicit. Given the same state and routing decisions, you can inspect why execution moved from one node to the next. That's much easier to debug than an open-ended conversation loop where an LLM silently decides the next speaker.

Explicit routing is not authorization. Keep read-only evidence branches separate from any node that can issue credits, change an address, or send an external notification. A write-capable node must still require policy checks, a bound approval when needed, and an idempotency key.

Building a workflow graph in LangGraph

LangGraph is a graph-based framework for multi-agent orchestration. Its core primitives are a state schema, node functions, and edges.^[7]

State: the shared whiteboard

A shared TypedDict (a Python type hint that adds static typing to dictionaries) flows through the graph, serving as the memory for all agents. The code below defines a compact teaching schema for our order-tracking graph. It includes message history, the current routing step, retrieved order data, and an error count. In production, sensitive or large order records should remain in an access-controlled store; graph state should carry a scoped reference plus the verified fields downstream nodes need.

Key insight: The Annotated[list[AnyMessage], add_messages] pattern tells LangGraph to merge updates by appending rather than overwriting. If two parallel branches both append messages, you get both sets in the final list.

Nodes: the workers

Nodes are standard Python functions that transform the graph's state. The functions below implement an order lookup and a shipping analysis step. Each takes the current AgentState, performs external read calls (like a database query or a carrier API call), and returns a dictionary of state updates. LangGraph merges those updates into the global state automatically.

In the snippet, order_db and carrier_api are application adapters. The complete local smoke test after the routing section uses in-memory adapters so you can run the graph without API keys.

Notice that each node returns only the fields it changes. LangGraph patches those into the existing state rather than replacing the whole object. That means the order_data produced by lookup_node is still visible when shipping_node runs, even though shipping_node doesn't return order_data again.

Edges: the routing logic

Conditional edges define the graph's control flow based on the current state. A triage node can write a routing decision into shared state, and a small router function can read that decision and return the next node name. That keeps the classification logic inside a node while keeping the edge function deterministic and easy to inspect.

If you invoke this graph with the query "Where is order 7421?", the trace looks like this:

1. triage_node reads the query, writes "lookup_agent" into next_worker.
2. The router returns "lookup_agent", so LangGraph calls lookup_node.
3. lookup_node fetches order 7421, writes order_data, and sets current_step to "shipping_check".
4. The fixed edge lookup_agent -> shipping_analyst triggers shipping_node.
5. shipping_node queries the carrier, writes shipping_status, stores final_answer, and sets current_step to "format".
6. The fixed edge shipping_analyst -> response_formatter triggers format_node.
7. format_node reads final_answer and appends the final message.
8. The fixed edge response_formatter -> END terminates the run.

Production tip: Keep router functions deterministic. If the routing decision depends on an LLM call, perform that call inside a node (like triage_node), write the result into state, and let the edge function read it. That way you can log the exact routing decision without re-running the LLM during debugging.

Routing also needs a capability boundary. A customer mentioning a refund may route to a policy lookup and review proposal, but it must not cause a payment write merely because a specialist node exists.

Here is the same graph as a complete local smoke test. It uses fake order and tracking data, but the LangGraph state, nodes, conditional edge, fixed edges, and invocation are real:

Four ways to organize a team

Once you can build a single graph, the next question is how to arrange agents when the problem grows. There are four common topologies, each with different trade-offs.

Supervisor agent

One LLM acts as a manager that delegates tasks to specialized workers. That pattern works when the sub-steps aren't known in advance. In our order system, a supervisor might look at the conversation history and decide whether to call the lookup agent, the policy agent, or a human escalation path.

When it helps

Supervisors offer simple, centralized control. When a refund request turns out to need both order lookup and policy explanation, the supervisor can sequence them explicitly rather than hard-coding every combination in the graph edges.

When it hurts

Supervisors become a bottleneck and a single point of failure. Every step costs an extra LLM call, which adds latency and token cost. Supervisors also rely on the LLM outputting structured routing instructions (like "order_lookup"), which can be brittle with smaller models.

Hierarchical teams

For complex enterprise workflows, a single supervisor often gets overwhelmed. The solution is supervisors of supervisors, forming a hierarchical tree. In a large fulfillment operation, a "Floor Manager" agent might manage an "Inbound Team" (a sub-supervisor coordinating receivers and inspectors) and an "Outbound Team" (a sub-supervisor coordinating pickers and packers).

When it helps

Hierarchical structures scale for massive tasks and offer clear separation of concerns. No single manager has to remember every detail of every sub-team.

When it hurts

Deep hierarchies add significant latency. Handoff dilution can occur, where the original customer intent loses detail as it passes through multiple layers of managers. A request that needs only one database lookup might now take three LLM calls just to decide who does the lookup.

Swarm (peer-to-peer handoff)

Agents hand off to each other directly without a long-lived supervisor. OpenAI's experimental Swarm repo^[8] made this pattern easy to study, and the current OpenAI Agents SDK exposes the same idea through first-class handoffs.^[9][10] Handoffs are represented to the model as transfer tools, and input filters can control what history the receiving agent sees.^[10] In the example below, the triage agent doesn't solve the task itself. It selects the specialist that should take over.

If the orchestrator should keep ownership of the conversation and only call specialists for sub-tasks, use specialists as tools instead of handoffs.

When it helps

There's no supervisor bottleneck, agents are loosely coupled, and it's easy to add new specialists.

When it hurts

It's harder to debug because the flow is emergent rather than explicit. Circular handoffs (A -> B -> A) are common and harder to detect than in a DAG.

Map-reduce agents

This pattern uses a parallel fan-out for "embarrassingly parallel" tasks, followed by an aggregation step. It's ideal for tasks like checking a large order from multiple perspectives (stock availability, warehouse location, carrier capacity). In the topology visual above, this is the fan-out shape: one router emits independent analyst tasks, and a reducer merges their findings into one delivery estimate.

To implement a true map-reduce pattern in current LangGraph, the cleanest option is the Send API. It lets one node emit a variable number of parallel tasks, each with its own payload, and then merge the branch outputs through reducers.^[7]

One practical detail: parallel branches should either write disjoint state keys or use reducers for shared keys such as Annotated[list[str], operator.add]. Otherwise the runtime has no safe way to merge their updates.

The reducer is also a trust boundary. It should admit only findings for the current tenant and request, with recorded evidence, before any formatter or decision node consumes them.

Reducers must also reject incompatible scalar writes. Two branches may append independent findings; they should not silently choose different delivery addresses or refund decisions.

Saving progress when agents crash

Checkpointing and persistence

In production, agents crash, APIs time out, and servers restart. You can't rely on in-memory state. LangGraph's InMemorySaver is convenient for local development, but production deployments usually use a durable checkpointer such as Postgres. The current docs use PostgresSaver.from_conn_string(...); one common pattern is wrapping it in a context manager so the runtime can persist each step and resume by thread_id after a crash.^[11]

Production tip: Always use a persistent checkpointer in production. Reusing the same thread_id lets the runtime resume from the latest checkpoint and inspect prior state snapshots during debugging.

One more durability rule matters in production: resumed runs replay from a checkpoint boundary instead of jumping back to the exact Python line where execution paused. That means side effects and non-deterministic work should either live in LangGraph tasks or be made idempotent.^[11]

The storage boundary, not the model, must enforce idempotency. If the execution node is replayed after a timeout, the same key retrieves the first result rather than issuing a second credit.

Getting human approval before critical steps

Autonomous agents are rarely fully trusted in production. You need breakpoints where a human can approve, modify, or reject an agent's plan. A refund over $500, a delivery address change, or a cancellation that affects a bulk order are all cases where the graph should pause and wait. Approval must apply to one exact proposed action, not to the conversation in general.

LangGraph supports this natively through interrupt(), backed by a checkpointer.^[12] Static interrupt_before / interrupt_after breakpoints still exist, but the current docs position them as debugging aids rather than production approval flows. In a production HITL flow, the node emits a structured approval request, the caller sees that payload under __interrupt__, and the graph resumes with Command(resume=...).

The downstream execute node must re-read the authoritative order version immediately before writing, require another review if it changed, and send the same idempotency key to the payment service. One subtle runtime detail matters a lot in production: after resuming, LangGraph restarts the node from the top rather than continuing from the exact line of the interrupt(). Any side effects before the pause need to be idempotent or moved after the interrupt.

How agents share memory

When orchestrating multiple agents, the choice of coordination contract shapes the whole architecture. In practice, three common patterns appear repeatedly: typed shared state, message-oriented coordination, and supervisor-based routing.

In a shared state model (used by LangGraph), a single typed object acts as a whiteboard. Every node reads from and writes to this schema. That makes intermediate variables explicit: if an early node fetches an order record, a node deep in the graph can still access it without repackaging it into a chat message. The downside is that the schema needs discipline. As the graph grows, the state can become bloated, and careless updates can overwrite important fields.

In a message-oriented model (common in AutoGen AgentChat teams), the conversation transcript becomes the main contract between agents. Agents coordinate by reading prior messages and producing new ones, which feels natural for delegation and review loops. The trade-off is weaker typing: the transcript can grow quickly, and downstream agents only know what the conversation history includes.

In a supervisor model, a central router or manager decides which specialist agent should act next and tracks overall progress. Use this when you want explicit control over delegation and completion criteria, but expect a central bottleneck and another policy layer to debug.

Talking to agents outside your system

Shared state and message passing solve coordination inside one runtime. Production systems often need something else: interoperability between independent systems. Two standards matter here, but they solve different problems.

MCP: tools and context access

MCP (Model Context Protocol)^[13], introduced by Anthropic in late 2024, standardizes how an assistant connects to tools, resources, and prompts exposed by an MCP server. It's best thought of as a tool and context protocol, not a full remote-agent delegation protocol.

A2A: remote agent delegation

A2A (Agent-to-Agent Protocol)^[14][15], announced by Google in April 2025 and now developed as a broader protocol community standard, enables direct communication between autonomous agents from different providers. A2A focuses on task delegation, status updates, and result sharing between independent agent systems. This lets one vendor's planner agent delegate work to another vendor's specialist agent without flattening that specialist into a single tool call.

Why the distinction matters

The two protocols solve different layers of the agent interoperability problem. MCP is a local context and capability protocol: it lets one host discover and invoke tools, read resources, and reuse prompts from any compliant server running in the same process or over stdio/HTTP. A2A is a remote delegation protocol: it lets one autonomous agent hand off a complete sub-task to another independent agent (possibly from a different company), poll for progress, and receive structured results or intermediate state.

Without this distinction, teams often end up wrapping every remote capability as an ad-hoc tool call and then wonder why long-running workflows, approvals, and status updates become awkward to implement.

In practice, many production systems use both layers together: MCP to give each agent rich local tools and context, and A2A when one agent's planner needs to delegate work to a separate specialist agent running in another environment or organization. The references at the end of this article link to the current protocol specs and community discussions.

AutoGen's conversation teams

Microsoft's AutoGen^[1] popularized a conversation-based multi-agent pattern where agents communicate through messages. That design space overlaps with work like ChatDev^[16], where agents role-play as software developers in a collaborative environment. In current AgentChat releases, chat-style teams such as SelectorGroupChat, RoundRobinGroupChat, and Swarm coexist with directed workflows through GraphFlow.^[17]

For the team abstractions below, the main contract is still the shared conversation history plus team policies such as speaker selection and termination conditions. The implementation below focuses on SelectorGroupChat, which is the clearest example of message-oriented coordination.

This snippet requires autogen-agentchat, autogen-ext[openai], and OpenAI credentials. It constructs a team; running it calls the configured model.

If you need explicit graph routing inside AutoGen instead of model-selected turn-taking, use GraphFlow rather than a selector-based team.

Picking the right framework

Choosing the right orchestration framework depends heavily on the use case. While all three frameworks support multi-agent patterns, they prioritize different approaches: LangGraph enforces strict graph-based execution, AutoGen spans conversation-centric teams and GraphFlow workflows, and MetaGPT structures agents around rigid software engineering roles.

Recommendation

Use LangGraph when you need strict control over routing, retries, checkpoints, and approvals. Use AutoGen when a conversation-centric team abstraction is the most natural fit or when GraphFlow gives you enough structure without dropping to a lower-level state machine. Use MetaGPT when you want rigid software roles and SOP-driven artifact generation.

Where these systems show up in production

Multi-agent orchestration is useful when it separates independent reads and keeps external writes behind one controlled execution boundary. Here are three plausible production designs, not claims about deployed performance.

Order fulfillment pipeline

A coordinator routes independent inventory and carrier lookups in parallel, then merges their source-backed findings. A permitted fulfillment service, not a free-form specialist, generates a label only after inventory and routing checks satisfy policy. The graph can draft a shipment confirmation or send an exception to review without giving every worker write authority.

Returns processing

When a customer requests a return, independent branches can retrieve purchase eligibility, the applicable return policy, and approved risk signals scoped to the customer and tenant. A reducer validates sources and produces a proposed action. If policy requires approval, the reviewer sees the exact amount, evidence references, and action digest; a payment service issues credit only after revalidation with an idempotency key.

Delivery exception handling

When a carrier reports a failed delivery, a triage node classifies the exception. Address validation can produce a correction proposal, while a notification branch drafts a message. Changing an address or scheduling a new attempt remains an authorized action bound to verified order state. After a bounded number of failed proposals, the graph escalates with evidence instead of continuing indefinitely.

What you should be able to defend

After this article, you should be able to defend these design choices in a system review or interview:

Follow-up questions

How do you handle infinite loops in a cyclic agent graph? Once you allow retries or review loops, you are no longer dealing with a strict DAG. LangGraph can model cyclic graphs and state machines, but production systems need hard termination through recursion_limit, step_count, max_turns, max retries, deadlines, or a forced transition to failure handling or human review.

When would you choose a supervisor pattern over a flat swarm pattern? Choose a supervisor when you need centralized control, strict process adherence, policy insertion, or arbitration between workers with overlapping capabilities. A swarm or peer-to-peer handoff pattern fits loosely coupled specialists, but it is harder to debug because routing decisions are decentralized.

Why is typed shared state often easier than message passing for structured workflows? It is not universally better, but it gives downstream nodes reliable fields such as order_data, approval_status, policy_result, and error_count without asking them to infer state from a growing transcript. Message-oriented teams can work well, but they need stronger serialization, summaries, and termination policies as history grows.

How does human-in-the-loop approval fit into graph architecture? Model it as an explicit approval node. In LangGraph, that node can call interrupt() to emit a structured review payload and pause execution through a checkpointer. The payload should contain the proposed action digest, evidence, source version, and idempotency key. After a human decision resumes the workflow with Command(resume=...), the execution node still revalidates authoritative state before writing.

Common pitfalls

Infinite loops and how to stop them

If you allow cyclic graphs (for example, "if the tracking lookup fails, try again with a different query"), you must enforce termination. Common guards include:

• A recursion_limit in the graph invoke or stream config.
• A step_count or error_count field in state that increments each retry.
• A max_turns guard in conversation loops that forces a transition to failure handling or human review.

Without one of these, two agents can pass a task back and forth forever.

Context bloat and how to avoid it

A frequent failure mode is passing the entire message history to every agent. If the conversation has twenty turns, every node sees all twenty even when it only needs the latest order ID. Fixes include:

• Using a summary node that compresses long histories into a short context string before routing.
• Storing large objects (order records, carrier responses) by reference ID rather than embedding the full JSON in state.
• Keeping the messages list for the transcript but adding dedicated state keys (like order_data) for data that downstream nodes need directly.

The state can expose a minimal verified projection while keeping sensitive record content behind the application authorization boundary:

Non-deterministic routing

Using an LLM to decide the next step when a simple if/else or regex would suffice increases cost and reduces reliability. A good rule: use deterministic routing for rules you can write down, and use LLM-based routing only when the classification genuinely requires language understanding.

Try it yourself

To check your understanding, try this small design exercise:

Scenario: A customer messages your store saying, "I ordered a blue shirt last week but received a red one instead. I also want to know if I can exchange it for a different size."

Task: Sketch a LangGraph StateGraph with at least three nodes and one conditional edge. Define the TypedDict state, name the nodes, and write the routing logic. Then answer these questions:

1. Which parts of this request can run in parallel, and which must run in sequence?
2. Where would you place an interrupt() if the refund amount exceeds $100?
3. What state keys would you need so the response formatter can compose a complete reply without copying the whole order record into graph state?
4. What must the payment boundary verify before it executes an approved refund?

Solution sketch

• Parallel: Order verification and return-policy lookup can run at the same time.
• Sequence: The response formatter must run after both parallel branches finish, because it needs the order status and the policy explanation.
• Interrupt: Place it after an action proposal contains the refund amount and evidence, but before any payment write. Bind approval to an action digest and source version.
• State keys: order_ref, minimal verified order fields, policy_result, customer_message, proposed_action, approval_status, and final_answer.
• Execution check: Re-read authoritative order state, reject stale or changed proposals, and use an idempotency key for the credit request.

Key takeaways

1. Multi-agent is a cost trade, not a default: Use it only when subtasks are separable and parallel and the task value justifies roughly 10 to 15x the token cost. For tightly coupled work, a single agent with strong context engineering wins.
2. Explicit graphs are the primitive: DAGs handle one-way workflows, while state machines add controlled cycles for retries and approvals.
3. State is the API: In LangGraph, the State schema defines the contract between agents. Keep it strict, typed, tenant-scoped, and limited to necessary evidence or references.
4. Patterns: Know when to use Supervisor (centralized), Swarm (peer-to-peer), or Map-Reduce (parallel).
5. Persistence: Use durable checkpointers for failures and approvals, and make replayed external writes idempotent.
6. Cost: Prefer deterministic structural nodes where rules are explicit; reserve model calls for branches that need language understanding.
7. Protocols: MCP exposes tools and context into a runtime; A2A connects independent agent systems across vendor boundaries.

This article closes the Advanced Agents and Retrieval section. You started with single agents and orchestration basics; you now know how to wire specialized agents into explicit graphs, choose a topology from the workflow shape, share typed state, persist progress, gate exact risky actions with bound approval, and decide when not to build a multi-agent system at all.

Next: Continue to Inference: TTFT, TPS & KV Cache, the start of the Inference and Production Scale section. Multi-agent paths can multiply model calls per user interaction, and the Anthropic research system above reported roughly 15 times the tokens of a single chat. The next article explains the latency (TTFT), throughput (TPS), and memory (KV cache) realities that decide whether your agent graphs can run fast and affordably at production scale.