The paper keeps the LLM fixed, changes one part of the framework at a time, and shows that the framework design alone can change speed by 60× and accuracy by tens of points.
A multi-agent LLM framework is the software layer around the model. It decides who runs next, what each agent remembers, how plans enter the prompt, and who talks to whom. The authors fix the model and the task, then change only one of these design choices. The result: architecture controls latency, recall, accuracy and coordination as much as the model does, and often more.
The paper treats a framework like a database engine. A database with the same data can be fast or slow because of its query planner, storage engine and coordination protocol. In the same way, an agent system with the same LLM can succeed or fail because of its framework [1, §1].
Earlier benchmarks measure the model or a single agent. They keep the framework fixed. Thus, they cannot show which framework choice causes a difference. MAFBench reverses this. It keeps the model fixed and changes the framework [1, §7].
The taxonomy sorts general-purpose frameworks on one axis: which entity holds control-flow authority. Exactly one entity holds it, so the classes do not overlap [1, §2.1]. The running example is a trip plan: Boston to New York, four days, $500, with four roles (Planner, Flight, Hotel, Budget), a web search tool and a calculator.
A fourth group, task-specific frameworks, is outside this axis. Claude Code, Cursor, Cline and OpenHands ship a fixed loop for one task family (software development). OpenClaw ships a fixed personal-assistant loop. The developer can add tools and sub-agents, but cannot change orchestration, communication or storage. Thus, the study does not vary them, except OpenHands as an overhead reference [1, Def. 2.3].
Choice of paradigm fixes some dimensions. For example, in a graph framework the execution graph is also the communication structure. Thus, each experiment varies a dimension only where the paradigm leaves it open: topology in graph frameworks, memory in graph and role frameworks, the planning interface in CrewAI [1, §2.2.4].
MAFBench is an evaluation suite. It wraps different frameworks behind one interface and runs existing benchmarks through them. It does not change the frameworks. It standardizes everything around them [1, §3]:
| Dimension | Benchmark | Metrics | RQ |
|---|---|---|---|
| orchOrchestration overhead | "What is 2+2?" × 50 trials | latency, throughput, output size | RQ5 |
| memMemory architecture | MemoryAgentBench [2] | accuracy, F1, Recall@5 | RQ1 |
| planPlanning interface | GSM8K, CSQA, MATH [5–7] | accuracy, failure rate, runtime | RQ2 |
| specSpecialization | CatDB ML tasks [4] | precision, recall, F1, MAE | RQ3 |
| topoCommunication topology | AgentsNet [3] | success, rounds, tokens, runtime | RQ4 |
To measure pure framework cost, the authors send a trivial query ("What is 2+2?") 50 times through each framework, with gpt-4o-mini. The model work is the same everywhere. Thus, any extra time comes from the framework. Residual is the total latency minus the time inside the LLM call [1, §4.1.1].
Six graph and role frameworks add a thin layer: residual stays under 21 ms and under 2.2% of latency. Agno is the exception, with 365 ms (39.3%). Concordia takes 30.6 s at the median, about 62× the direct call. Its throughput falls to 0.12 requests per second, and its output reaches about 100,000 characters [1, §4.1.1].
Is this cost part of the GABM paradigm? The authors built a minimal GABM skeleton to find out. The skeleton uses the same environment-mediated structure. Its residual is 13.7 ms. Thus, environment mediation costs milliseconds. Concordia's cost comes from its game master, which sends each action through the LLM, plus persistent loops and state serialization. The cost belongs to the implementation, not the paradigm.
Frameworks give agents memory in three ways [1, §4.1.2]:
The test is MemoryAgentBench with four competencies. All runs use gpt-oss-20b on Groq, with identical chunking and ingestion order.
A stronger model does not change the ranking. With Claude-Haiku-4.5, retrieval-only LangGraph gets AR 33.7. This is still below the hybrid design with the weaker model (44.9). Thus, the architecture is the bottleneck for AR. LRU rises from 30.4 to 46.8, so there model and architecture both contribute [1, Table 10].
Runtime follows the same logic. Under accumulation, each query replays the full history, so runtime grows steeply with W on tasks with many queries (AR, TTL). Retrieval does not repeat this work [1, Fig. 4].
Planning has two layers. The planning logic is the reasoning that makes the steps. It lives in the LLM. The planning interface is the framework mechanism. It decides if a plan is requested, what format the plan must have, and where the plan goes in the context [1, §2.2.1]. The experiment keeps the logic fixed (same model makes every plan) and compares three interfaces:
For weaker models, the strict schema is the problem. The model often reasons correctly but breaks the required format, so the framework cannot parse the plan. Qwen-7B violates the schema on 84.7% of GSM8K questions, and its accuracy falls from 13.0% to 3.4%. The free-form interface has 0.0% format failures for every model [1, Tables 11–12].
For frontier models (GPT-5.6-Luna, GPT-5.6-Terra, Claude-Opus-5), format failures are zero. But a plan also does not help them: accuracy stays almost flat. The cost stays. The extra call and the plan text make runs slower:
Many frameworks give an agent a role, like "Data Scientist". The question: does the role label make the model reason like an expert? The test uses five ML datasets from CatDB. GPT-4o-mini writes a data pipeline, and the authors score the executed pipeline, not the code text [1, §3.3, §4.1.4]. Four conditioning strategies:
Role labels change nothing: every profession gives the same score as No Role. Expert procedure gives large gains where the obstacle is method (Yelp: about 42 → 96–100; Volkert: about 65 → 95). The profile tool helps only where the obstacle is knowledge of the data (WiFi: 37.3 → up to 78.6). On EU-IT it hurts (down to 18.7), because it pushes the model toward features it cannot use. A role helps only when it supplies a procedure.
Here, agents can send messages only to their direct neighbors in a graph. The graph is the topology. The test is AgentsNet, with tasks such as Coloring (local conflicts), LeaderElection and Consensus (global agreement). A rewriting engine in MAFBench builds each structure from the same base graphs, so the result does not depend on one vendor framework [1, §3.5, §4.2].
First, see why structure matters. In one round, each agent can tell only its neighbors. The step-through below shows how a fact spreads from one agent. This is an illustration with 16 agents, not paper data.
Now the paper's result. Fully connected ("All") stands in for GABM centralization, where everything goes through one hub.
Local tasks tolerate sparse structure: Coloring succeeds on every graph family. Global tasks do not. For Consensus, only the fully connected structure succeeds at every size, and it converges in 3 rounds for any n. Sparse graphs break into clusters that do not merge in time, even with more rounds and tokens [1, §4.2.1]. In this run, fully connected also used the fewest tokens at n=100 (2.6 M against 5.7–18.3 M), because it finished in 3 rounds. The authors still note that full connectivity has a quadratic message cost in general.
Each principle states where it applies and where it does not [1, §5].
Judge a framework by its execution structure, not by its paradigm. Two GABM implementations differed by three orders of magnitude. Deep orchestration can be correct for multi-round tasks such as social simulation.
Use retrieval-first memory for knowledge tasks (multi-hop QA). Use bounded accumulation for session tasks (tutoring). For fact correction, you must add revise and remove operations yourself.
Do not ask for a plan when the task fits the model's context and the model plans internally. Use a strict format only when a downstream component must parse it.
Give tools when the agent lacks information. Give procedure when it lacks method. The two do not replace each other. Labels can still help routing. Tested with GPT-4o-mini only.
Sparse graphs (scale-free) suit local tasks. Global agreement (leader election, consensus) needs full or near-full connectivity.
Select orchestration depth, memory access mode, planning format and topology first. Then tune prompts or select models.
Future directions in the paper: memory with transactional semantics (versioned facts, write-ahead logs, revise/remove/invalidate), automatic "compilation" of a multi-agent architecture from a task specification, and security benchmarks for prompt-injection spread and memory poisoning [1, §6].
Citation marks such as [1, §4.1.2] point to a section, table or figure of source 1. The topology step-through is an illustration made for this page and is not paper data.