Github: https://github.com/CoDS-GCS/MAFBench
Paper: https://arxiv.org/abs/2602.03128
Published: Sep 2026!!
TL;DR
It is a benchmark for arch choices in LLM agents frameworks, they cover different axis of differentiation and shows good variance between them
Studies the effect of various Arch design choices on the final quality and discusses how different frameworks implement it and using these impls (Orchestration, Memory, Topology, Specialization, Planning interface)
They did the hard part of selecting benchmarks suitable to each dimension
- for example, GSM8K text-math benchmark is suitable for the planning dimension (planning vs no-planning) and AgentsNet for topology
Figure from Orogat, Rostam & Mansour 2026, arXiv:2602.03128. Open full size
On each dimension, They try to test the various options and their effect on the final quality/metric
Topology as an example: It affects communication to a huge degree, each agent can only tell its direct neighbors. fact spreading or global error handling will be a direct parallel
Figure from Orogat, Rostam & Mansour 2026, arXiv:2602.03128. Open full size
Testing on AgentsNet ; It’s a set of problems designed specifically to benchmark coordination
Gives us results like, showing that different topologies excell on different tasks
Their experiments came up with 6 principles, like what limits scalability, how to choose a memory technique and etc..
Each principle states where it applies and where it does not.
-
Measure orchestration overhead. It is the main scalability limit.
Judge a framework by its execution structure, not by its paradigm. Two GABM implementations differed by three orders of magnitude. Deep orchestration can be correct for multi-round tasks such as social simulation.
-
Select memory from task needs, not from context size.
Use retrieval-first memory for knowledge tasks (multi-hop QA). Use bounded accumulation for session tasks (tutoring). For fact correction, you must add revise and remove operations yourself.
-
Keep planning interfaces permissive, and planning optional.
Do not ask for a plan when the task fits the model’s context and the model plans internally. Use a strict format only when a downstream component must parse it.
-
Make specialization procedural. Role labels are secondary.
Give tools when the agent lacks information. Give procedure when it lacks method. The two do not replace each other. Labels can still help routing. Tested with GPT-4o-mini only.
-
Design the communication topology. More interaction cannot fix a bad one.
Sparse graphs (scale-free) suit local tasks. Global agreement (leader election, consensus) needs full or near-full connectivity.
-
Engineer system interfaces before prompts.
Select orchestration depth, memory access mode, planning format and topology first. Then tune prompts or select models.