DESBench runs four coordination protocols for LLM agents on the same simulated factory, and shows that each protocol fails in a different way.
DESBench keeps the factory simulator, the jobs, the breakdowns and the budgets the same. Only the coordination protocol Γ changes: who decides, who may refuse work, and where a failed assignment goes.
The protocol moves the results more than the LLM or the agent framework does. No protocol wins on all four dimensions: centralized is robust but slow, hierarchical is fast but violates budgets, heterarchical is flexible but sends the most messages, and holonic keeps budgets but leaves work unfinished.
To copy text into Obsidian, select it and press Ctrl+C (⌘C on Mac). The page copies it as Obsidian Markdown with callouts, tags, highlights, tables and math. Each figure also has two buttons. Copy DataviewJS copies an interactive block for Obsidian (Dataview plugin, JavaScript queries on). Copy HTML copies a static snapshot of the figure as it looks now.
The factory is a tree of agents. One Plant agent sits at the top. Five Area agents sit under it. Twelve Cell agents execute the work. A Cell is a group of machines. This figure follows one work item through each protocol. Select "Commitment fails" to see what happens when the chosen Cell cannot take the work.
DESBench (Distributed Event-driven Scheduling Benchmark) is a test bed for coordination in a multi-agent system (MAS). A MAS is a team of agents that share one environment and make decisions that affect each other. Here, the agents are LLM controllers that schedule jobs in a simulated factory.
Most agent benchmarks use weakly coupled tasks. An action has a short, local effect. In a factory, an action changes queues, machine setups, transport load and energy use. The effect can show up much later, at a different agent. The authors argue that this coupling is where coordination design starts to matter. Classic scheduling benchmarks assume one central solver, so they cannot compare coordination structures.
The paper asks: when do different coordination paradigms succeed or fail? A paradigm is a pattern for three things: how information flows, who has decision authority, and how conflicts get resolved. The paper tests four paradigms:
The world is a discrete-event simulation (DES). A DES changes state only when an event occurs, for example a job release, an operation completion or a machine breakdown. There is no fixed time step. Three runtime parts control the process.
1. Event engine G. It takes the state after the last decisions, x⁺, and runs the simulator until the next point where a decision is necessary:
Here, τk is decision epoch k, and Ek+1 is the set of world events that occurred in between.
2. Event interpreter I. It reads the new events and the new protocol objects Mk+1 (messages, commitments, settlements). Then it selects the agents U that must act and gives each one a protocol context κi:
3. Legal action mask. Each active agent sees only a local observation ωi, never the full state. The protocol Γ sets a mask mΓi that defines which actions are legal:
This mask is how one physical state gives different choices under different protocols. For example, "reject" is a legal Cell action under heterarchical and holonic coordination, but not under centralized or hierarchical coordination.
This section describes how a test instance is built, how one episode runs, what the LLM sees, and what the benchmark measures. The details come from Section 3, Section 4 and Appendices A to F of the paper.
An instance fixes the topology, the route grammar (which stages a job must pass), the arrival process, the failure model, the budgets and the seeds. Only the protocol changes inside one instance. The paper reports four instances. The event trajectories come from real operational datasets that earlier manufacturing benchmarks published (references 26 to 28 of the paper).
| Instance | Topology | Coordination stress | Failure model |
|---|---|---|---|
| A3C9-1 | 3 Areas, 9 Cells, 3-stage routes | Cross-area trade-off; transport ×1.15 | Weibull aging |
| A5C12-1 | 5 Areas, 12 Cells (4–3–2–2–1), 4-stage routes | Branch pressure: many routing choices; Cells 3, 6, 8, 10, 11 slowed; transport ×1.15 | Exponential |
| A5C12-2 | Same as A5C12-1 | Cluster pull: Cells 0, 1, 4 fast and 7, 9, 11 slow, which tempts greedy routing; transport ×1.20 | Weibull aging |
| A5C12-3 | Same as A5C12-1 | Late commitment: late-stage Cells 2, 6, 8, 10, 11 slowed, so early choices cost more later; transport ×1.25 | Load-dependent |
An episode is one full simulation run with one seed. The loop below repeats until all jobs finish or the run stops early.
All frameworks use the same two-message prompt. The system instruction tells the model to return exactly one legal action index as JSON, with no reasoning. Decoding is deterministic, with a short output budget. Below is the paper's example payload for a Plant decision in A5C12 (Appendix B.1):
{ "agent": "plant", "role": "plant", "kind": "area_selection",
"legal_actions": [0, 1, 2],
"decision_factors": [
"Use current context, visible feedback, and working_memory only.",
"Prefer feasible actions that reduce congestion and future reroute pressure." ],
"options": [
{"action": 0, "area_id": 0, "summary": {"active_jobs": 0, "backlog_jobs": 10, "energy_kwh": 0.0}},
{"action": 1, "area_id": 1, "summary": {"active_jobs": 0, "backlog_jobs": 10, "energy_kwh": 0.0}},
{"action": 2, "area_id": 2, "summary": {"active_jobs": 0, "backlog_jobs": 10, "energy_kwh": 0.0}} ],
"context": { "feedback": { "progress": {"released_jobs_total": 0, "completed_jobs_total": 0, "backlog_jobs_total": 10},
"capacity": {"admissible_for_new_work": true, "busy_machines_total": 0, "down_machines_total": 0} },
"recent_event_types": ["release_area_selection_window"],
"selected_job_id": 0, "selected_stage_id": 0 } }
Reply: {"action": 1}
The paper does not inject faults by hand during a run. Disruptions come from the instance's failure model: exponential failures, Weibull aging (the failure rate rises with machine age) or load-dependent failures. Each instance also sets WIP, energy and carbon budgets. A budget violation can appear long after the decision that caused it, because queues and setups carry the effect forward.
The main experiment uses topology A5C12 and its three scenarios. Each of the 4 protocols runs in 2 frameworks (LangGraph and AgentScope) with 3 LLMs (GPT-5.4, Gemini-3-Flash, Qwen-3.5). Each protocol–framework–LLM combination gets 10 seeds per scenario, so 30 episodes. Table 3 of the paper reports the mean over the three scenarios. A3C9-1 is reported separately in Appendix E. All runs used one Windows workstation (Intel i7-12700K, 32 GB RAM).
The paper reports 15 primary metrics in four dimensions. The appendix lists many more for diagnosis.
| Dimension | Metric | Meaning |
|---|---|---|
| Effectiveness | Gmk ↓ | Makespan: final simulation time at the end of the episode |
| Gtd ↓ | Tardiness: total lateness of completed jobs with due dates | |
| Gcj ↑ | Completed jobs per episode | |
| Constraint alignment | Gvc ↓ | Violation count: recorded budget overshoots |
| Geo ↓ | Energy overshoot magnitude above the budget, summed | |
| Ged ↓ | Energy overshoot duration | |
| Gen ↓ | Total energy use per episode | |
| Coordination efficiency | Gds ↓ | Decision steps per episode |
| Gcm ↓ | Protocol-level coordination messages | |
| Gfb ↓ | Decisions from the fallback controller | |
| Gllm ↓ | Decisions from the LLM | |
| Robustness | Gtr ↓ | Truncation rate: share of episodes stopped by deadlock or no progress |
| Gdj ↓ | Completion debt: unfinished jobs at the end | |
| Gut ↓ | Overdue tardiness of the unfinished jobs | |
| Gdo ↓ | Completion debt: unprocessed operations at the end |
Select a metric. Each row is one protocol. Each dot is one framework–LLM pair, and the vertical bar is the mean of the six dots. When the dots of one row stay close together and the rows stay apart, the protocol explains more of the result than the model does.
The paper reads the A5C12 results as follows:
Makespan is the time when the episode ends. A run that stops early, with jobs unfinished, also ends early. Thus, a low makespan can mean "fast" or "gave up". This plot puts makespan against completed jobs, so that you can see the difference.
In A5C12, four holonic rows sit in the lower-left corner: short episodes with only 3 to 4 of 10 jobs done. The paper names this effect itself and concludes that single-metric comparisons are misleading. In A3C9-1 the picture flips: holonic rows have the highest makespan (79 on average, against 42 for hierarchical). So the holonic result depends on the instance, not only on the protocol.