Paper: MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability
Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability - BUPT-GAMMA/MASBench
Key Idea
each agent sees only a local part of the environment. Thus, the agents must exchange information to reach the goal. The paper uses 3 different tasks (Reasoning, scheduling, game) and adds three mechanisms one at a time: a message Protocol, a persistent Memory, and Routing (who gets a message). On all tasks, the mechanisms often raise the score. Also, a good protocol cuts the message tokens, so efficiency often rises 2× to 4×.
The paper focuses on the partially-observable envs and how agents can coordinate to solve the task in question.
3 tasks included 
Reasoning:
Two agents, Alice and Bob, receive different evidence for the same question. They exchange clues and produce a final answer. This task compares free-form dialogue with a structured communication protocol.
Data adapted from HotpotQA Homepage
Scheduling:
Five agents negotiate a continuous 20-minute meeting while each sees only its own calendar. Calendars span five days at 10-minute granularity. The task compares memory and protocol settings.
Data is generated with their scripts.
Game:
Agents explore a grid world, collect resources, and deliver them to warehouses with resource-specific quotas. Each agent observes a local window and can carry only one resource type at a time, subject to capacity limits. Messages help agents coordinate across their separate views.
Data is generated with their scripts.
Each task has gives LLMs different capabilities and they compare between all of them as of how they affect the results; from their github README:
| Task | Collaboration challenge | Mechanisms | Included data |
|---|---|---|---|
| Reasoning | Combine evidence distributed across two agents | Protocol | 300 questions: 200 simple + 100 complex |
| Scheduling | Find a shared 20-minute meeting using five private calendars | Memory + Protocol | 200 scenarios: 100 simple + 100 complex |
| Game | Explore, collect, and deliver resources with limited local visibility | Memory + Routing + Protocol | 100 maps: 50 at 15×15 + 50 at 18×18 |
The Axis of variations:
Protocol; How messages look
- Reasoning: typed messages for an information request, shared evidence, and a final answer.
- Scheduling: a fixed sequence of phases: coarse search, then candidate proposal, then a vote.
- Game: structured fields for agent state, observations and plans.
Memory; What an agent keeps between rounds
- Scheduling: structured availability and negotiation state, with candidate slots and rejected slots. An ablation (cross-round memory) keeps only the state between rounds, with no availability preprocessing [1, App. B.5].
- Game: the world state that the agent discovered from its own observations and from received messages.
Routing; Who receives a message (Game only)
- RANDOM: the agent sends to a random teammate.
- GATE: the agent first decides if a message is necessary. Then it selects the recipient.
The Base configuration uses free-form dialogue in Reasoning and Scheduling. In Game, Base has no communication.
Their Code looks solid btw, and easy to use.