Criteria to consider (from proposal): Reproducibility, Objective Evaluation, Context Variability, Architectural Configurability, Extensibility and Implementation Effort
a) Multi-problem testbeds
Several existing benchmarks are wrapped into one testbed, and each architecture is run on all of them.
| Benchmark | Notes | Judgement |
|---|---|---|
| MAFBench | 🟢Good: - Exactly what we want to vary: the arch dimensions themselves (Orchestration, Memory, Planning, Specialization, Topology), one at a time with the LLM fixed - They already did the hard part of picking a suitable benchmark per dimension (AgentsNet for topology, MemoryAgentBench for memory, GSM8K/CSQA/MATH for planning, CatDB for specialization) - Lots of metrics to prioritize from (accuracy/F1/recall, latency, rounds, tokens, runtime) -> real trade-offs, not one option winning everything - One pipeline for all frameworks (same prompts, same stop criteria, same config) + LLM-judge scoring -> reproducible and comparable - Their 6 principles could be used as a baseline/ground truth for what the selector should pick 🔴BAD - Not one problem, it’s a mix of 5 benchmarks and not full factorial -> no single task where all dims vary together, so a selector would pick per dimension not a full arch - Tasks are generic (QA, math, ML tasks, graph coordination), far from the factory setting (machines + tasks) | |
| MASBench | 🟢Good: - simple, mostly implemented and the repo is clean with clear extension points - 3 problems, with 3 progressive capabilities (Protocol -> Protocol+Memory-> Protocol+Memory+Routing) - Results metrics (Score/tokens/eff) are clear - Arch config given with the option to extend them 🔴BAD - few metrics to prioritize from (score, tokens) and usually the choice with most capabilities wins success score - might be too easy for the selector | |
| Scaling Agent Systems | 🟢Good: - Properly made, seems robust with good results and variations - Arch/Task type variations 🔴BAD - Benchmarks can be a bit complicated to run, especially not sure about theTerminal-bench and SWE one |
b) Single-problem testbeds
A classic problem (e.g. job shop scheduling) is adapted to a multi-agent setting.
| Benchmark | Notes | Judgement |
|---|---|---|
| REALM-Bench | only JSSP considered | 🟢Good: - A bit similar to the real environment we are trying to do it for (machines and tasks) - Simple, no tools or environment setup needed, easy to reproduce. 🔴BAD: - Not sure they will produce enough variability for a selector to choose based on them. (only variables here are if the problem is static vs dynamic and If It’s big or small problem) -> would that really make one MAS better than another??? We are talking about 3 or 4 options that each should be good for some variant of the problem - Not much extensibility, We can generate more problems with or without disruption like the one mentioned in DynaSchedBench or take more problems from other collections like ScheduleOpt |
| BoxLift | 🟢Good: - similar to the factory settings - pre-given archs configs - Game like tasks are pretty simple and generated with code 🔴BAD - Repo is a bit old but It’s manageable - only 2 metrics (Success factir vs efficuency). Selector has few stuff to consider from | |
| DESBench | 🟢Good: - Similiar to the factory setting, machine+tasks+distruptions - Archs, metrics, env are all given and u can rerun the tests directly. - 4 Main metrics to priotrize from and even mare sub-metrics - JSSP underneath anyway 🔴BAD - not trully MAS oriented, It’s more about factory coordinations and orgnization. appartly these selected archs are common there like Holonic or Heterachical |
Conclusion? Ideas?
- It would be better to use a mix of benchmarks, as by nature, each problem and objective will favor one kind or two of agentic architecture. making it quite harder to test a proper range of variability and decreasing the decision accuracy (A selector could end up with a tie or very close uncertain decision)
- Even if the selected benchmarks are different in domain/focus, It shouldn’t matter as LLMs should be good enough to generalize to the different problems and using only one or few ones for all tasks can remove the LLM-uncertainty and focus more on the structure or archticture.