AI - LLM Social Intelligence
LLM benchmarks still judge models on code and math.
Useful, but narrow.
To probe a different axis: social intelligence, the ability to play a multi-agent game under uncertainty, adapt in real time, manage long context, invent strategies, form alliances, manipulate and resist manipulation.
Werewolf is a natural testbed: language-only, adversarial, procedural, and socially demanding.
It ran a first round-robin with 7 LLMs. Each pair played 10 matches in a controlled setup, then computed an Elo leaderboard. This gives an overall ranking and, above all, a comparable signal of how models behave when persuasion, coordination, and long-horizon planning actually matter.
The design builds on Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction (Google Research; S. Bailis, J. Friedhoff, F. Chen).
Extend that line by, among others: (i) a fully specified mayor-election and turn-taking protocol; (ii) head-to-head, role-balanced series with per-role Elo; and (iii) social-strategy indicators such as auto-sabotage, Day-1 wolf eliminations, and wolf-side manipulation success, plus per-message vote-swing instrumentation for persuasion analysis.
Agent framing. Models play as agents with tools: they can call a curated set of tools when appropriate to act across phases like real agents, rather than only producing free-form text.
On this initial Elo, GPT-5 leads clearly; GPT-OSS closes the table.
Started with a simple configuration of the game, already costly in tokens. The goal of this release is to show why this benchmark matters and, with additional API credits, scale up: longer and more complex games, more families of models, and a broader field to answer a simple question: who can beat GPT-5?
If you want to see full games, here are 4 of them, played by 5 different LLMs: github.com/Foaster-ai/Werewolf-bench (may be that you need github account).
.jpeg)
Comments
Post a Comment