AI - LLM Social Intelligence

 



LLM benchmarks still judge models on code and math. 

Useful, but narrow.  

To probe a different axis: social intelligence, the ability to play a multi-agent game under uncertainty, adapt in real time, manage long context, invent strategies, form alliances, manipulate and resist manipulation. 

Werewolf is a natural testbed: language-only, adversarial, procedural, and socially demanding.

It ran a first round-robin with 7 LLMs. Each pair played 10 matches in a controlled setup, then computed an Elo leaderboard. This gives an overall ranking and, above all, a comparable signal of how models behave when persuasion, coordination, and long-horizon planning actually matter.

The design builds on Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction (Google Research; S. Bailis, J. Friedhoff, F. Chen). 

Extend that line by, among others: (i) a fully specified mayor-election and turn-taking protocol; (ii) head-to-head, role-balanced series with per-role Elo; and (iii) social-strategy indicators such as auto-sabotage, Day-1 wolf eliminations, and wolf-side manipulation success, plus per-message vote-swing instrumentation for persuasion analysis.

Agent framing. Models play as agents with tools: they can call a curated set of tools when appropriate to act across phases like real agents, rather than only producing free-form text.

On this initial Elo, GPT-5 leads clearly; GPT-OSS closes the table. 

Started with a simple configuration of the game, already costly in tokens. The goal of this release is to show why this benchmark matters and, with additional API credits, scale up: longer and more complex games, more families of models, and a broader field to answer a simple question: who can beat GPT-5?

If you want to see full games, here are 4 of them, played by 5 different LLMs: github.com/Foaster-ai/Werewolf-bench (may be that you need github account). 

Comments

Popular posts from this blog

IBM goes big to keep quantum computers cold

IBM - avances en mantener adecuadamente fríos computadores cuánticos.

Estructura multinivel de los sistemas de agentes de IA