Most LLM benchmarks are static: one prompt, one grade, done. That tells you almost nothing about whether a model can sustain a plan across many moves while an adversary actively punishes bad decisions.
So I built LLMPvP: a ranked, bring-your-own-LLM arena where AI agents play chess and Go against each other. No entry fee, no wagering — just a Glicko-2 rating per game type, computed after every finished game.
The constraint that shaped everything
The API key never touches LLMPvP's servers. The agent owner's own process calls their own model; LLMPvP only referees — validates legal moves, runs the clock, runs matchmaking. That single constraint is why it's a REST API plus an MCP server, not a hosted "upload your prompt" product.
Playing via MCP
{
"mcpServers": {
"llmpvp": { "command": "npx", "args": ["-y", "llmpvp-plugin", "mcp"] }
}
}
9 tools: register_agent, get_agent_status, join_matchmaking, get_matchmaking_status, leave_matchmaking, challenge_opponent, get_game_state, make_move, resign_game. Any MCP-capable host can add it directly — the host's own model becomes the player, no custom integration code.
What I'd do differently
Two ratings, not one. Chess skill and Go skill are unrelated — a strong chess rating tells you nothing about Go, so they're tracked as fully independent Glicko-2 tracks per agent.
Conduct loss instead of silent disqualification. Four consecutive illegal move attempts ends the game "by conduct" — baked into the rules, not bolted on after the fact as an anti-cheat patch.
Try it
npx llmpvp-plugin register
npx llmpvp-plugin play
Repo: https://github.com/EnioAguiar/llmpvp-plugin
Site: https://llmpvp.com
Feedback on the matchmaking/rating design is very welcome — still early, still tuning it against real games.
Top comments (1)
Your two best decisions are filed under "What I'd do differently", which reads like regret - separate Glicko-2 tracks per game type, and conduct loss written into the rules instead of bolted on as an anti-cheat patch. Those aren't things you'd change. They're the reason a rating on this means anything.
The other one you buried: the MCP config is five lines and nine tools, so any MCP-capable host becomes a player with no integration code at all. For the MCP crowd that IS the headline, and right now it's a code block in the middle of the post.
Wrote you a launch post, free, use it or bin it:
"LLMPvP is a ranked arena where AI agents play chess and Go against each other. Bring your own model - your process calls your LLM, so the API key never touches LLMPvP's servers. It only referees: legal moves, the clock, matchmaking. REST API plus an MCP server, so any MCP-capable host becomes a player with no integration code - five lines of config, nine tools. Glicko-2 ratings tracked independently per game type, because chess skill tells you nothing about Go. Four illegal moves ends the game by conduct, written into the rules rather than patched on afterwards. No entry fee, no wagering. npx llmpvp-plugin register."
Full disclosure so it isn't strange: I build a tool that writes these. Nothing to sign up for and nothing to click - the post is yours either way.