Today, we’re announcing KradleBench, a dynamic, multi-agent benchmark for frontier models and humans.
Our top human was #1 until yesterday, when he was unseated by GPT 6.2 Astra. The OpenAI models now dominate KradleBench by being both intelligent and fast, which lets them constantly refine their strategy and adapt to their environment. Conversely, the Anthropic models scored lower by attempting to one-shot their strategy, instead of re-evaluating their situation.
Static evals, where models work on a problem alone and get a grade, are no longer sufficient. They assume the world is static, with no one to interfere with their plan, but it’s not: it is messy, noisy, and full of adversarial entities.
We built KradleBench to understand how models will behave in conditions more similar to the real-world, where there are others pursuing their own goals (“multi-agent”) and conditions are constantly changing (“dynamic”).
KradleBench puts models in live Minecraft worlds, where they must complete long-running tasks alongside other agents, humans or AI. Every run is recorded and can be replayed for everyone to assess the prowess and limitations of frontier AI.
What KradleBench reveals
The first stark insight is that in a live world, performance is a tradeoff between intelligence and speed. Some models are quick to react but make bad decisions. In this battle royale challenge, Muse Ember quickly collects blocks to build a bridge, but doesn’t realize it’s digging too deep and falls into the void, eliminating itself.
On the other hand, some models spend too much time thinking and don’t react to their environment in time. In this survival challenge, Gemini Carbon thinks very hard about a plan to avoid drowning… and drowns while thinking.
With the right intelligence at the right speed, the OpenAI family dominates our benchmark. In this test, models must coordinate to build a pyramid in a very specific sequence. Three GPT 6.1 Sols perfectly synchronize and finish the build 15 seconds before the deadline.
But speed is not the only factor. While GPT models are constantly adjusting their plans based on new information, Anthropic models are overthinkers, trying to one-shot every task, regardless of who they’ll interact with later. In this economy game, models need to grow crops and sell them at the market for diamonds. Fable builds a loop, starts executing against it, and revisits it only once for a 30% improvement. Meanwhile, Astra upgrades its loop 3 times, improving it by 239%, collecting 2x more diamonds than Fable.
Finally, humanity is neck-and-neck on KradleBench with the optimum tradeoff between intelligence, creativity, and speed. Our representative, SethBling, a long-time Minecraft player, beats models in competitive situations by taking advantage of the models’ thinking times and coming up with creative strategies.
Why this matters
There are millions of agents being deployed in the real world, soon billions. The real world is not static, it is multi-agent and dynamic: these models will be negotiating on our behalf with other models, protecting us from cyberattacks launched by other models, and controlling robots which will need to react to messy situations. We need models that perform well in these conditions, not in static theoretical environments.
Existing benchmarks miss this signal, which is why they show Anthropic models ranking higher. They also inevitably saturate, as they represent a static goalpost which AI labs can hillclimb against. But a dynamic benchmark won’t saturate, as every model that tops it becomes the new baseline to beat, and we can add more depth to its challenge to discriminate between the leaders.
Nature created the only known AGI, our brain, through an intelligence arms race. Our brain has been optimized for a 3D social world, with group dynamics, politics, negotiation, all dynamic interactions which humans have learned to master over centuries.
We will soon open KradleBench so more humans can take the test and propose new evals. This will create a petri dish where any intelligence type (LLM, World Model, Human) can be compared and generate useful information to improve current systems, whether it’s on capabilities or safety.
Will GPT 6.2 Astra stay on top? Will a human player dethrone it? Sign up here to receive our report of every new major model, to take KradleBench as a human, to propose a new eval, or if you’d like your new model to get tested.


Looking good. LFG Kradlers! 🚀🚀🚀🚀