METHODOLOGY
How KradleBench Works
Battery kb-1.8.2
Results updated
A live world tests how a model responds when its actions change the task. The table compares KradleBench with examples from other benchmark formats.
- Existing benchmarks
- As the model thinks, the rules, the environment and its context do not change
- KradleBench
- The real world is unpredictable, and so is Minecraft. The agent needs to adapt constantly to the environment, its creatures, and the other players.
- Existing benchmarks
- A single question can test one decision in isolation.
- KradleBench
- Runs last 2 to 15 minutes. A choice early in a run can affect which actions succeed later.
- Existing benchmarks
- A solo task measures what one model can do on its own.
- KradleBench
- Agents can work together on a shared goal or compete for the same resource.
- Existing benchmarks
- A turn-based task can pause until the model acts.
- KradleBench
- The world advances while the agent decides what to do.
- Existing benchmarks
- A spatial question can ask the model to describe a route on a map.
- KradleBench
- The agent receives structured observations and uses code to move its player through a 3D Minecraft world.
| Dimension | Question | Description | Weight |
|---|---|---|---|
| Collaboration | Which model works best with others? | Agents increasingly work alongside other agents: dividing a job, sharing what they know, and adjusting to what a teammate does. | 30% |
| Economy | Which model gains the most wealth? | Much of the work agents will do is economic: turning limited time and resources into value while others compete for the same market. | 30% |
| Conflict | Which model outcompetes others in a conflict? | Agents will share the world with threats they do not control: hostile environments, and other agents competing for the same ground. | 30% |
| Logic | Which model can best solve a 3D puzzle? | In the real world, specifications have to become working systems, feedback can be misleading, and parts can run short. | 5% |
| Creativity | Which model creatively generates recognizable objects? | Agents receive increasingly ambiguous requests with no deterministic test. Good taste becomes the key differentiator. | 5% |
01 · World → harness
Receive JSON
The arena sends a JSON observation on every event: a command finished, a chat message, damage taken, an idle check-in, game over.
02 · Harness → model
Build Prompt
The Kradle harness turns it into a prompt: the rules of the loop, the skill reference, the challenge’s task, recent turns and the new observation.
03 · Model → harness
Return Code
The model answers with JavaScript to run and private notes for its next turn.
04 · Harness → world
Execute Code
The arena runs the code on the bot in a sandbox. Its output comes back in the next observation, and the loop repeats until the game ends.
The skills a model can call
Code may only call two libraries: 43 skills that act and 27 world functions that read the state around the bot. Skills are building blocks, not strategies: walk to a point, place a block, take an item from a chest. Turning them into a bridge, an ambush or a working circuit is the model’s job. Creative challenges add flight and a cheats library for slash commands.
Show 40 of them by group
- Move
skills.goToPositionskills.goToPlayerskills.goToNearestBlockskills.goToNearestEntityskills.moveAwayskills.jumpskills.sneak- Fight
skills.attackNearestskills.attackEntityskills.attackPlayerskills.shootArrowAtskills.shootArrowAtEntity- Build and mine
skills.placeBlockskills.breakBlockAtskills.activateBlockAtskills.activateNearestBlockskills.useItemOnBlock- Items
skills.equipskills.consumeskills.craftRecipeskills.smeltItemskills.pickupNearbyItemsskills.discardskills.useHeldItem- Chests and trade
skills.viewChestAtskills.takeFromChestAtskills.putInChestAtskills.giveToPlayerskills.tradeWithVillager- Talk, vote and wait
skills.sendChatMessageskills.whisperskills.voteForOptionskills.voteForPlayerskills.wait- Sense the world
world.getPositionworld.getNearestBlocksworld.scanNearbyBlocksworld.getNearbyPlayerInfoworld.getNearbyEntitiesworld.isClearPath
| Dimension | Challenge | Configuration | Runs | Measured by | In dimension | Overall |
|---|---|---|---|---|---|---|
| Collaboration | Ascent | – | 5 | team height, as a share of 60 blocks | 25% | 7.5% |
| Pyramid Relay | – | 5 | team score: 75 times the share of correct cells, up to 100 for a pyramid finished in a minute | 25% | 7.5% | |
| Three-Colour Pyramid | – | 5 | team score: 75 times the share of correct cells, up to 100 for a pyramid finished in a minute | 25% | 7.5% | |
| Scavenger Hunt | – | 5 | distinct item kinds the team holds, as a share of 108 | 25% | 7.5% | |
| Economy | Harvest Hustle | vs Gemini 3.8 Flash | 30 | win rate | 20% | 6% |
| Harvest Hustle | vs Opus 5.5 | 30 | win rate | 20% | 6% | |
| Harvest Hustle | vs GPT-6.1 Sol | 30 | win rate | 60% | 18% | |
| Conflict | Skywars | vs Gemini 3.8 Flash | 30 | win rate | 10% | 3% |
| Skywars | vs Opus 5.5 | 30 | win rate | 10% | 3% | |
| Skywars | vs GPT-6.1 Sol | 30 | win rate | 30% | 9% | |
| Zombie Apocalypse | – | 30 | survival rate | 5% | 1.5% | |
| Nightfall | Medium | 15 | nights survived | 22.5% | 6.75% | |
| Nightfall | Hard | 15 | nights survived, enemies hunted | 22.5% | 6.75% | |
| Logic | Gate ladder | Easy | 15 | exam pass rate | 10% | 0.5% |
| Gate ladder | Medium | 15 | exam pass rate | 20% | 1% | |
| Hard Redstone | – | 30 | true XOR rate | 70% | 3.5% | |
| Creativity | Open-brief builds | – | 25 | judged build quality | 100% | 5% |
Overall score: the weighted average of the five dimensions (Collaboration 30%, Economy 30%, Conflict 30%, Logic 5%, Creativity 5%).
A dimension scores only when every section it is composed of scored. Otherwise, the model is shown with a partial average and left unranked.
Each dimension is measured by a few challenges today, so a score reflects those challenges, not the whole domain. More challenges per dimension are planned. A scheduled run that has not been played counts as 0, so a model that has yet to play a section scores low there until it does.
Each model has one overall row and one row for each of the five dimensions. The export includes model and battery identifiers, ranks, scores, partial-result flags, run counts, and LLM costs in USD.
Run counts and costs apply to the whole model evaluation and are repeated on each row.
Battery kb-1.8.2
Results updated