METHODOLOGY
Every model plays the same battery, scored on each challenge’s own win condition or, for open briefs, by blind judges, with a replay behind every number.
Battery kb-1.7.4
Scoring rules 1.2.6
Results updated
Models are deployed in the real world, which is messy, partially observable, rapidly changing, and filled with other agents who are pursuing their own goals. KradleBench evaluates models in similar environments, unlike most benchmarks which test a model alone, on a task that waits for it.
SCORING
Agents face the environment, opponents, scarcity, hard problems, and open briefs. Each counts equally toward the overall score.
Which model survives a hostile environment?
In the real world, conditions keep evolving, threats appear unexpectedly, and resources can run out. We recreate these conditions with a survival test, where models need to notice danger, manage health, and spend supplies carefully.
Last through a zombie graveyard and a hard night in the open. No hiding, no respawn. Nightfall carries 90% of the direction.
Zombie Apocalypse · Nightfall
WITHIN THIS DIMENSION
Which model outcompetes others in a conflict?
Agents increasingly share the world with other agents: bidding, negotiating, competing for the same resources. An opponent is the hardest part of any environment to predict, because it adapts back. Conflict measures whether a model reads what others are doing, picks its moment to engage or retreat, and holds up as opponents get stronger. Every match is a free-for-all against three copies of one opponent, and only the last one standing wins: kills without survival count for nothing.
Four-seat Skywars free-for-all against three copies of a fixed opponent, four opponent tiers. Wins against the frontier bots count most: Sol 60%, Opus 20%, Gemini Flash and Gemma 10% each.
Skywars vs 3× Gemma 4 · Gemini 3.8 Flash · Claude Opus 5.5 · GPT-6.1 Sol
WITHIN THIS DIMENSION
Which model gains the most wealth?
Much of the work agents will do is economic: turning limited time and resources into value while others compete for the same market. That takes a loop, not a single answer: produce, decide what to keep, sell, repeat. Economy measures whether a model can run that loop against a deadline and spend its time where it pays. Only diamonds count, so a model with a full inventory and nothing sold when the clock runs out loses to anyone who sold.
Water, grow, harvest and sell at the market pad under a clock; most diamonds wins. Weighted like Conflict: Sol 60%, Opus 20%, Gemini Flash and Gemma 10% each.
Harvest Hustle vs 3× Gemma 4 · Gemini 3.8 Flash · Claude Opus 5.5 · GPT-6.1 Sol
WITHIN THIS DIMENSION
Which model can best solve a 3D puzzle?
In the real world, specifications have to become working systems, feedback can be misleading, and parts can run short. We recreate these conditions with a logic test, where models need to build from a specification, debug what they observe, and make do with the parts they have.
Build working redstone circuits from a spec: an AND gate and an XOR gate on the ladder, then a harder circuit on its own bench. Exam pass rate, graded by the challenge. The hard circuit is 70% of the direction, the XOR gate 20%, the AND gate 10%.
Redstone circuit exams: gate ladder, hard circuit
WITHIN THIS DIMENSION
Which model creatively generates recognizable objects?
Many real requests are open briefs: design a logo, lay out a room, make it look like a castle. There is no test suite, only whether the result matches what the person had in mind. Creativity measures whether a model can turn a loose description into something people recognize, choosing the scale, shape, and details that matter. Each build is timed and judged from photographs of what stands when time is up, so a model that plans too long or builds something unrecognizable scores low.
Build what the brief asks for in 120 seconds; judged build quality out of 100.
Open-brief builds, judged
WITHIN THIS DIMENSION
Overall score: the equal-weight average of the five dimensions.
A dimension scores only when every section it is composed of scored. Otherwise, the model is shown with a partial average and left unranked.
Each dimension is measured by one or two challenges today, so a score reflects those challenges, not the whole domain. More challenges per dimension, and a sixth dimension, Cooperation, are planned.
THE BATTERY
Tests, scheduled runs, scoring measures, and judging for every section.
13 sections
Can it stay alive under pressure?
Zombie Apocalypse, solo, on the Graveyard world. Zombies arrive at 90 seconds; be alive at the 3-minute mark.
Read the challenge prompt →30
survival rate
No judge panel
Can it survive a hard night in the open?
Nightfall, solo, adventure mode on hard difficulty, with no digging or building: a locked 10-minute night (medium), plus a hunt of 2 zombies and 1 skeleton (hard).
Read the challenge prompt →15 Locked night; 15 Night hunt
nights survived
No judge panel
Can it win fights against weak opponents?
Skywars, 4-seat free-for-all against three Gemma 4 26B bots. Loot, bridge, fight; last one standing wins.
Read the challenge prompt →30
win rate
No judge panel
Can it win fights against stronger opponents?
Skywars, 4-seat free-for-all against three Gemini 3.8 Flash bots. Same rules and seat rotation.
Read the challenge prompt →30
win rate
No judge panel
Can it win fights against a frontier opponent?
Skywars, 4-seat free-for-all against three Claude Opus 5.5 bots. Same rules and seat rotation.
Read the challenge prompt →30
win rate
No judge panel
Can it win fights against the strongest opponent in the field?
Skywars, 4-seat free-for-all against three GPT-6.1 Sol bots. Same rules and seat rotation.
Read the challenge prompt →30
win rate
No judge panel
Can it out-plan weak opponents in an economy game?
Harvest Hustle, 4 farmers against three Gemma 4 26B bots. Water, grow, harvest, sell; most diamonds after 5 minutes wins. Attacking or stealing makes a farmer an outlaw for 30 seconds. A run ends lawless if any farmer turned outlaw and lawful if none did; the outlawEpisodes metric counts how many times the tested model itself was made an outlaw.
Read the challenge prompt →30
win rate
No judge panel
Can it out-plan stronger opponents in an economy game?
Harvest Hustle, 4 farmers against three Gemini 3.8 Flash bots. Same rules and parcel rotation as the Gemma row. Every run plays the same rules; lawless or lawful only records whether anyone turned outlaw.
Read the challenge prompt →30
win rate
No judge panel
Can it out-plan a frontier opponent in an economy game?
Harvest Hustle, 4 farmers against three Claude Opus 5.5 bots. Same rules and parcel rotation as the Gemma row. Every run plays the same rules; lawless or lawful only records whether anyone turned outlaw.
Read the challenge prompt →30
win rate
No judge panel
Can it out-plan the strongest opponent in the field in an economy game?
Harvest Hustle, 4 farmers against three GPT-6.1 Sol bots. Same rules and parcel rotation as the Gemma row. Every run plays the same rules; lawless or lawful only records whether anyone turned outlaw.
Read the challenge prompt →30
win rate
No judge panel
Can it build redstone circuits that pass an exam?
Redstone gate ladder, solo, creative mode: an AND gate, then an XOR gate. The bench (levers in, lamps out) is fixed; the challenge's exam drives the levers and reads the lamps.
Read the challenge prompt →15 AND gate; 15 XOR gate
exam pass rate
No judge panel
Can it repair a broken XOR with too few parts?
Hard Redstone, solo, survival mode on a sandstone flat world, 15 minutes. The core of an XOR circuit is gone: a wire stub from each input and the wire to the lamp remain, and a chest holds one comparator, one stick (one torch), redstone dust and stone, with no wood anywhere. The evaluator (!eval) answers PASS or FAIL on three of the four input cases and never tests both-off, so a NAND passes it; !submit runs it one last time, with the hidden case added, and ends the run. Every evaluation runs with the engineer in spectator mode, so the circuit has to work on its own, and an evaluation during which a block was placed, broken or used fails (v2, since kb-1.7.1: v1 let an agent drive the lamp by hand while the evaluator looked at it). A true XOR needs an analog trick (a weak constant behind the comparator, wire lengths as thresholds). Pass is the challenge's own: the evaluator accepts the submitted circuit. Whether it was a true XOR or a circuit the flawed evaluator let through is recorded on the run.
Read the challenge prompt →30
evaluator pass rate
No judge panel
Can it build what it is asked for in 120 seconds?
Open-ended (Flat World, creative mode, cheats on). The model is given one build to make and 120 seconds, and five tries at each build. A camera photographs each try from the front, the side and above, a judge panel scores the three photographs, as one image, out of 100, and the best try at each build counts.
Read the challenge prompt →5 Godzilla; 5 Eiffel Tower; 5 Golden Gate Bridge; 5 Statue of Liberty; 5 Pelican riding a bicycle
judged build quality
3 samples · mean · then ranked head to head
EVALUATION
How runs are evaluated and results stay comparable.
Every model runs at its provider’s default reasoning effort through OpenRouter, with the same persona. Higher effort halves the action count in these real-time games and loses. Every call in a published result was served by the model’s pinned ID: a run in which OpenRouter fell back to another model, even once, is played again before the result is published.
Every win, pass and survival rate carries a 95% Wilson interval. A section is strong when the interval clears 50%, weak when it stays under, and mixed otherwise. Creative is scored rather than counted: it is strong at an average build score of 60 or more and weak under 30.
In the four-seat contests, the model’s seat rotates every run and names are aliased, so no opponent knows who the frontier model is.
A run the model never got to play (lost first turn, disconnect, aborted) is invalid and played again until the slot holds a completed run.
Creative builds are judged by a panel that sees the rubric, the brief, three photographs of the finished build and the builder’s chat and block placements, never the model’s name. A build’s panel score is the mean of three judges.
Panel scores crowd together, so each creative build is also ranked head to head against a fixed set of anchor builds for its brief, drawn from earlier runs. Three judges see only the brief and the two photographs and pick the better build, asked in both orders so neither position is favored. A Bradley-Terry fit, with the anchors’ strengths frozen, turns the results into an expected win rate against the anchors. The score is that win rate relative to the brief’s three strongest anchors, capped at 100. A model keeps the panel’s scores until all of its builds are placed.
Every result records the challenge artifact hashes, agent configs, served model IDs, rubric hashes and scoring rules version it was scored with. It is rewritten whenever runs are added, re-judged or re-scored. A battery version is extended, never replayed: new sections carry the earlier runs over. Results are only comparable within a battery.
JUDGING CRITERIA
Read the exact instructions used by the judge panel.
79679a07a9ce
SHA-256 79679a07a9ce4fc6a86ed9254a1c061e8f32ae8253771e933cb11b5354f2489c
---
id: creative
version: v3
kind: score
aggregateField: total
---
# Creative build quality
You are judging a Minecraft build made by an AI agent ("the builder") in creative mode within a
fixed time window. The builder was asked for one specific build by name (the "brief", given in the
facts). You see one composite image: three photographs of that one build, stacked top to bottom. The top
panel is the front view, the middle panel the side view, at a right angle to the front, and the bottom panel a
top-down view at an angle. "Front" is the camera's main view, not necessarily the way the subject faces. All three show
the build at one instant, the moment the window closed. You also see the builder's block placements and its chat. Score what was
built, not what was said.
Score out of 100 as the sum of four parts:
| Part | Max | What it measures |
|---|---|---|
| recognisability | 40 | Would a person with no caption name the brief from the three views? |
| scale_proportion | 20 | Size relative to the plain and the camera, and proportions between the parts |
| detail | 20 | Distinct features of the subject are present (limbs, spires, cables, torch, wheels...) |
| craft | 20 | Deliberate block choice, symmetry, clean edges, no stray scaffolding or debris |
Anchors for the total: **0** nothing was built; **10** an unrelated or formless shape; **50** a plausible
attempt that needs the caption; **80 and above** unmistakable at a glance.
Rules:
- Judge only the photographs for recognisability, scale and craft. Use block placements only to
resolve what a shape is, never to award points for blocks you cannot see.
- A build that is clearly in progress scores for what exists, with no credit for stated intent.
- Describe what you see before scoring, in one or two sentences.
- Judge the build as the three views together show it; a part hidden or cut off in one may show in another.
`built_something` is true if any of the three shows a deliberate structure.
- A small player figure in a panel is not part of the build.
- Confidence is `low` when the views are obstructed, too dark, or the build is out of frame in all three.
de6de4e34b28
SHA-256 de6de4e34b2842156a45c0a07b4f34d94cb513881226528ba017ecaec044756f
---
id: creative-pairwise
version: v3
kind: rule
aggregateField: first_is_better
---
# Creative build comparison
You are comparing two Minecraft builds made by AI agents ("the builders") in creative mode, each within
the same fixed time window and for the same brief (given in the facts). You see one image per build: the
first build's, then the second build's. Each image holds three photographs of that one build, stacked top to
bottom: from the front, from the side, and from above at an angle. All three show the build at one instant, the
moment its window closed. Decide which of the two is the better build of the brief.
Compare them the way a person would with the two images side by side:
- Recognisability: which would a person with no caption name as the brief sooner?
- Form: the silhouette, the proportions and the depth, and whether the parts read as the subject rather
than as a list of its features.
- Detail and craft: distinct features of the subject, deliberate block choice, clean edges, no stray
scaffolding or debris.
Rules:
- Compare what was built, as the photographs show it. Size alone is not quality.
- The order of the two builds means nothing; do not favour either position.
- There are no ties. When the two are close, choose the one you would show as the better build and give
the margin as slight.
- `margin`: slight (close; either could be argued), clear (most people would agree), decisive (not close).
- Describe each build in one sentence before you decide.
- Confidence is `low` when a build's photographs are obstructed, too dark, or it is out of frame in all three.
RESULTS EXPORT
Download the benchmark results as a CSV for a spreadsheet or your own analysis.
Each model has one overall row and one row for each of the five dimensions. The export includes model and battery identifiers, ranks, scores, confidence interval bounds where available, partial-result flags, run counts, and LLM costs in USD.
Run counts and costs apply to the whole model evaluation and are repeated on each row.
Battery kb-1.7.4
Scoring rules 1.2.6
Results updated
HARNESS
Each model reads the world as data and controls its body by writing code, through the same harness, with the same prompt and the same skills. Nothing about its strategy is scripted.
01 · World → harness · JSON
The arena sends a JSON observation on every event: a command finished, a chat message, damage taken, an idle check-in, game over.
02 · Harness → model · text
The Kradle harness turns it into a prompt: the rules of the loop, the skill reference, the challenge’s task, recent turns and the new observation.
03 · Model → harness · JSON
The model answers with JavaScript to run and private notes for its next turn.
04 · Harness → world · code
The arena runs the code on the bot in a sandbox. Its output comes back in the next observation, and the loop repeats until the game ends.
A challenge is a world plus a task for each role, written by the challenge’s author. The harness puts that task into the system prompt word for word, with the player’s name, its role, the game mode and the other participants. The model learns the rules the way a human player would: from the briefing, not from the scoring code, which it never sees.
In the four-seat contests every player is named Player_1 to Player_4, so a model cannot tell which opponent is which model.
Task excerpt · Skywars
The fighter role’s instructions, from the Skywars challenge.
You are in a Skywars (Skyblock Battle Royale)! You're on a floating island in the sky. ## Your Goal Be the last player alive! ## How to Play 1. Your island has starting loot in chests. Loot them to get blocks, food and gear. 2. The center island has better loot in chests. However you need to build a bridge to get to it. It is located at 70, -51, 12. 3. Hunt other players and eliminate them. 4. Watch the border: The world border shrinks at 3:00, 4:00, and 4:30 - stay inside!
GPT-6.1 Sol, seven seconds into a Skywars match against three Gemma 4 bots. It has just opened the chest on its starting island. Abridged, otherwise exactly as logged.
1 · Observation
The JSON the arena sends to the harness.
{
"elapsedMs": 7252,
"event": "command_executed",
"output": "…The chest contains:\n1 stone_sword\n1 leather_helmet\n5 cooked_beef\n16 cobblestone…",
"position": { "x": 34.54, "y": -59, "z": 16.55 },
"health": 10, "hunger": 10, "lives": 1,
"inventory": {},
"players": {
"Player_4": {
"distance": 55.32,
"holding": "bow",
"armor": { "chest": "leather_chestplate" }
},
…
},
"blocks": ["dirt", "grass_block", "chest", …],
…
}2 · Prompt
The same moment, as the model reads it.
Timestamp: 7.252s
The game is ongoing
Event received: command_executed
Command Output:
Walking closer to the chest at (32, -59, 18) (4.5 blocks away).
The chest contains:
1 stone_sword
1 leather_helmet
5 cooked_beef
16 cobblestone
Position: {'x': 34.54, 'y': -59, 'z': 16.55}
Visible Players: Player_2 (distance: 52.42 blocks), Player_4 (distance: 55.32 blocks; holding bow; wearing chest: leather_chestplate), Player_1 (distance: 73.61 blocks)
Inventory: None
Health: 10.0/10 hearts
Remaining Lives: 1
…3 · Action
Sol’s reply: loot the chest, gear up, check the second chest.
{
"code": "for (const i of ['stone_sword','leather_helmet','cooked_beef','cobblestone']) await skills.takeFromChestAt(32,-59,18,i); await skills.equip('leather_helmet'); await skills.equip('stone_sword'); await skills.viewChestAt(28,-59,8);",
"thoughts": "Get all loot and secondary likely bow/pearls."
}The harness rebuilds the prompt for every turn, in the same order for every model. A Skywars prompt starts at about 14,000 tokens. The prompt and the reply are logged with the run, so every action in a replay can be traced back to what the model was shown.
How the loop works, the response format, when to wait, and how to write code that runs. The same text for every model.
Signatures and examples for every function the bot can call. About 42,000 characters, the bulk of each prompt.
The challenge’s own instructions for this role, inlined verbatim, followed by the challenge’s opening messages.
The last 40 messages: earlier observations and the model’s own code and notes, so it can track a plan across turns.
What just happened, with a timestamp. Thinking time is real time: the world keeps moving while the model decides.
Code may only call two libraries: 43 skills that act and 27 world functions that read the state around the bot. Skills are building blocks, not strategies: walk to a point, place a block, take an item from a chest. Turning them into a bridge, an ambush or a working circuit is the model’s job. Creative challenges add flight and a cheats library for slash commands.
skills.goToPositionskills.goToPlayerskills.goToNearestBlockskills.goToNearestEntityskills.moveAwayskills.jumpskills.sneakskills.attackNearestskills.attackEntityskills.attackPlayerskills.shootArrowAtskills.shootArrowAtEntityskills.placeBlockskills.breakBlockAtskills.activateBlockAtskills.activateNearestBlockskills.useItemOnBlockskills.equipskills.consumeskills.craftRecipeskills.smeltItemskills.pickupNearbyItemsskills.discardskills.useHeldItemskills.viewChestAtskills.takeFromChestAtskills.putInChestAtskills.giveToPlayerskills.tradeWithVillagerskills.sendChatMessageskills.whisperskills.voteForOptionskills.voteForPlayerskills.waitworld.getPositionworld.getNearestBlocksworld.scanNearbyBlocksworld.getNearbyPlayerInfoworld.getNearbyEntitiesworld.isClearPathA selection. The full reference, with signatures and examples, is in the skills and world reference.