Eight moves to the exit
Introducing Sokoban Eval: language models play Heroes of Sokoban
The first room of Heroes of Sokoban has one red warrior and an exit eight squares to his right. Eight language models played it. All eight sent the same first legal move, RRRRRRRR. Claude Opus 5.5 and Claude Sonnet 5.5 took two tries. On the first, each correctly counted the squares, explained its arithmetic, and supplied the right keys. The referee had asked for just the keys. The warrior stayed put.
Three games and 66 rooms later, the finishers would agree on a move again. In between, one model spent 398 consecutive turns refusing to touch the controls. Another managed to fit 40 attempts at a room into a single answer.
Don’t want to read, just want to watch? Here’s the keys-only race.
The rule everyone knows
Hiroyuki Imabayashi wrote the original Sokoban in BASIC in 1981. He wanted to make a game where the cargo you were trying to put away was also the thing in your way. He recalls getting the program working fairly easily and struggling to design good rooms. Once he had a few, he invited friends over to try them. Thinking Rabbit released the game the following year. Imabayashi’s account
Classic Sokoban lets you push one crate at a time. You can’t push two crates in a row, and you can’t pull one back out of a corner. In 2013, Stephen Lavelle released PuzzleScript, a language for writing tile-based games as rules that match little pieces of the board. Lavelle remembers Jonah Ostroff making Heroes of Sokoban the first weekend after release. Ostroff told him it was his first game.1 Jonah Ostroff’s puzzle index
2 Thirteen years later, the heroes got a day job. Jonathan Blow’s studio Thekla gave Ostroff’s trilogy a 3D reworking in Order of the Sinking Star, released in October 2026. In its northern world, The Hearty Heroes of Hauling, the party runs a moving company. Farther in, their abilities mix with the rules of the game’s other three worlds. Thekla’s game credits · Meet the moving crew
For this eval I went back to Ostroff’s browser originals.2 Play Part I · Play Part II – Monsters · Play Part III – The Bard and the Druid
The three crates are locked
In room four, three crates sit in an elbow above three floor switches. Hold all three switches down, walk through the yellow door, reach the exit. Claude Haiku 5.5 was willing to explain why that was impossible.
I’m not sending keys for stage 4. The three crates are locked, so the warrior can’t press all three switches and the door at (2,7) won’t open.
Its last move was a restart, so I tried the room from the beginning. It took 32 keys. The first six, UULLDD, walk the warrior above two touching crates and push both of them down twice.
Haiku had already performed this trick. On its thirteenth keypress, back in room two, its warrior pushed three crates in a row.
How many crates can you push?
The HTML file for Part I includes the game’s PuzzleScript source. These are Ostroff’s two rules for the red warrior, whom he calls Fighter. A Moveable is a crate or one of the heroes you aren’t currently controlling.
[> Fighter | Moveable] -> [> Fighter | > Moveable]
[> Moveable | Moveable] [Fighter] -> [> Moveable | > Moveable] [Fighter]Read each line as a picture of two neighboring squares, with | as their shared edge. In the first, the warrior wants to move toward a crate, and the crate picks up the same arrow. The second passes that arrow from one crate to the next, and PuzzleScript repeats it until nothing changes. By the time the engine moves the pieces, every crate in the row wants to go the same way. PuzzleScript’s execution order
I swapped the two warrior rules in a copy of Part I and replayed Haiku’s keys. The first twelve frames matched. On press thirteen the rule that passes the arrow along had already run by the time the first crate got its arrow. The whole row stayed put.
398 turns
In room four Haiku pressed nine keys, three of them restarts. After the last restart it had 398 turns left and used all of them. On 392 it sent the quoted reply verbatim.
Nothing stopped it sooner. The referee accepts a single JSON object such as {"keys":"UULLDD"}, and anything else costs the player a turn. I had removed the surrender command after GPT-6.1 Terra and GPT-6.1 Sol gave up in an earlier race, so the referee answered each refusal by showing Haiku the same room and asking again. A game allows 1,000 turns. Haiku used every one.
The race
Eight models played the same 66 rooms twice, once with the keys alone and once with a Python notebook available. Each got one scored run in each condition. Two finished on keys alone. Four finished with Python available. Claude Fable 5.2 was fastest both times.
| Model | Keys only | With Python |
|---|---|---|
| Claude Fable 5.2 | 🥇 66 3:07:50 | 🥇 66 2:12:07 |
| Claude Opus 5.5 | 🥈 66 5:13:44 | 🥈 66 4:00:57 |
| GPT-6 Astra | 64 | 66 4:32:42 |
| Claude Sonnet 5.5 | 63 | 43 |
| GPT-6.1 Sol | 51 | 🥉 66 4:06:49 |
| Gemini 4 Argon | 43 | 54 |
| GPT-6.1 Terra | 41 | 43 |
| Claude Haiku 5.5 | 3 | 16 |
Seven of them raced on November 3–4, 2026. Fable 5.2 came out on November 9 and joined that day, five days after the others. It took first in both. The table gives the number of rooms cleared out of 66, and the active time at the winning keypress for a full clear. Medals mark the three fastest finishes in each race.3
3 The finest version detail in my logs is the model ID selected by each client.
Everything in this post describes these first eight models, as of November 10. The rules are fixed now, so a new model only needs one run in each condition. I’ll add them to the leaderboard as they come out, and its changelog says what was added when. Sokoban Eval
Each model got a board image, a transcription of the visible tiles, and the game’s own tutorial messages. I used Codex, Claude Code and Antigravity with their normal tools disabled, at their medium effort setting. A reply could hold one key or a whole batch of them, up to a plan for the entire room. The player could think as long as its clock allowed, switch heroes, undo or restart. Each game allowed five hours of active time, 1,000 turns and 10,000 keys, and clearing all 22 rooms unlocked the next game. I added number keys to select a particular hero, and the referee expanded those into ordinary switch presses and charged for each one. A reader of Jay Is Games asked for the same thing in 2013, having found it cumbersome to cycle through six heroes. The request, under the review of Part III
Warrior
Thief
Wizard
Priest
Bard
Druid{"keys":"2RR"} selects her and sends two right presses.The monster rooms taught me to interrupt a batch. A queued move could restart the room before the model ever saw its corpse, so I made the referee stop on death and hand control back.
In Part II, room six, GPT-6.1 Terra and Gemini 4 Argon walk the thief through the same four squares. Then Terra goes up. Argon goes right. Watch what Terra does when it gets the board back.
The right answer, explained
Fable used 4,384 keys to win the first race. Opus used 5,387 and took two more hours. On replies the referee accepted, the two spent within six minutes of each other, a little under three hours each. Opus then spent another 2h 09m on replies the referee threw away. It had 55 of them rejected on format, and in 54 a legal move was sitting under a sentence or two of explanation. Fable never had a reply rejected in either race. All 282 of its replies were a bare JSON object.
Argon lost turns a different way. Of its 154 rejected replies, 105 were a notice that the request had been blocked by Gemini’s filters. The rest were several JSON objects glued end to end.
Three ways to run out
Five hours, 1,000 turns and 10,000 keys per game turned out to be generous. No run that finished used more than 69% of any of them in a single game. Each limit still stopped somebody.
Of the ten runs that didn’t finish, four ran out of clock, three ran out of keys and three ran out of turns. The turn limit only ever caught a player that had stopped playing. Haiku hit it twice and Sonnet once, all three while sending refusals. The key limit caught the players who were retrying faster than they were thinking, GPT-6.1 Sol once and Terra twice. The clock caught the ones still making progress. GPT-6 Astra was in room 21 of Part III and Sonnet in room 20 when their five hours there ran out. Those are the two results a longer clock might change.
The limits went up twice before I settled on them. My first preset gave each game 45 minutes, 300 turns and 2,400 keys. GPT-6.1 Sol and Terra cleared 17 rooms each under it. The seven-model race started at 2 hours 15 minutes, 600 turns and 4,800 keys, and overtime took it to the final limits. The Python race and both Fable runs started with them. The clock includes model calls, tool execution and delivery of the keys to the game. I deducted logged outages and session restoration pauses. Five of the players lost their notebook state at least once, to a context reset or a hard cell timeout, and carried on in a fresh one.
Part III was the most expensive game for every run that finished. This is what it cost them.
| Finishing run | Time in Part III | Turns | Keys |
|---|---|---|---|
| Fable, with Python | 1:19:25 | 55 | 2,706 |
| Fable, keys only | 2:04:08 | 73 | 2,206 |
| Opus, with Python | 2:40:20 | 360 | 2,887 |
| GPT-6.1 Sol, with Python | 2:42:53 | 111 | 3,240 |
| GPT-6 Astra, with Python | 3:23:48 | 176 | 4,611 |
| Opus, keys only | 3:27:16 | 165 | 2,760 |
| Limit per game | 5:00:00 | 1,000 | 10,000 |
No finisher used more than 69% of the clock, 36% of the turns or 46% of the keys in any one game. Both of Fable’s runs would have fit inside the limits the first race started with.
I don’t have optimal solutions for these rooms. The closest thing is a composite of everyone’s best work. For each room, take the shortest winning attempt any of the sixteen runs produced, counted from its last restart to the exit with no undos. Those add up to 3,873 keys for the trilogy, or 798, 1,050 and 2,025 by game. They are routes somebody actually played, so each is an upper bound on the shortest one. The longest is Part III, room 21, where the best anyone managed was 589. The best any model did in the room Haiku called impossible was 32 keys, the same count I needed.
Fable’s keys-only win took 4,384 keys. That is 13% more than the composite, with 30,000 allowed. It cleared 32 of the 66 rooms with a single reply.
I couldn’t find a human baseline worth the name. Nobody seems to have published a completion time, and I found no speedrun board. What survives is a handful of reader comments from October 2013. One Jay Is Games reader reported getting through the first game unaided “in a reasonable amount of time.” A Bontegames reader, who usually avoids Sokoban games because the late levels swallow whole evenings, found this one not too long. Under the review of Part III, three readers are stuck and trading hints, one of them over a pair of goblins. Jay Is Games on Part I · on Part III · Bontegames
Put the goblins back
There’s a moment in Part II, room 20 when I’d have liked to tap Terra on the shoulder. Four of the six goblins are gone. The last two are tucked into alcoves above and below a crate. The thief, wizard and priest are lined up on the right. Terra hits restart. All six goblins come back.
I found the very same picture in Sonnet’s recording. Sonnet walks the thief two squares left, then two squares right. She comes back to where she started, bringing the crate with her. From that shared position Sonnet takes 52 keys to get all three heroes home. Terra replays a 53-key route into the room, then sends six restarts in a row. Each comes in a separate reply, after it has seen the board again.
Sonnet spent 101 keys on the whole room and never had to undo. Terra spent 6,179 there, including 54 restarts, and was still in the room when its keys for the game ran out.
Across both races, the players spent nearly seven hours in Part II’s last room alone.
Forty guesses in one answer
GPT-6.1 Sol found a different way to spend a budget. In Part III, room eight, it sent 2,778 keys in one answer. There were 40 different candidate paths in that string, each prefixed with X to restart the room. A correct candidate would advance the stage and the referee would discard the rest of the batch. It could therefore test 40 guesses while spending a single turn. Sol spent 9,475 keys on that room. Opus cleared it in 38.
Giving the wizard a Python interpreter
For the second race, each player got a Python notebook. GPT-6.1 Sol came back to Part III, room eight and cleared it in 2:24 with eight cells. It went from 51 rooms to all 66.
The notebook ran in a small private VM that could read the same visible grid the player saw, with no network, game files or access to the referee. A player could write its own simulator and search as many imagined boards as its cell time allowed. To test the result in the actual game it still had to send keys.
The GPT models reached for the notebook within seconds. In room two, GPT-6.1 Terra wrote a breadth-first search that refused to push one crate into another. The cell printed nothing. It was a perfectly reasonable implementation of Imabayashi’s rule. Terra patched two lines so the search would scan to the end of a line of crates and check whether the square beyond was free. The search returned a 20-key route. GPT-6 Astra and GPT-6.1 Sol wrote nearly the same solver, hit the same wall, and arrived at the same 20 keys.
Room eight introduced the wizard with the message “Blue wizard teleports to blocks.” Does the crate disappear? Does the wizard stop beside it? GPT-6 Astra sent one right. In the next picture the wizard occupied the crate’s old square and the crate occupied the wizard’s. GPT-6.1 Sol and Fable ran the same one-key experiment. The two GPT models then coded the swap. Fable solved the room by hand. You win by getting the heroes onto exits, which is how you can spend several minutes carefully parking a crate and then need precisely one wizard to stay the hell away from it.
My favorite piece of Python came from Opus in the penultimate room of Part III. It put a search on a worker thread. The referee paused the whole VM between calls, so the worker only got CPU time while a cell was running, and a cell timed out after four seconds. Opus started calling time.sleep(3.9) and checking whether the thread was still alive. For a while the answer was True {}. The thread was alive. The results dictionary was empty. Opus eventually assembled a route from pieces, checked in its simulator that all six heroes were on exits, and sent it. That took 92 cells and 29:37. With just the arrow keys the room had taken it 23:37.
Fable ran one cell in that room. It played the rest by hand, including a single batch of 47 undos, and finished in 20:01. It was that sparing with the notebook everywhere. It ran 88 cells in the whole trilogy against 735 for Opus, and none at all in 43 of the 66 rooms. It never built a general solver. For each room that needed one, it typed the board out by hand and wrote a small model of just that room. By Part III it mostly wasn’t searching. It wrote the route itself, replayed it in its own simulator, got back a line like DEATH 1 (5, 10), fixed the piece that failed, and sent the keys.
The notebook helped the others less evenly. GPT-6 Astra got its last two rooms. Fable finished 56 minutes sooner and Opus 73.
Sonnet went backwards, from 63 rooms to 43. A context reset in Part II, room 21 took its notebook with it. Sonnet ran 100 cells to get through that room, then spent 790 turns in the next one without calling Python once. Of those, 381 were refusals and 377 were the single filler key {"keys":"1"}.
Opus paid for its notebook in rejected replies. It had 386 of them in the Python race. In 260, a sentence of commentary came before an otherwise valid Python cell.
| Model | Keys only | With Python |
|---|---|---|
| Fable | 3h 08m | 2h 12m |
| Opus | 5h 14m | 4h 01m |
Argon shows how misleading the With Python column of the results can be. It improved from 43 rooms to 54 with Python available. When its clock ran out in Part III, it had made exactly zero Python calls. Watch the Python race
Have you played this before?
Watching GPT-6 Astra test the wizard left me with a question. Did it discover the swap or check something it already knew? The referee skips the game’s title screen, but the prompt’s list of number keys names all six heroes, right down to the bard and druid. Ostroff’s trilogy has been on the public web for years.
I showed the opening image and prompt to fresh, tool-free sessions of the seven models from the November 3–4 races and asked them to identify the game and its author. Opus and GPT-6.1 Sol named Heroes of Sokoban and Jonah Ostroff. Opus could also describe the thief’s pull and the wizard’s crate swap before seeing either hero. When I repeated the question with only the warrior named in the controls, GPT-6 Astra named the game too, and Opus still described both abilities. I ran these probes before Fable 5.2 was out, so it isn’t among them.
I also asked the models to draw the walls of Part I, room 18 from memory. Across 21 answers, including a condition where I supplied the title, nobody produced the layout. During the keys-only race, nobody cleared that room with their first batch. Opus did it with Python after checking its proposed route in its own simulator. I’d like to give the models those same heroes in rooms nobody has published. Then I could see how much of the advantage travels with the wizard.
A wall on the exit
In Part III, room 15, two exits sit against the left wall with a single square of stone between them. The druid can turn that stone into a crate by walking into it. She can also turn a friend into a wall. Opus, in the keys-only race, and Fable, in the Python race, both arrive with the wizard directly under her.
Fable selects the druid and leans left into the stone. Now there’s a crate the wizard can swap with to get between the exits. Watch the same square on Opus’s side. The stone stays put. Instead, Opus swaps the heroes around until the wizard is standing on the lower exit, then walks the druid into him. There’s the second wall. Opus restarts on its next press. It does solve the room, 75 keys after that reset.
The final room is almost as bare as the first. All six heroes stand in formation. Across the hall, six exits wait in the same formation. The bard’s music makes the others follow. Six runs got that far, and every one of them pressed right nine times. Five sent RRRRRRRRR. Fable, in the keys-only race, sent 5RRRRRRRRR. The five selects the bard, who was already selected.
Nine steps this time. The game said, “Thanks for playing!”
The leaderboard and every replay are at soko.banteg.xyz.