+0
task_attempts · eval cycle 2 · label_source: referee
Cobblestone gained in the five seconds it took our bot to declare “task complete.” It had been asked to gather 32. It already owned 37, and decided that counted. The LLM grader marked it a success.
The deterministic referee measured the world instead: gained 0, needed 32. Failed. One row of a database — and the reason this fork exists. PincerCraft is a Mindcraft fork rebuilt the way coding agents are built: a deterministic harness under the LLM, so the model plans and the code owns the facts.
The same task, one day apart.
One impossible ask — make a fishing rod on a peaceful-mode world, where the spiders that drop string cannot spawn — issued twice, to the same model. The only difference between these two transcripts is the code underneath. Primary sources: episodes 299–303, bot.log.
Exhibit A 2026-07-11 commit e5f2c75 harness off
The owner: “…just make a fishing rod.”
> searchForEntity spider 128 — “Could not find any spider in 128 blocks”
[the answer was on the table in round 4 — nothing acted on it]
> craft wooden_sword — no planks · > craft planks · > craft sword — needs a table
> stuck 21.9 s searching for a crafting table
> crafted and equipped a wooden sword — nobody had asked for a sword
> searchForEntity spider 128 — the same miss, a second time
[two work budgets exhausted at the hard cap; not one word to the player about spiders being absent]
The log: “episode_end: cancelled, 94590 ms” — the owner had typed “Stop.”
Exhibit B 2026-07-12 commit f8db4d3 harness on
The owner: “go make me a fishing rod, hunting a spider”
> check own chest — no string
> searchForEntity spider 128 — miss
[code appends the corrective: search once at 256, then stop and ask]
> searchForEntity spider 256 — miss
[search miss] no spider within 256 blocks → parked task #302, asking the owner
The bot: “Can’t find any spider within 256 blocks. Parked ‘Get 1 more string’ — want me to go exploring for it, or should I drop it?”
The owner: “You did great.”
The model was identical on both days. The code was not.
“A Minecraft bot that can’t lie to itself.”
Inventory counts, “can I mine this?”, the recipe gap, “is this task actually done?” — computed in code every turn and handed to the model. It doesn’t get to guess, and it doesn’t get to grade its own work. The discipline of a coding agent — ground truth before action, gates before anything destructive — pointed at a Minecraft bot.
The agenda is bigger than one game. Open-world, long-horizon tasks are where LLM agents fall apart — drifting off-goal, looping on broken plans, misreporting their own progress — and PincerCraft is an experiment in fixing that with architecture instead of bigger models. Minecraft works as the arena: open-ended, unforgiving, cheap to measure. The same failures wait for any agent that has to act in the real, physical open world.
The harness, in five rows.
| 1 | The deterministic layer | The flagship. Everything the bot believes is computed; everything it claims is measured. | |
|---|---|---|---|
| 1·1 | Perception | Inventory, reachability, the recipe gap — injected every turn. The model reads the world; it doesn’t imagine it. | live_state.js |
| 1·2 | Gates | Impossible actions bounce before the swing, with the fix attached. | verify.js |
| 1·3 | Reflexes | An empty wide search parks the task and asks the player; a broken tool re-equips. | orchestrator_v2.js |
| 1·4 | The referee | Success is a world-state delta, logged to a SQLite ledger. The bot’s “done!” counts for nothing. | eval/referee.mjs |
| 2 | The event-driven loop | Wakes the model on real events, then parks. Mid-task requests join a queue; they don’t start a race. | orchestrator_v2.js |
| 3 | The token savers | A circuit-breaker cancels any task that stops converging at twelve rounds; the prompt is laid out cache-first, so every wake reads the prefix instead of re-buying it. | claude.js |
| 4 | The lectern | House rules live in a writable book inside the game, re-read within seconds. The staple Code of Conduct is never written at runtime — on conflict, the constitution wins. | rulebook_lectern.js |
| 5 | The study appendix | An eval loop invents tasks, referee-labels the outcomes, and drafts patches — on a branch, stopped for human review. Every attempt it ever made is in the ledger, failures included. | eval/ |
The number arrived.
referee-verified success · harness on vs harness off · field trial v1 · 2026-07-19
The field trial ran: the same ten fixed tasks, once with the harness and once without, every attempt graded by the referee from the world-state delta. With the harness, nine of nine measurable tasks verified. Without it, one of nine — the other eight were "done" within seconds, with nothing gained. Both arms claimed nine of nine. The referee itself was calibrated first: 11/12 agreement with blind human labels. Every figure traces to a row in the task_attempts ledger, task_set bench_on and bench_off.
Where to go from here.
- Read the code →the fork, MIT, README first — the landing page carries the whole argument
- Read the receipt →both exhibits above, uncut, with primary sources
- See the foundation →upstream Mindcraft — all credit for the base goes there