Methodology · FarmerBench

Why this game

The Farmer Was Replaced is a programming game. A drone farms the field, and the only way to control it is to write programs for it in the game’s Python-like language. That makes it a test of what agents are increasingly asked to do: write code that has to work inside a running system, watch what it does, and fix it.

Every action goes through code. The agent never plants or harvests directly. It writes a program, runs it and reads the result, so the programs themselves are the evidence of its strategy. Opus rewrote one program 26 times on seed 1.
The language and the manual unlock as it plays. Loops, variables, functions, lists and dictionaries are upgrades bought with resources; loops cost 5 hay. The in-game documentation opens with them, so the agent learns a toolset it doesn’t yet have, mid-game. Knowing the game in advance can mislead: Grok said it knew the game, then guessed the field “may only reach 10x10”. It goes to 32×32.
It is a long game where decisions compound. Six game-hours take between 58 and 366 tool calls and up to 70 minutes of real time. What to unlock first, when to grow the field and whether to spend gold on drones all shape what is possible hours later.
Time is a resource the agent manages. Each action costs 10 game-seconds while the farm keeps running, so the agent decides when to check in and when to let a program run. Agents differ: median program stints run from under three minutes to over ten.
There is no single right answer and no ceiling. Every tenfold increase in any of eleven resources adds a point. On seed 2, Opus and Sol built quite different farms and finished 0.34 points apart. In the October 2026 campaigns, model means ranged from about 16 to 76.
The late game is real engineering. Solving mazes, sorting cacti in place, steering a snake for bones, planting crops beside their companions and splitting work across up to 32 drones are algorithmic and concurrency problems, and they separate the top of the table from the rest.
It can be checked and watched. The game is seeded and deterministic, so every run replays exactly: researchers can audit each tool call, and clips are genuine footage of the run.

What it doesn’t test: the agent reads text, not pixels; the game is a single-player toy domain with a deliberately limited language and no real-world tools; and it is a published game with community guides, so prior exposure is possible.

What is measured

The subject is a model plus its first-party agent: Claude in Claude Code, GPT in Codex, Grok in Cursor (owned by xAI), open models in OpenCode. We name it that way on purpose. A result says how well that pairing played, not how well the model would play in any harness.

Each agent is launched with its built-in tools disabled (no shell, files or web) and only the FarmerBench MCP server attached: over the Agent Client Protocol for Claude Code and Codex, and through Cursor’s own headless mode for Cursor, whose Agent Client Protocol mode doesn’t report token usage. Every agent sees the same tools, the same task prompt and the same continuation message. There is no per-provider prompt tuning. A planned ablation runs one model through both its own agent and OpenCode to measure how much the agent matters.

The clock

Game time is the only scored clock. The game is paused while the agent thinks. Every tool call costs a fixed 10 game-seconds, during which the current program keeps running. run_program and wait add the game time they ask for. This gives a human cadence (write a program, let it run, check back) without letting provider latency into the score: the clock counts actions, not seconds of thinking.

The budget is 6 game-hours from a fresh save. When the agent ends its turn early, the runner sends a fixed message (“Continue playing. N of game time remain.”), up to 20 times. A run ends on the game-time budget, the tool-call cap (2,000) or the continuation cap, all of which are valid endings. The real-time cap (4 hours), an agent or harness error, or an operator stop ends it as invalid.

The budget and action cost were set from one calibration run and frozen on 16 October 2026. Every run records the task file’s SHA-256, and changing anything in it makes a new task version.

Tools

Tool	Advances game time
`observe`, `inspect_tile`	Only the 10-second action cost
`docs_list`, `docs_read`, `docs_search`	Only the action cost
`research`, `buy_upgrade`	Only the action cost
`list_programs`, `write_program`, `stop_program`	Only the action cost
`run_program(name, seconds)`	Up to `seconds`, or until the program ends or errors
`wait(seconds, until…)`	Up to `seconds`, or the exact moment a resource, affordability or program-end condition holds

Every result reports game time used and remaining. Documentation is filtered through the game’s own unlock rules, and there is one player-visible way to observe the farm, with no omniscient view.

Scoring

The agent is told to build the largest and most diverse farm economy it can, and that its result is the total amount of each resource the farm produces. The headline rubric, economy-v1, measures exactly that:

score = Σ over resources r of log10(1 + produced_r)

produced_r is the cumulative amount of resource r the run’s own farm produced within the budget. Spending does not reduce it. Each tenfold increase in any resource is worth one point, so 1,000 hay is 3 points and 1,000,000 hay is 6. Unlocking a new production chain opens a new term. There are no exchange rates to tune and no ceiling. The resources counted are hay, wood, carrot, pumpkin, cactus, power, bone, weird substance, gold, water and fertilizer.

An alternate view, economy-weighted-v1, is published for sensitivity checks: log10(1 + Σ w_r × produced_r), with weights from the game’s late-game sink prices (hay, carrot and cactus 1; wood 0.1; gold 10; bone 5). Unpriced resources are ignored and listed as such.

Runs record; rubrics score. A run never contains a score. A rubric is a pure, versioned function of the run record, so everything can be rescored at any time. Every score file records the rubric’s source and SHA-256. Changing the headline rubric is a logged decision that rescores everything.

Score over time is the same rubric applied to every 10-game-second telemetry sample. Upgrade timelines and milestones are reported alongside, never folded into the score.

Valid runs

A run counts in a model’s mean only if it:

ended through its own play (budget, continuation cap or tool-call cap),
was not superseded by a retry,
has telemetry and a final state, and
passes the conservation check: start + produced − final ≥ 0 for every resource.

Invalid runs are still scored and listed with every reason. Runs with command-line task overrides can be valid but are marked non-standard and are not comparable with the frozen task.

Determinism and replay

World generation is seeded, and the simulation advances in fixed ticks only when asked, as fast as the host allows. The same seed and program produce identical telemetry at any speed, so variance between runs comes from the model, not the game. Every entrant plays the same fixed seeds.

Because of that, a run can be replayed exactly: re-run the seed and re-issue every recorded tool call through the same MCP server. The replay compares each result with what the agent saw and the final state hash with the recorded one. Strategy clips on this site come from the same mechanism. An exact clip is genuine footage of the run, regenerated. A fresh clip rebuilds a moment’s upgrades, inventory and programs on a new field when the original moment doesn’t show the strategy well, and it is labelled as a reconstruction.

What is and isn’t comparable

Comparable

Scores from standard runs of the same task version (same task SHA-256) under the same headline rubric, across entrants and seeds.
Means over valid runs on the same seed set, reported with min and max. A model with fewer valid seeds than planned is marked provisional: its mean can still move. With three seeds per model, small gaps are not meaningful, and we make no significance claims.

Not comparable, or only with care

Different task versions. Only runs recorded against the frozen task file (matched by its SHA-256) and with no overrides reach the leaderboard. The calibration run used a pre-freeze task file, and harness tests use shortened budgets. Both are listed separately, below the leaderboard.
Model alone. A result belongs to a model in its agent. Differences include the agent’s own loop, context management and system prompt.
Tool-call and turn counts across agents. Codex can script tool calls, issuing several from one step of code it writes, so its call and turn counts don’t measure the same thing as another agent’s. Every call still costs the same 10 game-seconds, so the game clock stays fair, but the counts are not like-for-like.
Reasoning tokens across vendors. Reasoning visibility differs by vendor. Claude’s thinking tokens are read from the agent’s session log where it exists. Codex folds reasoning into output tokens, and Cursor reports none separately. A missing thinking count means “not reported”, not zero, and thinking counts should not be compared across vendors.
Cost. Cost is an API-equivalent estimate: recorded tokens priced with a versioned list-price table. Agents run on their vendors’ own logins, so it is not an actual charge. Where the agent reports its own cost, both are shown. Cursor reports tokens only as a total per turn, with no per-request breakdown, so Grok’s token and cost figures can’t be checked request by request and may include Cursor’s own background calls.
Real time. Wall-clock time depends on provider latency and load. It is reported, never scored.

Efficiency

Next to the score, each run reports tokens by category (input, cache reads, cache writes, output and, where available, thinking), the cost estimate and its price table, tool calls by tool, turns and continuations, real time, and context size over game time. Deliberation shows up here, not on the clock.

Data behind this site

Every page is built from two generated folders and nothing else. exports/ holds a deterministic bundle per run: conditions and outcome, a column-oriented time series every 10 game-seconds, and evidence items with what the agent saw, what it wrote, and what only the referee saw. scores/ holds one score file per run and rubric. Run records themselves stay offline.

Evidence carries a visibility: model visible (tool results the agent saw), model authored (its code), referee only (telemetry the agent did not see directly) or derived. Pages say which is which, for example that a program’s runtime error came from telemetry rather than from what the agent was shown.