leaderboard/gpt-6.1-sol-codex-mediumrank 2

GPT-6.1 Sol

OpenAI · in Codex · medium reasoning effort

All 3 seeds

One mixed farm, then sixteen drones to work it

On every seed Sol built the same engine: one program farming grass, trees, carrots and sunflowers in repeating columns, scaled up to a 32×32 field and sixteen drones. Gold, cactus and bones were side trips. It stumbled at full size, where some older programs stopped producing, but still finished top on every seed at about a quarter of Astra's cost.

74.08 mean71.82 / 75.87 / 74.56 by seed$0.47 a runRead the write-up

at-a-glanceeditorial, from the write-up

At a glance

What worked

  • One mixed-crop program carried the whole economy, and grew from one drone to sixteen as the farm expanded. [1][2]
  • When the dinosaur program nearly stopped paying on the full-size field, it shrank the field for dinosaurs and got bones flowing again on seeds 2 and 3. [1][2][3]
  • When its cactus sorter crashed for lack of pumpkins, it farmed pumpkins and re-ran the same program instead of rewriting a working one. [1][2][3]
  • Cheap to run, at $0.32–0.65 a seed.

Where it struggled

  • Some programs written while the farm was smaller stopped paying at 32×32. On seed 1 the last 23 game-minutes of dinosaur farming returned about 18,000 bones, just after it had paid about 3 million cactus for dinosaur upgrades. [1][2][3][4][5]
  • The rarer resources stayed far behind the crops, with a few hundred thousand gold produced against hundreds of millions or billions of wood.
  • Made the same beginner's mistake with its first wood program on every seed, losing about five minutes each time. [1][2]
write-up.mdeditorial · across the seeds

Five minutes of nothing

Every seed starts the same way, and so did Sol. It read up on bushes, wrote a four-line program to grow wood, and ran it for three minutes.

wood.pyseed 1 · r1 · written at 3m 05s · ranin the run
while True:
 harvest()
 plant(Entities.Bush)
 move(North)

The program harvests each bush the moment it plants it, long before anything has grown, so three minutes of running produced no wood at all. Sol’s thinking found the problem straight away:

… I’m thinking about how harvesting immature plants can be damaging. …
GPT-6.1 Sol thought, seed 1 at 6m 25s · in the log · thinking as the agent reported it

The fix bounced, though: the game won’t let you edit a program while it runs, so the new version was rejected, and Sol ran the old one for two more fruitless minutes before writing a second program with a different name. The same slip, in the same order, happened on seed 2 and seed 3. GPT-6 Astra wrote the identical four lines on its seeds too. Five minutes out of 360 doesn’t matter much; it’s a nice reminder that a seed fixes the game’s randomness, not the model’s.

One farm, in columns

From about fifteen minutes in, the rest of the run is built around one idea. Instead of a field of one crop, Sol plants the farm in repeating columns: grass left to grow on its own, trees spaced out in a checkerboard (the game rewards trees with room around them), carrots, and later a column of sunflowers for power. Fertilizer on the trees yields weird substance as a by-product. A single program tends all of it, so every pass of the drone feeds five or six resources at once.

On seed 3 that program is multi_econ, written once at 1h38m. Instead of hard-coding the farm, it asks the game how many drones it has and how big the field is, gives each drone its own columns, and sets them off:

multi_econ.pyseed 3 · r1 · written at 1h 38m · ranin the run
clear()
def worker():
	start = get_pos_x()
	while True:
		x = get_pos_x()
		y = get_pos_y()
		if can_harvest():
			harvest()
		if x % 4 == 3:
			if get_ground_type() != Grounds.Soil:
				till()
			plant(Entities.Sunflower)
		elif x % 2 == 0:
			if y % 2 == 0:
				plant(Entities.Tree)
				if num_items(Items.Fertilizer) > 0 and not can_harvest():
					use_item(Items.Fertilizer)
		else:
			if get_ground_type() != Grounds.Soil:
				till()
			plant(Entities.Carrot)
		if get_water() < 0.3:
			use_item(Items.Water)
		move(North)
		if get_pos_y() == 0:
			for i in range(max_drones()):
				move(East)
			if get_pos_x() < start:
				while get_pos_x() != start:
					move(East)
for i in range(max_drones()-1):
	spawn_drone(worker)
	move(East)
worker()

Each drone works its own set of columns, stepping across the field by the number of drones; the last lines start one worker per drone.

Because it reads the drone count and field size each time, that one program ran in five stints to the end of the game as the farm went from a few drones to sixteen and the field to 32×32. On seeds 1 and 2 Sol did the same thing by rewriting: economy, then economy4, economy16, economy32 and final_economy, each built for the drones it had at the time (economy32 spawns exactly fifteen workers). Both styles worked. Seed 3’s single program didn’t produce the best score (seed 2 did), so this is a difference in how Sol wrote code, not evidence for which way is better.

These stints added more score than anything else on every seed, and Sol seems to have known it. Near the end of seed 1 it went back to the mixed farm instead of another specialist:

… I’m using the remaining time for the mixed-crop program, which has delivered the strongest sustained production …
GPT-6.1 Sol said, seed 1 at 5h 46m · in the log
Scoreover game time
Seed 374.6

Ticks under the axis: moments in the write-up.

Seed 3's score. The steepest climbs line up with multi_econ's stints; the flatter stretches between them are specialist programs for pumpkins, cactus, gold and bones.
Sixteen drones working the column farm on seed 3. Exact replay of seed 3, 5h 46m–5h 48m, running multi_econ.py · 8× speed · the run

Pumpkins first

The specialist resources mostly went smoothly, with one revealing hiccup. On seed 1 at 1h30m, Sol’s first cactus program sorted the field and then crashed when it tried to compare a cactus with a square that had nothing in it. Cactus is planted with pumpkins, and the farm had run out of pumpkins, so some squares were never planted. Sol read the situation correctly: it bought a pumpkin upgrade, farmed pumpkins for fifteen minutes, and then re-ran the same cactus program unchanged for 41,472 cactus. The same crash happened on seed 2.

The reasoning summary from that moment is a small curiosity of its own. Codex shows a summary of the model’s thinking, not the thinking itself, and here the summary wandered off the farm entirely:

… So, I should probably check local stores or farms that sell them. …
GPT-6.1 Sol thought, seed 1 at 1h 31m · in the log · thinking as the agent reported it

What Sol actually did next was farm pumpkins in the game, so whatever it was thinking, the summary didn’t capture it.

Sixteen drones

In the last two hours of every seed, Sol had the full 32×32 farm and sixteen drones, the most the game allows. That is where the economy took off: a single ten-minute stint of economy32 on seed 1 produced 235 million wood. Astra also reached sixteen drones on seed 1, but stopped at eight on seeds 2 and 3, and the difference shows most on seed 2:

Score on seed 2over game time
GPT-6.1 Sol75.9GPT-6 Astra63.2
Score on seed 2. Sol's late climb comes from economy16 and final_economy running on sixteen drones; Astra worked the same seed with eight.

When the field got big

The farm’s growth had a cost. Some programs written while the field was smaller kept running when it reached 32×32, and stopped paying.

Dinosaurs are the clearest case. Bones are paid out when the dinosaur hat comes off, and the bones program walks the dinosaur in a loop around the whole field, only taking the hat off once the dinosaur can no longer move. On a smaller field that happened often enough: on seed 1 it produced 81,800 bones in a fifteen-minute stint at 2h18m. On 1,024 squares, as far as the code and the inventory show, it never happened within a stint, so each tail was paid only when a later program changed the hat or cleared the field.

On seed 1, Sol paid about three million cactus for the top dinosaur upgrades at 5h12m, then ran bones for fifteen minutes and eight more. Neither stint showed a bone while it ran; the tails were cashed in by the programs that followed, for 17,424 bones and 1,024. In between it tried shrinking the field, which didn’t take, and its thinking worked out why too late:

… I think a change in world size might require a clear reset. …
GPT-6.1 Sol thought, seed 1 at 5h 46m · in the log · thinking as the agent reported it

With thirteen minutes left, it went back to the mixed farm. On seed 2 the same thing happened at 4h35m: ten minutes of dinosaur farming, cashed in for 8,100 bones when the next program cleared the field (and the cactus program stalled too), and this time Sol wrote a new version that shrinks the field to 16×16 and clears it first:

bones16.pyseed 2 · r1 · written at 5h 17m · ran · lines 1–6 of 29in the run
set_world_size(16)
clear()
while num_items(Items.Cactus) > 0:
	change_hat(Hats.Dinosaur_Hat)
	alive = True
	while alive:

The first lines of bones16: a smaller field, cleared, before the dinosaur loop.

That brought in over a million bones in ten minutes. On seed 3 Sol wrote bones16 straight away when it needed bones on the full field, for 4.2 million. Astra’s late bones stints look just as empty while they run, but its tails were longer when the hat came off: 258,064 bones on seed 3.

bones16 on seed 3: the dinosaur on a 16×16 field. Exact replay of seed 3, 5h 38m–5h 40m, running bones16.py · 8× speed · the run

What decided the score

The score adds up one term per resource, roughly one point for every tenfold of production, so a few hundred million of something is worth about eight or nine points and a few hundred thousand about five or six. Sol’s column farm put hay, wood, carrots and weird substance into the hundreds of millions or billions on every seed, and those four alone account for around 33–35 of its 72–76 points. That is also where it pulled away from Astra: on seed 2, wood is worth 9.3 points to Sol and 6.9 to Astra.

The rarer resources were never close. Sol produced between 230,000 and 650,000 gold depending on the seed, power and fertilizer stayed small, and bones swung from about 100,000 on seed 1 to 5 million on seed 3, depending largely on how the dinosaur farming went in the last hour. Sol’s choices there decided the gap between its own seeds; its column farm decided the gap to everyone else.

The numbers every valid seed, from the published runs

pointslog₁₀(1 + produced)

Where the points came from

ResourceSeed 1Seed 2Seed 3Mean
Hay8.388.918.258.51
Wood8.899.338.939.05
Carrot8.308.578.138.33
Pumpkin6.957.167.427.18
Cactus7.256.887.717.28
Power4.885.155.185.07
Bone5.006.246.715.98
Weird substance7.558.427.617.86
Gold5.475.815.375.55
Water5.215.315.145.22
Fertilizer3.944.094.114.04
Total71.8275.8774.5674.08
unlocksfirst produced, game time

When each resource was first produced

ResourceSeed 1Seed 2Seed 3
Hay1m 00s1m 00s1m 00s
Wood9m 20s9m 00s8m 50s
Carrot15m 00s14m 50s20m 40s
Pumpkin53m 30s43m 00s54m 10s
Cactus1h 52m1h 41m1h 06m
Power2h 55m1h 19m1h 38m
Water25m 10s20m 10s20m 30s
Fertilizer36m 10s30m 40s42m 30s
Weird substance1h 04m53m 10s1h 16m
Gold1h 20m1h 25m1h 26m
Bone2h 22m2h 55m4h 11m
how-it-playsper seed, from the export's behaviour block

How it plays

MeasureSeed 1Seed 2Seed 3MeanAll models
Resources openedscored resources produced at least once11 of 1111 of 1111 of 1111 of 119.8 of 11
Studydocs and research reads2525252541
Action overheadgame time spent looking, reading and typing (10 s per call)6%7%6%6%9%
Tool callsevery action the agent took126145132134205
Program writesaccepted saves of a program2225202234
Program stintstimes a program was started and left running3036313234
Median stinthow long a program usually ran before the next change10m 45s10m 30s10m 40s10m 38s9m 25s
Cost per pointAPI-equivalent estimate ÷ score$0.0060$0.0085$0.0043$0.0063$0.074
runs3 valid