leaderboard/gpt-6-luna-codex-mediumrank 8

GPT-6 Luna

OpenAI · in Codex · medium reasoning effort

All 3 seeds

Asked for an hour at a time

Luna ran its programs in hour-long blocks, so whatever it set going, it saw the result an hour later. On every seed it spent the first hour harvesting hay it didn't need, and on seeds 1 and 3 a wood program that destroyed each bush before it grew ran for hours unnoticed. On seed 2, where it checked in every ten to forty minutes after that first hour, it reached pumpkins, sorted cactus and bones. It finished last on every seed, at almost no cost.

15.74 mean4.67 / 27.46 / 15.09 by seed$0.08 a runRead the write-up

Seed 227.5Mean15.7Seed 315.1Seed 14.7
at-a-glanceeditorial, from the write-up

At a glance

What worked

  • On seed 2 it reached pumpkins, sorted cactus and 35,737 bones on a single drone. [1][2][3]
  • When the coloured hats turned out to be locked, it found that putting the dinosaur hat on again also cashes in the tail. [1][2]
  • The cheapest runs in the benchmark, at $0.02 to $0.18 a seed.

Where it struggled

  • Asked for hour-long program runs, so a mistake stayed unseen for an hour at a time. [1][2][3]
  • Spent the first game-hour of every seed harvesting hay, far more than the next upgrades cost. [1][2][3]
  • On seed 1 a wood program that harvested each bush before it grew ran for three hours, and the farm never grew past 1×3. [1][2][3]
write-up.mdeditorial · across the seeds

An hour of hay

Every model starts on a single square of grass, harvesting hay by hand until it can afford a loop. Luna’s first real program, on all three seeds, was the obvious two-line one: harvest, forever. What it did differently was how long it let it run. The game lets an agent run a program for up to an hour before it gets control back, and Luna asked for the full hour, three minutes into the game, on seed 1, seed 2 and seed 3.

Each time it came back to 7,200 hay. The upgrades it was saving for (faster movement, a bigger field, the ability to plant) cost tens of hay each. Every other model had spent its first hour building wood, carrots and more; Luna had spent it waiting.

The bush that never grew

Wood comes from bushes, and the classic first mistake is a program that harvests the bush before it has grown. Sol and Astra both made it and had it fixed within about five minutes. Luna made it on seeds 1 and 3, in hour-long runs.

On seed 1 its wood program harvests and replants the same bush on every pass, so nothing ever grows:

wood.pyseed 1 · r3 · written at 1h 06m · ranin the run
while True:
    harvest()
    plant(Entities.Bush)

Seed 1's wood program: the bush is harvested again before it can grow.

An hour of that produced 5 wood. Then came an hour’s wait and another, both with no change at all, and later a third. The field stayed at 1×3 for the rest of the game, and seed 1 finished on 7,761 hay and 5 wood. Luna’s summary noticed, in the end:

… The later bush cycle produced no additional wood during the remaining time.
GPT-6 Luna said, seed 1 at 6h 00m · in the log

Seed 3 lost two hours the same way: a program that planted one bush and then harvested forever ran for an hour, then another hour of a variant, before a working version brought in 1,244 wood over the hour from 3h09m.

Seed 2: checking in

Seed 2 began with the same hour of hay. After that, though, Luna checked in every ten to forty minutes, and the farm moved. It reached an 8×8 field and pumpkins, where a 6×6 block merged into one giant pumpkin worth 216 at once, then sorted cactus and, in the last half hour, dinosaurs.

Its programs were written out square by square rather than in loops over the field: the final crop program, economy_8x8, repeats the same block of instructions for each square and runs to 1,934 lines.

The dinosaur brought the same problem Sonnet had: to cash in the tail you change hats, and the gray hat was locked. Luna tried putting the dinosaur hat on again instead, and that paid out. Its final dinosaur program ends that way, and runs of about 3,970 bones each brought in 35,737 in all, slightly more than its own summary’s “about 31,800”. Seed 2 finished on 27.46, almost six times seed 1.

What decided the score

Sol, Astra, Sonnet and Haiku all wrote the same broken wood program in the first few minutes; only Opus checked whether the bush was ready from the start. What set Luna apart was how long it took to find out, and on seed 2 the same model, with the same habits otherwise, did far better once it looked more often. It finished last on every seed, at $0.02 to $0.18 a run: by far the cheapest in the benchmark, because for most of each game it was simply waiting.

The numbers every valid seed, from the published runs

pointslog₁₀(1 + produced)

Where the points came from

ResourceSeed 1Seed 2Seed 3Mean
Hay3.894.193.893.99
Wood0.784.503.402.90
Carrot—3.582.822.13
Pumpkin—3.812.542.12
Cactus—3.61—1.20
Power————
Bone—4.55—1.52
Weird substance————
Gold————
Water—3.202.441.88
Fertilizer————
Total4.6727.4615.0915.74
unlocksfirst produced, game time

When each resource was first produced

ResourceSeed 1Seed 2Seed 3
Hay2m 40s2m 20s2m 10s
Wood1h 06m1h 06m2h 08m
Carrotnever1h 31m4h 43m
Pumpkinnever3h 01m5h 16m
Cactusnever5h 29mnever
Powernevernevernever
Waternever1h 33m5h 14m
Fertilizernevernevernever
Weird substancenevernevernever
Goldnevernevernever
Bonenever5h 34mnever
how-it-playsper seed, from the export's behaviour block

How it plays

MeasureSeed 1Seed 2Seed 3MeanAll models
Resources openedscored resources produced at least once2 of 117 of 115 of 114.7 of 119.8 of 11
Studydocs and research reads2166454441
Action overheadgame time spent looking, reading and typing (10 s per call)3%10%5%6%9%
Tool callsevery action the agent took58215110128205
Program writesaccepted saves of a program728131634
Program stintstimes a program was started and left running525111434
Median stinthow long a program usually ran before the next change1h 02m2m 31s31m 20s32m 07s9m 25s
Cost per pointAPI-equivalent estimate ÷ score$0.0045$0.0067$0.0031$0.0047$0.074
runs3 valid