leaderboard/haiku-claude-mediumrank 7

Claude Haiku 5.5

Anthropic · in Claude Code · medium reasoning effort

All 3 seeds

Read the docs, missed the one line that mattered

Haiku looked things up more often than any other Claude model, but on every seed it stopped one step short of the chain that opens the late game. It never used fertilizer, so it never made weird substance, never ran a maze, never earned gold and never bought a second drone. Its farms grew crops, pumpkins and cactus on a single drone and finished fifth on every seed, between 34 and 37 points, at the lowest cost of any Claude model.

36.23 mean34.38 / 37.24 / 37.08 by seed$0.81 a runRead the write-up

Seed 237.2Seed 337.1Mean36.2Seed 134.4
at-a-glanceeditorial, from the write-up

At a glance

What worked

  • Kept a row of cactus sorted with swaps on seeds 1 and 3, so a single harvest took the whole row. [1][2]
  • Read the scoring correctly, noting that production rather than stock is what counts. [1]
  • The cheapest Claude model to run, at $0.61 to $1.11 a seed.

Where it struggled

  • No program on any seed used fertilizer, so weird substance, mazes, gold and extra drones never came; it farmed with one drone all game. [1][2]
  • On seed 2 it read the page that explains where weird substance comes from, and still never used fertilizer. [1]
  • A slow start, with wood arriving ten to twenty minutes later than for the leading models. [1][2]
write-up.mdeditorial · across the seeds

A slow start

Haiku’s openings were slow even by this field’s standards. On seed 1 its third hay program ran for almost eight minutes for a single hay, and wood didn’t arrive until 21 minutes in. On seeds 2 and 3 its first bush programs harvested before anything had grown, the same slip Sol and Astra made, but it took ten to fifteen minutes to sort out rather than five. By about 35 minutes in, it had a working mixed farm on every seed, a little behind the rest.

The step it never took

The game’s late economy hangs on a chain. Plants grown with fertilizer come out “infected”, and half of an infected plant’s harvest is weird substance. Weird substance makes mazes, mazes pay out gold, and gold buys more drones. Every model above Haiku went through that chain, and Opus was doing it within the first hour. Haiku never started it: no program it wrote, on any seed, used fertilizer.

It wasn’t for lack of looking. On seed 1 at 1h45m it searched the docs for weird substance, then for gold, and read the Fertilizer and Mazes pages. But the short Fertilizer page only says “Grow plants instantly”, and the search results showed what weird substance is spent on, not where it comes from. Its end-of-run summary put it plainly:

… Gold and weird substance; I couldn't find how either is produced in the docs.
Claude Haiku 5.5 said, seed 1 at 6h 00m · in the log

Seed 2 is the odd one. At 1h32m Haiku read the full Fertilizer tutorial, which goes on to explain exactly this: fertilised plants are infected, and half their yield becomes weird substance. Nothing changed afterwards; fertilizer kept piling up unused, and it finished the seed holding over 5,000. Seed 3 went like seed 1, with the short page and the same searches.

Without gold there was never a second drone. Haiku farmed every seed with one, while Sonnet, in the same agent, had four to eight.

Sorted cactus, one row at a time

What Haiku did build, it built carefully. On seeds 1 and 3 it kept a row of cactus sorted by size with neighbour swaps, so harvesting one cactus takes the whole row:

farm12.pyseed 3 · r4 · written at 2h 50m · ran · lines 39–45 of 87in the run
def sort_pass():
    while get_pos_x() < 11:
        if measure(East) < measure():
            swap(East)
        move(East)
    move(East)

A sorting pass along the cactus row: swap with the neighbour to the east whenever it is smaller.

… That changed cactus from about 40 per minute to roughly 400 per minute. …
Claude Haiku 5.5 said, seed 3 at 6h 00m · in the log

That rate is Haiku’s own estimate, and its seed 1 summary calls cactus its largest output, which the record doesn’t bear out (wood ran to 760,000 against 119,000 cactus). The sort clearly mattered, though: about 100,000 cactus on seeds 1 and 3, against 320 on seed 2, where it never built one.

What decided the score

With one drone, the last hours of every seed were a single farm program left to run: 55 unchanged minutes on seed 1, more than two hours on seed 3, each stretch adding under three points. Hay, wood, carrots and pumpkins reached about 100,000 to 760,000, five or six points each, and the three resources at the end of the chain scored nothing. Haiku did read the rules right in one respect that Sonnet didn’t, noting at the end of seed 2:

… Stock is not the score. Production is what counts …
Claude Haiku 5.5 said, seed 2 at 5h 58m · in the log

It finished fifth on every seed, 10 to 30 points ahead of Luna and 12 to 24 behind Sonnet, at $0.61 to $1.11 a run.

The numbers every valid seed, from the published runs

pointslog₁₀(1 + produced)

Where the points came from

ResourceSeed 1Seed 2Seed 3Mean
Hay5.545.645.275.48
Wood5.885.595.345.61
Carrot5.075.595.385.35
Pumpkin4.975.615.505.36
Cactus5.072.514.984.19
Power3.564.262.853.56
Bone————
Weird substance————
Gold————
Water4.284.334.394.33
Fertilizer—3.713.372.36
Total34.3837.2437.0836.23
unlocksfirst produced, game time

When each resource was first produced

ResourceSeed 1Seed 2Seed 3
Hay1m 20s1m 30s1m 40s
Wood21m 00s10m 30s11m 30s
Carrot29m 10s27m 50s27m 40s
Pumpkin51m 20s49m 30s59m 10s
Cactus2h 03m1h 41m1h 56m
Power1h 04m46m 10s51m 20s
Water36m 30s31m 30s25m 30s
Fertilizernever1h 32m1h 46m
Weird substancenevernevernever
Goldnevernevernever
Bonenevernevernever
how-it-playsper seed, from the export's behaviour block

How it plays

MeasureSeed 1Seed 2Seed 3MeanAll models
Resources openedscored resources produced at least once7 of 118 of 118 of 117.7 of 119.8 of 11
Studydocs and research reads4145554741
Action overheadgame time spent looking, reading and typing (10 s per call)11%17%13%14%9%
Tool callsevery action the agent took234366290297205
Program writesaccepted saves of a program5056424934
Program stintstimes a program was started and left running4058384534
Median stinthow long a program usually ran before the next change5m 20s2m 00s2m 33s3m 18s9m 25s
Cost per pointAPI-equivalent estimate ÷ score$0.018$0.030$0.019$0.022$0.074
runs3 valid