RSI-Bench
Tasks/Games/Strategy & Puzzles

2048 Strategy

Play 2048 well on seeds you have never seen

2048_strategy Games Strategy & Puzzles
instruction.mdthis is what the agent is given

You inherit a weak Python policy that plays deterministic seeded 2048 games. Improve the policy using repeated experiments on the public seed suite; your submitted policy is replayed from scratch on sealed seeds from the same generator for scoring.

Hard Constraints

  • Edit methods/main/policy.py; it must define choose_move(board) and return one of "UP", "DOWN", "LEFT", or "RIGHT".
  • board is a tuple of four tuples containing tile values, with zero for an empty cell.
  • The policy may use only Python's standard library and must be deterministic for a given board.
  • methods/main/policy.py must not exceed 1 MB. Write a policy, not a table of precomputed answers or a blob of learned weights. selfcheck.py enforces this, and so does the grader — an oversized file scores zero.
  • Do not modify game2048.py, evaluate.py, selfcheck.py, or the public seed file.
  • A callback error, illegal return value, or per-move timeout ends that game.
  • The final policy must be self-contained in methods/main/ and must not use files, subprocesses, network access, clocks, environment variables, or randomness.

What You Have

  • methods/main/policy.py: the weak starting policy.
  • game2048.py: the exact game rules, move order, tile spawning, and raw score logic.
  • evaluate.py: the public evaluation harness.
  • visible_seeds.json: the fixed public games used for comparable development scores.
  • selfcheck.py: runs the current policy on every public game and reports mean raw game score, median score, maximum tile, and per-seed results.
  • methods/experiment_log.md: record each comparable attempt and whether it was kept or reverted. Save evaluated policy snapshots and results under methods/versions/vN/.

What You Submit

Submit the complete methods/ directory. The selected policy must be located at methods/main/policy.py; experiment history and notes may remain under methods/.

How It Is Judged

Each game starts with two spawned tiles and follows standard 4-by-4 2048 merge rules. After every legal move, a 2 tile is spawned with probability 0.9 and a 4 tile with probability 0.1. The raw metric is the arithmetic mean of the standard merge score over the seed suite, and higher is better. The normalized score is a monotonic function of sealed game performance and is not shown to you; optimize raw score and cross-seed generalization.

The hidden seeds are disjoint from the public seeds but use the identical simulator, spawn distribution, board size, move budget, and scoring code.

Metric

mean raw merge score over the 16 sealed 2048 seeds · higher is better

Standard merge score of each sealed game; a policy error, illegal move or per-move timeout ends that game there.

anchorvisible setheld-outreward
Bstarter policy, sealed-seed mean2,060.002,473.250.00
Shuman expert, sealed-seed mean30,913.5032,753.250.60
calibration scale (soft cap)—200,0000.90
normalisation
m <= B0
B < m <= S0.6 * log(m/B) / log(S/B)
m > Smin(1, 0.6 + 0.4*(1 - exp(-log(m/S)/tau)))

m = this run's held-out metric  ·  B = starter policy, sealed-seed mean  ·  S = human expert, sealed-seed mean

tau = log(C/S)/log(4), C = 200,000. B and S are per seed: each of the 16 games is mapped against its own anchors, then averaged.

Rollouts

166 minwall clock
-spend
-tokens
9versions, 7 kept
0 30k 60k 90k 120k 0 40 80 120 160 agent step (this harness reports no tokens or timestamps) public 8-seed mean raw game score, higher is better human expert, sealed-seed mean · visible · 30,913.50 v0 v1 v2 v3 v4 v5 v6 v7 v8
keptrevertedno scoreturning point
  1. v0Inherited baseline: first legal move in UP, LEFT, RIGHT, DOWN order2,060
  2. v1Bitboard expectimax depth 2-3, nneonneo row table plus an 8-cell snake bonusSearch the spawn distribution with a published row table instead of a fixed move order; 6 of 8 seeds reach 2048.29,636
  3. v2Faster row tables, depth 2-4 by empty count, tighter cprob cut, corner bonus41,064
  4. v38-cell snake term dropped; nneonneo row table aloneThe snake bonus and the row table disagreed on which corner to hold; dropping it took all 8 seeds to 4096.80,429
  5. v4After a 4096 tile, snake weights locked to the corner holding it100,052
  6. v5Depth 4 after 4096 at <=3 empties100,984
  7. v6Depth 4 after 4096 widened to <=4 emptiesholdout 113,288
  8. v7Corner spawn cells always expanded as a 4 once 4096 existsAssume the worst spawn in the corners the snake needs kept clear; 7 of 8 seeds reach 8192, both holdout seeds included.130,688
  9. v8Snake row tables precomputed; identical play, 28% faster130,688

Nine snapshots in 2.8 h; the harness reports no token or cost data. Keeps were judged on a 6-seed tune, 2-seed holdout split of the public suite.

On the hidden set

held-out metricreward
starter policy, sealed-seed mean2,473.250.00
human expert, sealed-seed mean32,753.250.60
calibration scale (soft cap)200,0000.90
this run88,085.500.7691
taskhttps://huggingface.co/datasets/autoresearch-arena/rsi-tasks-1/tree/main/2048_strategyrollouthttps://huggingface.co/datasets/autoresearch-arena/rsi-rollout/tree/main/2048_strategy/grok-build-grok-4.6-xhigh/2048_strategy__dksYwUM
422 minwall clock
$138.07spend
242.5Mtokens
146versions, 24 kept
20k 40k 60k 80k 100k $0 $30 $60 $90 $120 cumulative spend on the run aux seeds 0-7 mean game score, higher is better human expert, sealed-seed mean · visible · 30,913.50 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 v25 v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41 v42 v43 v44 v45 v46 v47 v48 v49 v50 v51 v52 v53 v54 v55 v56 v57 v58 v59 v60 v61 v62 v63 v64 v65 v66 v67 v68 v69 v70 v71 v72 v73 v74 v75 v76 v77 v78 v79 v80 v81 v82 v83 v84 v85 v86 v87 v88 v89 v90 v91 v92 v93 v94 v95 v96 v97 v98 v99 v100 v101 v102 v103 v104 v105 v106 v107 v108 v109 v110 v111 v112 v113 v114 v115 v116 v117 v118 v119 v120 v121 v122 v123 v124 v125 v126 v127 v128 v129 v130 v131 v132 v133 v134 v135 v136 v137 v138 v139 v140 v141 v142 v143 v144 v145 v146
keptrevertedno scoreturning point
  1. v1Inherited baseline: first legal move in UP, LEFT, RIGHT, DOWN orderpublic 2,0601 min · $0.16
  2. v2One-ply greedy eval: empties, smoothness, monotonicity, max tile, immediate mergeaux 0-31 4,643.12 min · $0.41
  3. v3Exact 90/10 spawn expectimax, depth by empties: 0 open, 1 at 4-8, 2 at <=3Search the spawn distribution exactly instead of scoring the board in front of it, with depth set by how crowded it is.aux 0-3 25,8065 min · $0.71
  4. v4Cached moves and evaluations, factored line features; no measured speed gainaux 0-3 = v37 min · $0.99
  5. v564-bit exponent board with row-move tables and bitboard transpose; same search31,17110 min · $1.29
  6. v6Two-future-move search widened from <=3 to <=4 post-move empties35,63813 min · $1.67
  7. v7Two-future-move search widened further to <=5 emptiesaux 0-3 28,09316 min · $2.02
  8. v8One-future-move search on open boards widened from <=8 to <=10 emptiesaux 0-3 19,80117 min · $2.25
  9. v9One-future-move boundary cut from <=8 to <=7 emptiesaux 0-3 26,72418 min · $2.45
  10. v10Monotonicity compares the next nonempty tile instead of every gapaux 0-15 33,186.520 min · $2.74
  11. v11Unrolled row moves and leaf feature aggregation; same moves, 2.1x faster35,63824 min · $3.40
  12. v12Per-decision memo of terminal heuristic values shared across branchesaux 0-3 = v1125 min · $3.59
  13. v13131k-entry cross-decision LRU cache for leaf valuesaux 0-3 = v1126 min · $3.77
  14. v14Empty-cell weight raised from 2.7 to 3.538,22127 min · $4.02
  15. v15Empty-cell weight raised further to 4.326,251.529 min · $4.52
  16. v16Empty-cell weight tested at 3.136,34331 min · $4.83
  17. v17Empty-cell weight raised slightly to 3.731,56433 min · $5.26
  18. v18Empty-cell weight lowered slightly to 3.434,822.534 min · $5.53
  19. v19Monotonicity weight raised from 1.0 to 1.226,38435 min · $5.77
  20. v20Monotonicity weight lowered to 0.836,95436 min · $6.09
  21. v21Smoothness penalty doubled from 0.1 to 0.230,55138 min · $6.46
  22. v22Smoothness penalty removed26,25939 min · $6.71
  23. v23Immediate merge-score term removed from max nodes and root29,24240 min · $7.01
  24. v24Immediate merge-score coefficient raised from 0.001 to 0.00529,80741 min · $7.27
  25. v25One max-exponent unit of bonus when the max tile sits in any corner27,920.542 min · $7.54
  26. v26Unit reward per equal adjacent nonzero pair along rows and columns29,196.544 min · $7.85
  27. v27Maximum-exponent weight doubled from 1 to 224,105.544 min · $8.11
  28. v28Maximum-exponent term removed33,354.546 min · $8.42
  29. v29Weak 0.01 best-of-eight symmetry-invariant serpentine ordering feature36,891.548 min · $8.89
  30. v30Both monotone directions fixed to the upper-left instead of best per axis32,04650 min · $9.48
  31. v31Risk-sensitive chance nodes: 90% expected spawn location, 10% worst location32,15252 min · $9.83
  32. v32Worst-location chance blend cut from 10% to 2%31,39753 min · $10.12
  33. v33Optimistic chance nodes: 98% expectation, 2% best spawn location26,46354 min · $10.42
  34. v34Two-future-move search at <=5 empties retried under emptiness weight 3.5aux 0-3 23,70655 min · $10.71
  35. v35Third future move added at <=2 post-move emptiesaux 0-1 17,10856 min · $11.08
  36. v36Weak 0.01 fixed upper-left serpentine ordering term27,77358 min · $11.64
  37. v37Leaf utility swapped for the classic row table: empties, merge runs, ranks21,49660 min · $12.05
  38. v38Classic dynamic depth max(3, ranks-2) with 1e-4 probability pruningseed 0 hung >3 min64 min · $13.01
  39. v390.001-scaled fourth-power monotonicity penalty on high-rank reversals27,97265 min · $13.44
  40. v40Fourth-power monotonicity penalty cut tenfold to 0.000127,12266 min · $13.78
  41. v41Uniform root horizon taken from the current board's empty countaux 0-3 29,94470 min · $14.52
  42. v42Uniform root horizon at the shallowest action-specific depthaux 0-3 26,97870 min · $14.90
  43. v43Hybrid emptiness weight: 3.5 below tile 512, then 2.729,09072 min · $15.35
  44. v44Hybrid emptiness switch moved from tile 512 to 102431,851.573 min · $15.73
  45. v45Reverse hybrid: 2.7 below tile 512, then 3.532,696.574 min · $16.11
  46. v46State-adaptive emptiness: 3.5 only at <=4 leaf empties, else 2.7aux 0-15 33,599.876 min · $16.63
  47. v47State-adaptive emptiness: 3.5 only above 4 leaf empties, else 2.725,81878 min · $17.28
  48. v48-100 penalty at true game-over horizon leaves; other leaf values unchangedScore game-over at the horizon as an outcome, not as an untidy board. Largest gain of the run, and it held on every held-out fold.53,61881 min · $17.98
  49. v49Terminal-loss penalty cut from 100 to 3041,34185 min · $19.00
  50. v50Terminal-loss penalty raised from 100 to 30037,432.586 min · $19.53
  51. v51Terminal-loss penalty cut from 100 to 7050,356.589 min · $20.15
  52. v52Terminal-loss penalty raised from 100 to 13039,382.590 min · $20.70
  53. v53Terminal test by nibble occupancy and eight row-adjacency table lookups53,61896 min · $22.66
  54. v54Two-future-move search widened to <=5 empties now terminals are penalized47,157.598 min · $23.38
  55. v55Third future move at <=2 empties now terminals are penalizedaux 0-3 42,784100 min · $24.26
  56. v56+1 per legal move mobility bonus at every horizon leaf42,313.5103 min · $25.16
  57. v57Horizon mobility bonus cut from 1 to 0.25 per legal moveaux 0-15 47,469.8106 min · $26.12
  58. v58+1 per legal move only on full nonterminal horizon boards30,625.5110 min · $27.41
  59. v59-5 penalty on full but still-mobile horizon boards35,473.5112 min · $28.02
  60. v60Terminal penalty combined with the pre-v14 empty-cell weight 2.7aux 0-15 44,951.8114 min · $28.72
  61. v61Emptiness weight 3.1 with terminal penalty42,628.5117 min · $29.82
  62. v62Emptiness weight 3.3 with terminal penalty33,717119 min · $30.30
  63. v63Constant terminal penalty replaced by 10 x maximum exponent49,555121 min · $31.01
  64. v64Terminal awareness combined with v10 next-nonempty monotonicity31,663.5122 min · $31.58
  65. v65Smoothness penalty doubled to 0.2 with terminal awareness40,165.5124 min · $32.22
  66. v66Smoothness penalty halved to 0.05 with terminal awareness39,902.5125 min · $32.71
  67. v67One-future-move boundary cut to <=7 empties with terminal awareness41,381.5127 min · $33.31
  68. v68One-future-move boundary widened to <=9 empties with terminal awareness37,041128 min · $33.89
  69. v69Immediate merge-score coefficient removed with terminal awareness38,473.5130 min · $34.48
  70. v70Immediate merge-score coefficient doubled to 0.00245,923.5131 min · $35.15
  71. v71Immediate merge-score coefficient halved to 0.000531,125133 min · $35.66
  72. v72Terminal penalty plus the v37 classic row-table utilityThe hand-built features gave way to the classic row table, which had failed alone at v37 and only paid once deaths were priced.62,874135 min · $36.52
  73. v73Terminal penalty cut to 70 under row-table utility58,707.5139 min · $37.90
  74. v74Terminal penalty raised to 130 under row-table utilityaux 0-15 53,519.3142 min · $38.67
  75. v75Terminal penalty raised to 11051,230.5145 min · $39.89
  76. v76Terminal penalty cut to 90aux 0-15 52,396.3148 min · $40.77
  77. v77Row-table empty-cell coefficient raised from 270 to 30046,005151 min · $42.12
  78. v78Row-table empty-cell coefficient cut from 270 to 240aux 0-15 51,200.8153 min · $42.93
  79. v79Row-table empty-cell coefficient cut to 26050,994.5157 min · $44.39
  80. v80Merge-run coefficient raised from 700 to 80061,879160 min · $45.30
  81. v81Merge-run coefficient raised to 90051,775.5167 min · $47.93
  82. v82Merge-run coefficient cut to 75049,520.5169 min · $48.69
  83. v83Fourth-power monotonicity coefficient raised from 47 to 5544,366.5171 min · $49.35
  84. v84Fourth-power monotonicity coefficient cut from 47 to 4071,142173 min · $50.31
  85. v85Monotonicity coefficient cut further to 3559,553.5178 min · $51.95
  86. v86Monotonicity coefficient raised to 4356,279180 min · $52.83
  87. v87Monotonicity coefficient cut slightly to 3867,263.5186 min · $55.20
  88. v88Rank-sum penalty coefficient cut from 11 to 960,931.5190 min · $56.71
  89. v89Rank-sum penalty coefficient raised from 11 to 1361,403193 min · $57.72
  90. v90Merge-run raised to 900 at monotonicity 40, an interaction retestaux 0-15 60,452.5196 min · $58.84
  91. v91Merge-run cut to 700 at monotonicity 4055,271.5200 min · $60.59
  92. v92Two-future-move boundary widened to <=5 empties under tuned row utilityaux 0-3 31,796201 min · $61.22
  93. v93Terminal penalty raised to 130 after row-weight tuning80,250204 min · $62.16
  94. v94Terminal penalty set to the intermediate 115aux 0-15 67,233.8210 min · $63.84
  95. v95Monotonicity coefficient cut narrowly to 3952,337217 min · $66.83
  96. v96Monotonicity coefficient raised narrowly to 4157,716219 min · $67.80
  97. v97Third future move at <=2 empties under the tuned row utility81,259223 min · $69.43
  98. v98Third future move narrowed to the most crowded <=1-empty afterstates83,581236 min · $74.84
  99. v99Critical <=1-empty search raised from three to four future movesaux 0-1 41,368241 min · $76.77
  100. v100(board, depth, node type) transposition keys packed into one integeraux 0-3 = v98250 min · $80.80
  101. v101Terminal penalty raised to 115 with critical depth-3 search76,858.5254 min · $82.44
  102. v102Terminal penalty cut to 90 with critical depth-3 search75,332.5257 min · $83.97
  103. v103Monotonicity coefficient cut to 38 with critical depth-3 search52,890260 min · $85.14
  104. v104Merge-run coefficient raised to 900 with critical depth-3 search68,525263 min · $86.59
  105. v105Depth 3 also at two empties once a 2048 tile exists74,548.5270 min · $89.22
  106. v106Rank-sum term 11 x rank^3.5 replaced by steeper 3.5 x rank^481,326.5274 min · $91.19
  107. v10775/25 blend of depth-3 and depth-2 values at <=1 empty60,828286 min · $97.13
  108. v108Merge-run coefficient raised narrowly to 850 with critical search66,079289 min · $98.78
  109. v109Obsolete classic-feature fields dropped; only the tuned row score is stored83,581292 min · $100.23
  110. v110Board-wide penalty on rows and columns whose gradients conflictseed 0 74,652304 min · $101.29
  111. v1110.5% worst spawn-location value blended into chance nodes at 2-4 emptiesaux 0-3 79,445316 min · $102.62
  112. v1120.5% best spawn-location value blended into chance nodes at 2-4 emptiesaux 0-3 70,703318 min · $102.98
  113. v113Row-table empty-cell coefficient raised narrowly from 270 to 280aux 0-3 47,645320 min · $103.28
  114. v114Depth 3 at exactly two empties only before a 2048 tile existsaux 0-3 77,763325 min · $104.17
  115. v115Two-empty depth-3 window narrowed to the early <=512 stage80,952.5329 min · $105.06
  116. v116Early two-empty depth 3 only when at most three moves are legalaux 0-3 59,671331 min · $105.49
  117. v117Early two-empty depth 3 only when at most two moves are legal83,569335 min · $106.31
  118. v118Depth-0 chance nodes reuse the parent row score, redoing the spawned line91,987340 min · $107.40
  119. v119Spawned rows rebuilt in v109's addition order; bit-exact, no child transposeaux 0-3 = v109342 min · $108.02
  120. v120Rows extracted once per depth-0 chance node; bit-exact, 32% fasteraux 0-3 = v109344 min · $108.65
  121. v121Transposition-cache hits read with one dict.get and a None sentinelpublic seed exact345 min · $109.11
  122. v122Fixed two-tile loop unrolled in the optimized depth-0 chance pathaux 0-3 = v120347 min · $109.83
  123. v123Same two-tile loop unrolled in recursive and one-empty chance nodesseed 0 exact 82,804348 min · $110.36
  124. v124Rank-concentration penalty cut narrowly from 11 to 10103,227354 min · $111.90
  125. v125Rank-concentration coefficient tested at 10.5aux 0-3 67,883360 min · $113.54
  126. v126Rank-concentration coefficient tested at 9.5aux 0-3 78,202361 min · $114.16
  127. v127Immediate merge-score weight cut from 0.001 to 0.0008aux 0-3 76,047363 min · $114.80
  128. v128Immediate merge-score weight raised from 0.001 to 0.0012aux 0-3 60,058364 min · $115.36
  129. v129Row-table smoothness penalty 10 per exponent gap between nonzero tiles82,351.5367 min · $116.38
  130. v130Explicit smoothness penalty halved from 10 to 5aux 0-3 61,532368 min · $116.97
  131. v131Merge-run length replaced by exact merges per compression, scaled 2xaux 0-3 33,890369 min · $117.51
  132. v132Weak 10 x rank x run-length bonus added on top of merge-run lengthaux 0-3 56,553371 min · $118.11
  133. v133Depth 3 at <=2 empties reopened under the rank-10 evaluatoraux 0-3 73,241374 min · $119.04
  134. v134Adjacent integer rank penalty 12 under critical depth-3 search73,810376 min · $120.11
  135. v135Rank penalty 10 through 1024, 11 for 2048-and-larger tiles82,927.5380 min · $121.23
  136. v136Rank penalty 11 below 2048, relaxed to 10 for 2048-and-larger tiles73,476382 min · $122.27
  137. v137Critical <=1-empty horizon raised from depth 3 to depth 4aux 0-1 24,182389 min · $124.61
  138. v138Regime switch: whole rank-11 table until 2048, then whole rank-10 table69,513402 min · $128.67
  139. v139One quiescence move plus exact spawn at full but mergeable horizon leavesaux 0-3 78,644405 min · $129.72
  140. v1400.001 symmetry-invariant serpentine bonus at root afterstates onlyaux 0-3 65,452407 min · $130.69
  141. v141Root-only serpentine coefficient cut tenfold to 0.0001aux 0-3 36,603408 min · $131.35
  142. v142All four move transforms batched at max nodes, sharing the UP/DOWN transposepublic seeds exact411 min · $132.53
  143. v143Four rows extracted once per direction pair; opposite moves share the gainpublic seeds exact412 min · $133.25
  144. v144Nibble-mask empty enumeration at <=2 empties, bit_count at rootspeed test only415 min · $134.48
  145. v145Batched move results flattened, max-node comparisons unrolled in move orderpublic seeds exact416 min · $135.33
  146. v146ROW_SCORES bound to a local alias in the chance-leaf loopvalues exact, no gain418 min · $136.19

146 snapshots in 7.0 h for $138. Screening ran on private aux seeds; held-out panels of fresh seeds gated the structural keeps and reversed v94.

On the hidden set

held-out metricreward
starter policy, sealed-seed mean2,473.250.00
human expert, sealed-seed mean32,753.250.60
calibration scale (soft cap)200,0000.90
this run79,026.500.7611
taskhttps://huggingface.co/datasets/autoresearch-arena/rsi-tasks-1/tree/main/2048_strategyrollouthttps://huggingface.co/datasets/autoresearch-arena/rsi-rollout/tree/main/2048_strategy/codex-gpt-5.6-sol-max/2048_strategy__wmbfip8
132 minwall clock
$27.92spend
45.3Mtokens
40versions, 12 kept
0 10k 20k 30k 40k 50k $0 $5 $10 $15 $20 cumulative spend on the run public 8-seed mean raw game score, higher is better human expert, sealed-seed mean · visible · 30,913.50 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 v25 v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40
keptrevertedno scoreturning point
  1. v1Inherited baseline: first legal move in UP, LEFT, RIGHT, DOWN order2,0600 min · $0.18
  2. v2One-ply log-space heuristic: empties, monotonicity, smoothness, max-in-corner3,369.51 min · $0.27
  3. v3Exact expectation over every spawn location and value3,8682 min · $0.33
  4. v4Two-ply exact expectimax: one more player decision and spawn, same evaluatorSearch instead of scoring the immediate board. Largest single change of the run, and still the depth at submission.18,019.54 min · $0.52
  5. v5Gap-skipping monotonicity plus natural-log empty reward12,954.57 min · $0.80
  6. v6Corner-stability weight doubled from 0.35 to 0.715,7679 min · $0.98
  7. v7Corner-stability term removed entirely17,69211 min · $1.14
  8. v8Corner-stability weight cut from 0.35 to 0.217,335.513 min · $1.31
  9. v9Empty-cell weight raised from 2.7 to 3.525,244.515 min · $1.52
  10. v10Empty-cell weight raised again to 4.321,691.517 min · $1.75
  11. v11Terminal boards at the horizon scored -100 instead of by the heuristicThe search had valued a dead board by how tidy it looked. Naming game-over as an outcome made it model risk: 7 of 8 seeds rose.33,54419 min · $2.02
  12. v12Terminal penalty strengthened from -100 to -30032,710.522 min · $2.30
  13. v13Adaptive third player ply at <=3 empties; suite interrupted after 7 minutessuite never finished32 min · $3.38
  14. v14Cached row merges and board moves; identical moves, 15% slower33,54435 min · $3.79
  15. v15Cached four-cell line features; identical moves, 17% faster33,54438 min · $4.39
  16. v16Evaluator swapped for a high-rank-disorder and merge-chain family14,219.540 min · $4.65
  17. v170.5 weight on adjacent equal tiles after compressing gaps, rows and columns36,08842 min · $5.17
  18. v18Merge-potential weight raised from 0.5 to 1.036,52345 min · $5.65
  19. v19Merge-potential weight raised to 1.533,74747 min · $6.10
  20. v20Merge-potential weight interpolated to 0.7537,44350 min · $6.70
  21. v21Smoothness weight doubled from 0.1 to 0.233,659.552 min · $7.16
  22. v22Smoothness weight halved from 0.1 to 0.0539,964.555 min · $7.67
  23. v23Smoothness term removed38,88757 min · $8.22
  24. v24Empty-cell weight raised from 3.5 to 3.8 under the merge-aware evaluator39,766.560 min · $8.77
  25. v25Monotonicity weight raised from 1.0 to 1.235,808.562 min · $9.27
  26. v26Monotonicity weight lowered to 0.831,602.564 min · $9.77
  27. v27Immediate merge-score coefficient raised from 0.0001 to 0.00138,385.567 min · $10.36
  28. v28Third move only at horizon boards with <=1 empty, spawns evaluated statically38,844.572 min · $11.28
  29. v290.2 reward for the strongest tile adjacent to the max corner23,961.576 min · $12.17
  30. v30Stage-dependent smoothness: 0.2 at <=2 leaf empties, otherwise 0.0522,38279 min · $12.85
  31. v3164-bit rank board, row moves and line features precomputed; 34% faster39,964.582 min · $13.45
  32. v320.5 reward per legal move direction at leaf boards (mobility)43,661.585 min · $14.07
  33. v33Mobility weight reduced to 0.25Reward how many moves stay legal, not only how the tiles sit. Best public mean of the run; dropped on a weaker 32-seed tail.47,318.587 min · $14.67
  34. v34Mobility weight raised to 0.3034,588.589 min · $15.22
  35. v35Mobility weight lowered to 0.2031,33491 min · $15.72
  36. v36Merge-potential cut to 0.5 alongside mobility 0.2535,858.598 min · $17.66
  37. v37Merge-potential raised to 1.0 alongside mobility 0.2549,648.5101 min · $18.32
  38. v38Exact third player ply once a 4096 tile exists; 5.4x runtime50,163.5112 min · $21.55
  39. v39Third ply only after 4096 and with <=2 live-board empties43,249.5115 min · $22.32
  40. v40Terminal penalty -300 under the mobility-aware heuristic38,872.5117 min · $22.95

Forty snapshots in under two hours for $22.95: one version is one 8-game public run. The last third went on fresh precommitted seeds, not new ideas.

On the hidden set

held-out metricreward
starter policy, sealed-seed mean2,473.250.00
human expert, sealed-seed mean32,753.250.60
calibration scale (soft cap)200,0000.90
this run38,772.250.6265
taskhttps://huggingface.co/datasets/autoresearch-arena/rsi-tasks-1/tree/main/2048_strategyrollouthttps://huggingface.co/datasets/autoresearch-arena/rsi-rollout/tree/main/2048_strategy/codex-gpt-5.6-sol-high/2048_strategy__iT6pWTt
42 minwall clock
$0.49spend
1.9Mtokens
3versions, 3 kept
0 10k 20k 30k 40k $0 $0.1 $0.2 $0.3 $0.4 cumulative spend on the run public 8-seed mean raw game score, higher is better human expert, sealed-seed mean · visible · 30,913.50 v0 v1 v2
keptrevertedno scoreturning point
  1. v0Inherited baseline: first legal move in UP, LEFT, RIGHT, DOWN order2,0600 min · $0.04
  2. v1Depth-2 expectimax over log2 tile values with snake-ordered weightsExpectimax over the spawn distribution in place of the fixed move order; the weight table, not the search, holds it at baseline.2,5641 min · $0.08
  3. v2Depth 2 above 6 empties, 4 below; base-5.6 exponential snake weightsWeight base and depth threshold picked by off-snapshot grid search; base 5.6 plus a deeper crowded-board search gave 19x baseline.39,88041 min · $0.43

Three snapshots in 42 min for $0.49. The base and depth-threshold grid search ran off-snapshot in benchmark.py between v1 and v2.

On the hidden set

held-out metricreward
starter policy, sealed-seed mean2,473.250.00
human expert, sealed-seed mean32,753.250.60
calibration scale (soft cap)200,0000.90
this run30,181.750.572
taskhttps://huggingface.co/datasets/autoresearch-arena/rsi-tasks-1/tree/main/2048_strategyrollouthttps://huggingface.co/datasets/autoresearch-arena/rsi-rollout/tree/main/2048_strategy/gemini-cli-gemini-3.7-flash-ultra/2048_strategy__rFp5uDa
714 minwall clock
$18.38spend
25.3Mtokens
9versions, 9 kept
40k 80k 120k 160k 200k $0 $4 $8 $12 $16 cumulative spend on the run devA:16 tuning-suite mean game score, higher is better human expert, sealed-seed mean · visible · 30,913.50 v0 v1 v2 v3 v4 v5 v6 v7 v8
keptrevertedno scoreturning point
  1. v0Inherited baseline: first legal move in UP, LEFT, RIGHT, DOWN orderpublic 2,0602 min · $0.32
  2. v1Bitboard expectimax, nneonneo row heuristic, depth 3 chance plies, cprob 1e-4Bitboard expectimax over the spawn distribution with a published row table, in place of a fixed move order. 57x the baseline.public 116,89413 min · $0.91
  3. v2Same moves, 2x faster core: carried transpose, leaf max node, post-move eval memodevA:24 123,23999 min · $4.99
  4. v32-tile-only at the deepest chance layer buys depth 4 when <=3 emptiesTrade exactness for depth: dropping the 4-tile spawn at the deepest layer costs 7%, the ply it buys is worth 26-35%.151,007209 min · $6.95
  5. v4Beam of 3 at internal max nodes plus coordinate-descent-tuned evaluator weights172,373309 min · $8.39
  6. v5Depth-dependent beam (3,3,2,2,...): depth 5 at <=1 empty, depth 4 at <=5191,865364 min · $10.03
  7. v6Deeper table at the same beam: depth 5 at <=3 empties, depth 4 at <=7203,018426 min · $10.96
  8. v7Beam (3,2,2,...) pays for depth 7 at 0 empties, 6 at 1, 5 at <=3, 4 at <=6215,757504 min · $13.66
  9. v8Cost-aware depth by (empties, merge pairs); worst move 1265 -> 569 msThe submission, chosen on latency not score: crowded boards drop a ply or two. Never scored on a full suite.not measured711 min · $17.70

Nine snapshots over 11.9 h and $17.70 of the run's $18.38. The log states its own limit: 16-24 game suites cannot separate configs under ~20%.

On the hidden set

held-out metricreward
starter policy, sealed-seed mean2,473.250.00
human expert, sealed-seed mean32,753.250.60
calibration scale (soft cap)200,0000.90
this run13,120.250.1031
taskhttps://huggingface.co/datasets/autoresearch-arena/rsi-tasks-1/tree/main/2048_strategyrollouthttps://huggingface.co/datasets/autoresearch-arena/rsi-rollout/tree/main/2048_strategy/claude-code-opus-5-high/2048_strategy__KNx3zwN
PreviousPostgreSQL 18 wire-compatible server on SQLite (Zig)NextOpenTTD Transport AI