Tasks/Operations Research/Black-Box Optimization

Noisy Continuous Black-Box Optimizer Design

Optimize a noisy multimodal function on a fixed query budget

bbo_noisy_continuous_v1 Operations Research Black-Box Optimization
instruction.mdthis is what the agent is given

Your task is to design a reproducible, noise-aware optimizer that minimizes 10-dimensional continuous black-box objectives with smooth periodic, multimodal structure. You inherit a weak uniform-random baseline, and every run uses the box [-5, 5]^10 with a strict budget of 120 objective queries. Your submitted optimizer is re-run on disjoint sealed instances.

Hard Constraints

  • Edit only /app/methods/main/; solver.py must define Optimizer.
  • Use only the Python standard library and NumPy 2.2.6.
  • The required constructor is Optimizer(dim, lower, upper, budget, seed, rng). No task-information argument is supplied.
  • ask(n) must return a finite NumPy-compatible matrix with 1 through n rows, exactly dim columns, and all coordinates within the supplied bounds.
  • The verifier owns the objective and query counter. Extra returned points do not increase the 120-query budget.
  • Import failures, crashes, malformed output, non-finite values, and out-of-bounds proposals invalidate the complete submission.
  • The submitted process cannot read or modify trusted evaluator assets and has no verifier network access.

Runtime budget.

Submitted optimizer code receives a 120-second aggregate soft budget across the complete sealed suite, not a separate 120 seconds for every run. The suite contains 400 independent optimizer runs, so each run must average about 0.3 seconds. There is no five-second scoring cutoff on every individual ask or tell response; when the aggregate soft budget expires, the verifier stops requesting new work and evaluates the best completed state. A stalled process can still be terminated at a sealed safety cap. Use bounded, vectorized per-query work; repeated dense refits or hundreds-wide candidate scans at every observation are unlikely to fit. This budget applies to sealed execution, not to your research time.

What You Have

  • /app/data/visible.json contains twelve public development instances from the same noisy continuous family; sealed instances are distinct.
  • /app/methods/main/solver.py is a uniform-random baseline.
  • /app/selfcheck.py evaluates the same higher-is-better normalized anytime/final metric family used by the sealed evaluator on 20 deterministic runs per visible instance. It also reports diagnostic components and latent final-objective summaries.
  • The supplied rng is np.random.default_rng(seed) and should drive all randomness for deterministic replay.
  • The verifier repeatedly calls ask(n), evaluates the returned points, and calls tell(X, y) (or tell(X, y, metadata) if accepted). Each value in y is an observed noisy loss, so lower is better even though the aggregate self-check score is higher-is-better.
  • A positive integer self.batch may request a preferred batch size; the verifier negotiates and caps it to the remaining budget.

What You Submit

Submit optimizer code, not a one-shot point or precomputed answer. The entire submitted optimizer must be self-contained in /app/methods/main/solver.py, which must define the required Optimizer class; sibling modules are not copied to the trusted verifier.

How It Is Judged

The trusted parent evaluates your optimizer on sealed instances and fixed seeds. It records authoritative latent best-so-far traces, aggregates them robustly across seeds, and combines anytime quality with final-query quality under the same metric definition exposed by the visible self-check. Higher normalized quality is better; sealed instances, calibration assets, and evaluator internals remain hidden.

Metric

oracle-normalized anytime/final quality, 20 sealed instances · higher is better

0.70*anytime + 0.30*final quality; uniform-random floor = 0, exact oracle = 1; median of 20 seeds, 120 queries

anchorvisible setheld-outreward
uniform-random floor (shipped starter)0.0000.0000.00
Rhuman reference (fixed centered CMA)0.4620.5040.30
exact oracle (planted minimum)1.0001.0001.00
normalisation
q <= 00
0 < q <= R0.3 * q / R
q > R0.3 + 0.7 * (q - R) / (1 - R)

R = human reference (fixed centered CMA)

R is the instance's own sealed reference quality (mean 0.504). Each of the 20 instances is mapped and clipped, then averaged.

Rollouts

176 minwall clock
$50.35spend
78.6Mtokens
39versions, 35 kept
0 0.15 0.30 0.45 0.60 $0 $10 $20 $30 $40 cumulative spend on the run visible self-check, 12 instances x 20 seeds, higher better human reference · visible · 0.462 v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 v25 v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37 v38
keptrevertedno scoreturning point
  1. v0Shipped uniform-random baseline01 min · $0.21
  2. v1Bounded full-covariance CMA evolution strategy, batch 10, normalised sigma 0.450.2237625 min · $0.64
  3. v2Initial normalised CMA scale 0.45 -> 0.350.3087487 min · $0.90
  4. v3Initial normalised scale 0.35 -> 0.300.3748947 min · $1.06
  5. v4Initial normalised scale 0.30 -> 0.250.3801468 min · $1.19
  6. v5Initial normalised scale 0.25 -> 0.200.3954828 min · $1.42
  7. v6Initial normalised scale 0.20 -> 0.150.3914089 min · $1.54
  8. v7Batch size screened 4-30; 10 -> 7 for more adaptation generations0.44506814 min · $2.25
  9. v8Batch 6 confirmation0.43584115 min · $2.36
  10. v9Batch 8 confirmation0.43449415 min · $2.47
  11. v10Initial sigma swept 0.10-0.35; cross-fold-stable 0.175 tried0.43975216 min · $2.71
  12. v11Antithetic sampling: three mirrored pairs plus one free sample per batchMirrored pairs cancel first-order noise that seven independent samples cannot, so each generation adapts on a cleaner signal.0.51310917 min · $2.93
  13. v12Spend the query left by 120 mod 7 on evaluating the learned mean0.51895921 min · $3.63
  14. v13After generation 10, replace each batch's free mutation with a mean evaluation0.53108122 min · $3.98
  15. v14Fixed-radius mutually orthogonal directions for each mirrored trio0.51441331 min · $5.44
  16. v15One random orthonormal frame per run, cycled across generations0.53327433 min · $5.83
  17. v16Sample through a Cholesky covariance root, keeping the frame's orientationEigendecomposition re-rotates the frame each generation; a Cholesky root keeps the planned directions intact as covariance adapts.0.59009134 min · $6.13
  18. v17Switch the odd seventh query to mean evaluation from generation 6, not 100.592437 min · $6.82
  19. v18Three regularised local parabolas per batch, blended 50/50 with rank recombinationMirrored pairs already measure curvature along each direction, so fit it and take a Newton step instead of only ranking.0.61086544 min · $8.35
  20. v19Start centre sampling and quadratic blending at generation 2 instead of 60.6252646 min · $8.89
  21. v20Quadratic curvature regularisation floor 2 -> 50.62231450 min · $10.22
  22. v21Quadratic displacement clip +/-0.75 -> +/-1.00.62942652 min · $10.72
  23. v22Conditional restart at query 49, sigma 0.225, when best loss stays above 400.62921469 min · $15.64
  24. v23Shrink sigma 25% once at query 42 when best noisy loss is already below 150.63088773 min · $16.94
  25. v24Relative restart trigger: loss above 30 and over 70% of its query-14 level0.63155886 min · $21.00
  26. v25Scale the quadratic blend by the positive-curvature fraction for 10 generations0.63637691 min · $23.16
  27. v26Evaluate each model-assisted batch's mean before its six mutations0.63582694 min · $23.93
  28. v27From generation 8, order mutations by evolution-path alignment0.63680797 min · $25.26
  29. v28Size-generic odd-request pair bookkeeping; sequential ask/tell contract tests0.636807100 min · $26.09
  30. v29Require measured directional curvature above 3, not merely positive0.637221105 min · $27.82
  31. v30First-attempt initial sigma 0.20 -> 0.225, matching the restart0.631106110 min · $31.64
  32. v31Ten-fold scale study selects initial sigma 0.230.619496113 min · $33.67
  33. v32Revert the curvature threshold to the plain positive-curvature test0.621136117 min · $36.13
  34. v33Drop the generation-10 switch; positive-curvature fraction used throughout0.620658120 min · $38.61
  35. v34Curvature-fraction blend coefficient 0.5 -> 0.60.629025130 min · $42.24
  36. v35Noisy-parabola curvature denominator floor 5 -> 80.635345134 min · $42.86
  37. v36Both CMA covariance learning rates multiplied by 0.250.647654146 min · $44.23
  38. v37Relative restart requires 75% of the query-14 loss, not 70%0.653211154 min · $45.76
  39. v38Cap the endpoint slope at 0.25 on negative-curvature directions, not discard0.661903166 min · $48.18

39 snapshots, 2h56, $50.35; only v6 and the two batch-size confirmations reverted. From v25 on every keep was re-checked on fresh 20-seed folds.

On the hidden set

held-out metricreward
uniform-random floor (shipped starter)0.0000.00
human reference (fixed centered CMA)0.5040.30
exact oracle (planted minimum)1.0001.00
this run0.66660.5487
219 minwall clock
$18.13spend
22.8Mtokens
8versions, 8 kept
0 0.2 0.4 0.6 0.8 $0 $3 $6 $9 $12 cumulative spend on the run visible self-check, 12 instances x 20 seeds, higher better human reference · visible · 0.462 v0 v1 v2 v3 v4 v5 v6 v7
keptrevertedno scoreturning point
  1. v0Shipped uniform-random baseline015 min · $2.04
  2. v1Own numpy CMA-ES from the origin, sigma0=1.0, lambda=100.410415 min · $2.04
  3. v2GP-LCB with a kernel derived from the family: SE l=3.28, s^2=289, mu0=90Read the kernel off the reverse-engineered generator instead of fitting one: a GP at length-scale 3.28 plus a 90-unit dimple.0.520816 min · $2.10
  4. v3Tuned GP-BO: LCB beta 1.5 -> 3.5, init design 20 -> 10, span 3.33 -> 2.00.722557 min · $4.19
  5. v4Incremental K^-1 by Sherman-Morrison; analytic gradient refinement of the LCB0.755592 min · $6.08
  6. v5Analytic Gaussian-well prior mean replaces the constant 90; full-box candidatesPut the centre bias in the model rather than in the candidate clipping, so the search can cover the whole box.0.7798119 min · $7.55
  7. v6Objective-scale adaptivity from the initial design; wall-clock guard 0.25 s/runEvery derived constant assumes the family's 90-unit scale, so estimate that scale from the initial design and rescale mu and s2.0.7749161 min · $10.96
  8. v7Hardened rewrite: box- and dim-relative constants, sanitised output, 0.27 s guard0.7731186 min · $14.32

Eight snapshots, 3h39, $18.13, nothing reverted. Every kept decision was re-checked on fresh synthetic instances; an untouched holdout agreed.

On the hidden set

held-out metricreward
uniform-random floor (shipped starter)0.0000.00
human reference (fixed centered CMA)0.5040.30
exact oracle (planted minimum)1.0001.00
this run0.56260.4899
91 minwall clock
-spend
-tokens
17versions, 15 kept
0 0.15 0.30 0.45 0.60 0 50 100 150 200 agent step (this harness reports no tokens or timestamps) visible self-check, 12 instances x 20 seeds, higher better human reference · visible · 0.462 v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16
keptrevertedno scoreturning point
  1. v0Shipped uniform-random baseline0
  2. v1Mirrored CMA-ES from the origin, lambda=8, sigma=1.1, last 12 queries polish0.446681
  3. v2Clip samples to [-pi,pi] where the family's minimisers live; sigma 1.20The planted minima sit inside [-pi,pi], so spend no queries on the rest of the box.0.515405
  4. v3CMA for 105 queries, then a ridge-quadratic Newton polish0.511947
  5. v4Evaluate the same CMA points nearest-to-best first, for anytime credit0.497502
  6. v5Covariance-shaped (1+1) polish over the last 10 queries, 1/5 success rule0.51909
  7. v6Polish step 0.18 -> 0.220.520645
  8. v7After 4 CMA generations, screen 32 mirrored candidates by a ridge quadraticA generation only ranks what it pays for; score 32 candidates on a local model first so the 8 real queries land on good points.0.550917
  9. v8Quadratic screening starts a generation earlier, warmup_gens 4 -> 30.576564
  10. v9Screen the last partial CMA generation too0.580341
  11. v10Rank candidates by a z-scored blend of ridge quadratic and Gaussian RBFOne surrogate misreads a multimodal landscape; averaging two z-scored models keeps the ranking usable where they disagree.0.592248
  12. v11Blend weight 0.5, leaning on the RBF surrogate0.606318
  13. v12Polish budget to 8, two more filtered CMA queries0.608385
  14. v13Initial sigma 1.20 -> 1.180.610802
  15. v14Candidate pool 32 -> 340.616678
  16. v15RBF fitted on the 80 nearest points; the quadratic stays at 400.624856
  17. v16RBF neighbourhood 80 -> 900.629431

17 snapshots, 1h31; this harness reports no cost or tokens. The verifier's 120 s cap stopped it at 284 of 400 runs, zeroing 6 instances.

On the hidden set

held-out metricreward
uniform-random floor (shipped starter)0.0000.00
human reference (fixed centered CMA)0.5040.30
exact oracle (planted minimum)1.0001.00
this run0.43890.3041
29 minwall clock
$3.23spend
4.7Mtokens
27versions, 9 kept
0 0.1 0.2 0.3 0.4 0.5 $0 $0.8 $1.5 $2.2 $3 cumulative spend on the run tuning score, visible instances 0-8, 8 seeds, higher better human reference · visible · 0.462 v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 v25 v26
keptrevertedno scoreturning point
  1. v0Shipped uniform-random baseline0$0.12
  2. v1Full-covariance CMA-ES, population 10, random start, normalised sigma 0.220.092646$0.24
  3. v2Six 20-point paired coordinate finite-difference/Adam iterations0.032535$0.36
  4. v3SPSA with 2-point antithetic updates and Adam smoothingUnder 120 noisy queries in 10 dimensions a two-query antithetic gradient buys more per query than a covariance model.0.159799$0.48
  5. v4Screened nine SPSA perturbation/rate pairs; c=0.50 physical, rate=0.0500.19629$0.60
  6. v516-point uniform basin seeding, then 52 SPSA updatesSPSA is local and the objective multimodal, so spend an eighth of the budget deciding which basin the rest polishes.0.227545$0.72
  7. v6Initial design screened at 8/12/24/32 points; 8 selected0.237962$0.84
  8. v7Randomised rows of a 16x16 Hadamard block replace iid Rademacher directionsIndependent directions interfere across coordinates; an orthogonal block cancels that, so the same queries buy a cleaner gradient.0.269541$0.96
  9. v8Momentum and rate screened; beta=0.70, rate=0.0600.282116$1.08
  10. v9Perturbation decay gamma=0.2 and learning decay alpha=0.10.288508$1.20
  11. v10Reset the SPSA centre to the best symmetric-pair mean every 16 updates0.279126$1.32
  12. v118-point jittered Latin hypercube instead of eight iid global seeds0.269806$1.44
  13. v12Constrain every SPSA centre to [c, 1-c] so paired perturbations stay symmetric0.282575$1.56
  14. v13Spend the last 20 queries on coordinate polishing, radius 0.350.282934$1.68
  15. v14Randomised sequential coordinate finite differences, 56 two-query updates0.209473$1.80
  16. v15First-moment injection corrected to 0.30; rate lowered to 0.0350.28438$1.91
  17. v16Average two balanced direction estimates per 4-query update0.239539$2.03
  18. v17Extend the perturbation-decay screen to gamma = 0.3/0.4/0.54-seed only, 0.284445$2.15
  19. v18Advance the top two seeds eight updates each, then commit to the better0.264446$2.27
  20. v19Base learning rate 0.060 -> 0.0650.289519$2.39
  21. v20Periodic latent evaluation of the SPSA centre, six checks then two late0.28737$2.51
  22. v21Exactly orthogonal 12-direction Paley-Hadamard block instead of 160.281632$2.63
  23. v22Random dense orthogonal QR direction blocks, c screened at 0.04/0.054-seed only, 0.290189$2.75
  24. v23Re-screen initial seed counts 4/6/10/12 after the Hadamard and decay changes4-seed only, 0.285441$2.87
  25. v24Second-moment beta2 screened; 0.80 selected0.294159$2.99
  26. v25Conservative beta2 = 0.85, between v9 and v240.287957$3.11
  27. v26Consolidate the exact Hadamard-block gradient into momentum every 16 updates0.288824$3.23

Nothing was snapshotted, so these 27 points come from the log alone. 29 min, $3.23, 18 reverted; v9 restored byte-for-byte after an audit.

On the hidden set

held-out metricreward
uniform-random floor (shipped starter)0.0000.00
human reference (fixed centered CMA)0.5040.30
exact oracle (planted minimum)1.0001.00
this run0.28780.1896
69 minwall clock
$1.48spend
9.0Mtokens
5versions, 5 kept
0 0.1 0.2 0.3 0.4 0.5 $0 $0.3 $0.6 $0.9 $1.2 cumulative spend on the run visible self-check, 12 instances x 20 seeds, higher better human reference · visible · 0.462 v0 v1 v2 v3 v4
keptrevertedno scoreturning point
  1. v0Shipped uniform-random baseline02 min · $0.10
  2. v1Matern 5/2 GP-BO with LCB, y-normalised, 10 random init, 500 candidates120 queries are too few to search, so fit a GP and let LCB pick where the model is both low and uncertain.0.3399443 min · $0.13
  3. v2Init design 10 -> 5 points; LCB kappa decays linearly 3.0 -> 0.1Tie exploration to the budget: wide kappa early, near-greedy by the last query, and half the init design given back.0.38490931 min · $0.47
  4. v3Latin hypercube for the global half of the candidate pool; init stays random0.39742742 min · $0.71
  5. v4Minimum 0.02 distance to observed points; nested Cholesky fallbacks0.39742769 min · $1.44

Five snapshots, 1h09, $1.48, nothing reverted; 8 unsnapshotted screens all failed. The 120 s cap stopped it at 159 of 400 runs, zeroing 12.

On the hidden set

held-out metricreward
uniform-random floor (shipped starter)0.0000.00
human reference (fixed centered CMA)0.5040.30
exact oracle (planted minimum)1.0001.00
this run0.15890.0784