Tasks/Operations Research/Black-Box Optimization

Query-efficient stochastic inventory policy search

Tune an (s,S) inventory policy through a stochastic simulator

bbo_simopt_inventory_v1 Operations Research Black-Box Optimization
instruction.mdthis is what the agent is given

Design a reproducible optimizer for four normalized controls of a replenishment policy under stochastic demand and lead times. Minimize simulated operating cost within 48 queries over [0, 1]^4. You inherit a uniform-random starter and receive noisy objective feedback through a bounded ask/tell interface.

Hard Constraints

  • Edit only /app/methods/main/; solver.py must define Optimizer.
  • Use only the Python standard library and NumPy 2.2.6.
  • The constructor is Optimizer(dim, lower, upper, budget, seed, rng).
  • ask(n) must return a finite matrix with 1 through n rows, exactly dim columns, and every coordinate inside the supplied bounds.
  • The trusted evaluator owns the simulator, independent rescoring, query counter, and timeout. Extra returned points do not increase the query budget.
  • Import failures, crashes, malformed output, non-finite values, and out-of-bounds proposals invalidate the submission. If the aggregate runtime expires, completed valid work is retained under the published partial-work rule.
  • The submitted process cannot read or modify trusted evaluator assets and has no verifier network access.

The sealed evaluator runs many short optimizer sessions under one aggregate runtime boundary. Keep each ask and tell bounded and vectorized; expensive dense refits or very large candidate scans at every query can exhaust that shared budget.

What You Have

  • /app/data/visible.json contains public development cases spanning the same inventory-policy regimes as evaluation.
  • /app/data/visible_anchors.json contains public calibration traces used by the matched visible scorer.
  • /app/methods/main/solver.py is the uniform-random weak starter.
  • /app/selfcheck.py runs the same query, independent-rescoring, aggregation, and mapping semantics on public cases.
  • The four coordinates encode reorder level, order-up-to level, expedite threshold, and smoothing, each normalized to [0, 1].
  • The supplied rng is np.random.default_rng(seed) and should drive optimizer randomness. Simulator noise is controlled by the trusted evaluator.
  • The evaluator calls ask(n), evaluates the returned controls, and calls tell(X, y) or tell(X, y, metadata) when accepted. Values in y are costs, so lower is better.
  • A positive integer self.batch may request a preferred batch size; the evaluator caps it to the remaining budget.

What You Submit

Submit general optimizer code, not a final control vector or precomputed case answers. The submission must be self-contained in /app/methods/main/solver.py; sibling modules are not copied to the trusted verifier.

How It Is Judged

The trusted parent re-runs the optimizer on distinct cases with the same 48-query interface used publicly. When an observed incumbent improves, it is independently re-simulated with a candidate-keyed random stream. Best-so-far independent traces are summarized across repeated runs, with both sustained progress and final quality contributing to the metric. Better cost reduction across the full case mixture is better. Calibration assets and the exact leaderboard transformation remain verifier-only.

Metric

mean log-cost progress over 16 sealed inventory instances · higher is better

per instance 0.7*mean best-so-far log1p cost progress vs the floor trace over 48 queries + 0.3*final, median of 5 seeds

anchorvisible setheld-outreward
uniform random search (floor)0.00000.00000.00
Sfrozen reference optimizer0.02130.30
Uzero-cost bound1.00001.00
normalisation
m <= 00
0 < m <= S0.3 * f(m / S)
m > S0.3 + 0.7 * f((m - S) / (U - S))

m = this run's held-out metric  ·  S = frozen reference optimizer  ·  U = zero-cost bound

f(p) = 1 - log(1+3*(1-p))/log 4; S is that instance's reference quality, U = 1. Mapped per instance, then averaged over the 16.

Rollouts

83 minwall clock
$28.16spend
40.5Mtokens
62versions, 11 kept
0 0.005 0.010 0.015 0.020 0.025 $0 $7.5 $15 $22 cumulative spend on the run visible self-check raw quality, higher is better v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 v25 v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41 v42 v43 v44 v45 v46 v47 v48 v49 v50 v51 v52 v53 v54 v55 v56 v57 v58 v59 v60 v61
keptrevertedno scoreturning point
  1. v0Shipped uniform-random starter, batch 101 min · $0.26
  2. v1Seed-shifted Halton sequence replaces IID sampling0.0039965 min · $0.61
  3. v216-point regime portfolio of high-service and lean-stock policies, then HaltonSpend the budget on a hand-built spread of policy regimes instead of space-filling points. Largest single gain of the run.0.0223076 min · $0.85
  4. v3Rank-based Gaussian mutation of the top five observed policies0.0225018 min · $1.12
  5. v4Gaussian-process LCB search on log cost after the portfolio0.02241810 min · $1.39
  6. v5Systematic 16-point axis path: order-up-to in eighths, then reorder level0.02235914 min · $2.02
  7. v6Branch after five probes on the cost ratio y5/y1 < 0.83First use of feedback: read the five probe costs to tell a lean case from a service case, then refine the coordinate that matters.0.02264816 min · $2.29
  8. v7Zero expedite on probe 2, extra order-up-to 0.125 before the branch0.02170518 min · $2.63
  9. v8Four-probe branches, then perturb one of the three best observed controlsNine structured queries, then spend the rest resampling near the incumbents; near-equivalent resampling beat broad late search.0.02313320 min · $2.92
  10. v9Late resampling jitters only the expedite threshold0.02310722 min · $3.29
  11. v10Perturb only the single best observed parent0.02272923 min · $3.56
  12. v11Local perturbation radii halved to 0.006-0.0020.02311424 min · $3.88
  13. v12Local perturbation radii doubled to 0.024-0.0080.02279126 min · $4.15
  14. v13Local radii raised again to 0.040-0.0123-panel mean only26 min · $4.36
  15. v14Lean branch shortened from four probes to two3-panel mean only27 min · $4.58
  16. v15Both branches extended from four probes to six3-panel mean only28 min · $4.80
  17. v16Route on endpoint ratio 0.88 plus log-cost slope 0.15 over five probes0.0233530 min · $5.30
  18. v17Top-two parents weighted 0.75/0.25 instead of top-three3-panel mean only30 min · $5.53
  19. v18Top-three parent weights flattened to 0.50/0.30/0.200.02387232 min · $5.83
  20. v19Regime-anisotropic mutations: lean and non-lean stress different axes0.02428233 min · $6.15
  21. v20Mutate effective policy coordinates and remap them back to controlsinvalid proposal34 min · $6.43
  22. v21Same three-coordinate mutation with output clipping added3-panel mean only34 min · $6.67
  23. v22Prepend the static prior [1,.875,0,1] and route after six queriesofficial reward only36 min · $7.03
  24. v23Replace the first service prior with [1,.875,0,1], refit routing3-panel mean only37 min · $7.29
  25. v24Rank parents by 50/50 log observation and three-nearest-neighbour mean3-panel mean only38 min · $8.08
  26. v25Force the three local parents at least 0.05 apart3-panel mean only38 min · $8.35
  27. v26Randomize local radii over three scales3-panel mean only39 min · $8.62
  28. v27Drop the query-28 shrink, keep radius 0.024 through local search0.02342740 min · $8.96
  29. v28Constant local radius 0.0163-panel mean only41 min · $9.23
  30. v29Constant local radius 0.0303-panel mean only41 min · $9.50
  31. v30Lean-branch point [.875,0,0,.75] replaced by [.625,0,0,1]0.02342742 min · $9.96
  32. v31Move [.625,0,0,1] to the front of the lean branch8-panel mean only44 min · $10.65
  33. v32Route on endpoint and half-sweep ratios instead of the log slope13-panel mean only45 min · $11.03
  34. v33Probe 2 expedite set to zero, everything else fixedofficial reward only46 min · $11.49
  35. v34Single-best-parent local search retried on the stronger method3-panel mean only47 min · $11.82
  36. v35Parent pool widened to top four, weights 0.55/0.25/0.13/0.078-panel mean only48 min · $12.21
  37. v36Top-four parents through query 27, top-three after13-panel mean only50 min · $12.68
  38. v37Local mutations drawn in antithetic pairs around one parent3-panel mean only50 min · $13.01
  39. v38One random coordinate at radius 0.048 instead of four at 0.0248-panel mean only51 min · $13.42
  40. v39Reordered lean path with a conservative router (0.90 / 0.85)3-panel mean only53 min · $14.00
  41. v40Lean branch reordered only for high-confidence lean responses8-panel mean only55 min · $14.55
  42. v41Every fourth local proposal is a trigger-only resample3-panel mean only56 min · $14.90
  43. v42Parent weights sharpened to 0.75/0.20/0.0513-panel mean only57 min · $15.43
  44. v43Differential mutation along top-eight elite differences3-panel mean only58 min · $15.79
  45. v44Fast path reorders non-lean refinements when probe cost exceeds 1.05x8-panel mean only60 min · $16.63
  46. v45Fast-path threshold tightened from 1.05 to 1.208-panel mean only61 min · $17.90
  47. v46Fast-path threshold tightened again to 1.308-panel mean only62 min · $18.37
  48. v47Service-like branch ending in expedite and smoothing variants3-panel mean only64 min · $19.01
  49. v48Microscopic trigger jitter on every structured probe3-panel mean only66 min · $20.13
  50. v49Weighted centroid of the top three used as every local parent3-panel mean only67 min · $20.68
  51. v50High-cost sessions swap two grid probes for service variants3-panel mean only69 min · $21.17
  52. v51The same two service variants added at queries 10-11 instead3-panel mean only69 min · $21.58
  53. v52Mutations at the 0/1 bounds stay clipped 75% of the time3-panel mean only70 min · $22.01
  54. v53Bounded uniform mutations with matched per-coordinate variance3-panel mean only70 min · $22.42
  55. v54Radius 0.012 after a lean branch, 0.024 otherwise3-panel mean only71 min · $22.84
  56. v55Radius 0.036 after a lean branch only3-panel mean only72 min · $23.25
  57. v56Lean-only radius 0.0308-panel mean only73 min · $23.78
  58. v57Non-lean radius 0.030 with lean radius kept at 0.0243-panel mean only73 min · $24.21
  59. v58Router sensitivity search picks endpoint ratio 0.92 and slope 0.2013-panel mean only77 min · $25.20
  60. v59Success-direction momentum adds 25% to later mutations3-panel mean only78 min · $25.68
  61. v60Five nonadaptive probes requested as one batch, then batch 10.02342779 min · $26.59
  62. v61Four branch probes batched too, then batch 1 for local search0.02342780 min · $27.05

v0-v8 took 20 min and $2.93; the next 53 versions cost $24 more and moved the visible panel 0.02313 to 0.02343. Most were judged on multi-seed panels.

On the hidden set

held-out metricreward
uniform random search (floor)0.00000.00
frozen reference optimizer0.02130.30
zero-cost bound1.00001.00
this run0.0264010.2753