Tasks/LLM Systems & Training/Inference-Time Compute

Budgeted Branch Evidence Control

Spend a fixed evidence budget across opaque branches

parallel_tts_qwen3_vnext LLM Systems & Training Inference-Time Compute
instruction.mdthis is what the agent is given

Improve a deterministic Python controller that allocates a limited observation budget across opaque evidence branches and returns one observed integer answer. The controller must generalize across unseen sessions whose branch reliability, calibration, dependence, and evidence timing vary.

Hard Constraints

  • Edit only /app/methods/main/controller.py and retain solve(session) -> dict as the entry point.
  • Use only the Python standard library. Do not access the network, launch external programs, read files, inspect process state, or depend on state retained from another invocation.
  • Interact with each session only through branch_ids, budget, spent, remaining, and probe(branch_id).
  • Do not probe an unknown branch, a finished branch, or any branch after the total budget is exhausted.
  • Return exactly {"answer": value}, where value is a non-boolean integer in [0, 999] that appeared in an observation returned during the current invocation.
  • Be deterministic for identical session transcripts. Do not assume fixed branch identifiers, panel sizes, answer values, branch order, or observation order.

What You Have

The editable starter is /app/methods/main/controller.py. Three visible panels under /app/public/ contain labeled development sessions; these files support research and diagnostics, but submitted code receives only a session proxy.

The proxy exposes an immutable tuple session.branch_ids, integer counters budget, spent, and remaining, and session.probe(branch_id). Each paid probe returns exactly answer, confidence, support, evidence_key, and finished. Confidence and support are bounded floats, evidence keys are opaque dependence identifiers, and finished states whether that branch has no later observation.

Run python /app/score_visible.py for panel and anonymous-group exact-answer accuracy. Run python /app/selfcheck.py to validate the public data, interface, budget accounting, determinism, and starter reproducibility.

What You Submit

Submit the edited /app/methods/main/controller.py with this callable interface:

def solve(session) -> dict:
    ...

The return object must have exactly one key and its answer must have been observed through a paid probe in that invocation.

How It Is Judged

A separate no-network evaluator stages only bounded Python source and runs a fresh unprivileged process for each disjoint private session. A trusted parent owns the branch streams, target, counters, timeouts, and output validation and sends the child only requested observations.

For every panel, exact-answer accuracy is computed independently within each of six precommitted anonymous groups. Each group result is calibrated on its matching frozen scale, and the final score is the equal arithmetic mean across all panel-group units. Higher exact-answer accuracy is better; invalid, non-deterministic, over-budget, unreadable, or nonconforming submissions fail closed.

Metric

mean exact-answer accuracy over the 18 sealed panel-group units · higher is better

Share of held-out sessions answered exactly right, scored per panel-group unit of 32 episodes.

anchorvisible setheld-outreward
Bround-robin majority baseline0.15630.16320.00
Ssequential evidence pooling reference0.69100.73440.30
Utarget-aware privileged oracle1.00001.00
normalisation
m <= B0
B < m <= S0.3 * (m - B) / (S - B)
S < m < U0.3 + 0.7 * (m - S) / (U - S)
m >= U1

m = this run's held-out metric  ·  B = round-robin majority baseline  ·  S = sequential evidence pooling reference  ·  U = target-aware privileged oracle

B, S, U are that unit's own anchors and U is 1.0. Mapped per panel-group unit, then averaged over the 18 with equal weight.

Rollouts

75 minwall clock
$19.20spend
23.4Mtokens
15versions, 15 kept
0.2 0.4 0.6 0.8 1.0 $0 $4 $8 $12 $16 cumulative spend on the run visible mean exact-answer accuracy, higher is better sequential evidence pooling r… · visible · 0.6910 v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14
keptrevertedno scoreturning point
  1. v0Unchanged starter: round-robin probing with a raw majority vote0.156251 min · $0.29
  2. v1One probe per branch, greedy deepening by mean support, cubic-support rankingRank answers by support and drop duplicates that share an evidence key, instead of counting votes that may share one source.0.92013910 min · $1.21
  3. v2Score bonus of 0.20 when an observation revises its branch's previous answer0.94444413 min · $1.79
  4. v3Sharper support transform, smaller best-support bonus, centered confidence term0.94791718 min · $2.77
  5. v4Majority ensemble of three nearby calibrations, v3 as the tie-break0.95312522 min · $3.65
  6. v5Capped 0.5 penalty per answer/evidence-key pair seen more than once0.95833324 min · $4.35
  7. v6Covariance-gated switch to a quartic support and revision scorer0.96180630 min · $5.97
  8. v7Posterior-guided branch index replaces mean-support deepeningChoose where the next probe goes from answer frequency and pull count, not from branch support alone.0.9687533 min · $7.08
  9. v8Confidence-led sessions get +0.30 confidence weight and no exploration term0.97048636 min · $8.09
  10. v9Ensemble respanned over soft, medium and sharp support transforms0.97222239 min · $9.39
  11. v10Three observable calibration-regime switches select the scorerRead the regime off the initial sweep and pick a scorer per session, instead of one scorer for every session.0.98263948 min · $11.09
  12. v11Diminishing-return branch penalty for independent consensus-heavy sweeps0.98784759 min · $13.33
  13. v12Extra scorer for the low-support, negative-covariance tail regime0.98958364 min · $15.06
  14. v13Dependence-aware depth allocator when the initial sweep repeats a key0.99131967 min · $16.05
  15. v14Heterogeneous-consensus allocation gate for independent initial sweeps0.99479272 min · $17.71

v1 alone took 9.7 min and $1.21 and won 0.764 of the 0.839 total visible gain; the last four versions cost $6.63 more for 0.012.

On the hidden set

held-out metricreward
round-robin majority baseline0.16320.00
sequential evidence pooling reference0.73440.30
target-aware privileged oracle1.00001.00
this run0.94970.885