Design one efficient agent workflow for graduate MATH *and* STATISTICS/PROBABILITY problems

You inherit a reasoning workflow that orchestrates a fixed base LLM (injected as an llm proxy, temperature=0) under a tight per-question call/token budget. The starting point in /app/methods/main/ is a deliberately weak single-agent chain-of-thought solver — no stronger reference is shipped, and going weak→strong is the task. You submit an algorithm (the methods/main/ directory exposing answer_batch(examples, llm, budget)); a sealed verifier copies it into a clean box and re-runs it on a hidden test split of graduate math + statistics/ probability problems, grading each returned solution against a sealed answer key to optimize accuracy_pct (higher is better).

Hard Constraints

1. You may only edit code under /app/methods/main/; add sibling .py helpers and import them from solver.py.
2. No internet, no model downloads, no external API calls of your own. Use only the llm proxy passed in by the judge.
3. Per question: at most 4 base-LLM calls, at most 8000 total tokens, at most 90s wall-clock. Exceeding any of these scores that question wrong. Total tokens means the provider-reported input/prompt plus output/completion tokens, summed across every call. Text repeated into later calls (the problem, drafts, critiques, etc.) is counted again; adding only your requested max_tokens values is not a valid budget calculation.
4. The llm proxy uses a fixed model, fixed temperature=0, and fixed decoding. Do not change these.
5. Each returned element is a full worked solution (reasoning + an explicit final answer). End with a clearly stated Final answer: ... line; for multi-part problems give every part on that line (e.g. Final answer: V(A)=900/271, V(B)=1000/271, V(C)=810/271).
6. You must not read the hidden answer key, key behavior on id, or bypass the proxy's budget counting. (Reading answer_key / grading_scheme / gold.jsonl / sealed paths, or opening your own network/process, invalidates the run.)
7. Any workflow architecture is allowed, but it must generalize across both domains within the budget and obey every constraint above.
8. A submitted workflow may use genuinely general symbolic/numeric computation, but it must not dispatch on distinctive visible-problem phrases, proper nouns, story entities, or conjunctions of wording that identify individual dev templates. Do not improve the suite by adding one lexical special case per observed failure. Such visible-template routing is overfitting even when the numeric values are parsed rather than copied; select a domain-general reasoning or computation architecture instead.

What You Have

• The problem family mixes exact-computation questions with derivation-heavy reasoning questions. A workflow tuned narrowly to either style will not generalize across the family within budget.
• Visible dev data (/app/data/): visible_dev.jsonl (problems: id, problem, difficulty, subtopic). The answer key is held by the self-check service and is not readable from the agent environment; self-check reports only per-problem PASS/fail and aggregate accuracy. The task pool contains exactly 25 visible dev problems and 12 sealed test problems across graduate mathematics, statistics, and probability. The split is ID-, text-, and template-disjoint; even 25/25 dev is useful debugging evidence, not proof of held-out generalization.
• The editable baseline /app/methods/main/ — this directory is what gets graded. It is a weak single-agent CoT (solver.py + reusable prompt_utils.py); one base-model call per problem, no verification. Improve it in place or rewrite it.
• Your self-check (/app/selfcheck.py): python /app/selfcheck.py always runs the complete, fixed 25-problem visible dev set with the same grader the verifier uses. It reports correctness together with calls, provider-reported total tokens, wall time, and final-line validity. Every experiment must use all 25 problems so checkpoint scores remain directly comparable. Each run costs real LLM calls. Visible scores do not equal the hidden test score.
• Before selecting a final workflow, run python /app/selfcheck.py --budget-smoke. This invokes the submitted workflow on four answer-free, private diagnostic problems outside the visible file and reports only calls/tokens/time/final-line validity, not correctness or answers. It is a fallback-path resource/contract check, not an additional dev set and not a score to optimize.

What You Submit

Edit /app/methods/main/solver.py, keeping this exact signature:

def answer_batch(examples, llm, budget):
    """
    examples: list[dict] — each has id, problem, difficulty, subtopic (hidden examples carry NO answer).
    llm:      callable    — llm(messages, max_tokens=..., stop=...); messages is OpenAI-style
                            [{"role": "...", "content": "..."}]. temperature=0; the proxy records
                            token + call counts and raises when the per-question budget is exceeded.
    budget:   dict        — max_llm_calls_per_question, max_total_tokens_per_question, ...
    returns:  list[str]   — same length as examples; each a FULL worked solution ending in
                            'Final answer: ...'.
    """

Add other .py files inside methods/main/ and import them from solver.py, but keep the entrypoint name + signature. There is no submit step and no per-attempt feedback — leave your best methods/main/ in place; it is graded once at the end on the hidden test.

How It Is Judged

A sealed verifier copies your methods/main/ into a clean box and runs it on the hidden test split under the fixed base model + per-question budget. It imports solver.answer_batch, collects one solution per problem, and grades each final answer against the sealed key. Grading is deterministic exact matching first: your final answer is parsed and checked for symbolic / numeric equivalence with sympy (fractions = decimals = percents, tuples, sets/multisets, and multi-value answers all count; 29/256 == 0.11328125, 1/2 == 0.5). When the exact parser cannot confidently resolve an unusual answer format, and for genuinely open-ended items (proofs, "insufficient-data" trick questions, case-dependent answers), grading falls back to a strong independent LLM judge. The raw metric is:

accuracy_pct = 100 * (problems judged correct) / (total problems)

Because most problems are matched exactly, a correct value in a messy or missing Final answer: line still scores wrong — state every part cleanly. Budget overrun, wrong signature/length, reading the sealed key, or reaching the internet directly → that submission (or that question) is scored 0. The normalized score is a monotonic function of held-out accuracy and is not shown to you; optimize raw accuracy and generalization.

Common Pitfalls

• Overfitting the dev split. The hidden test uses different problems; a high dev score does not transfer. In particular, 25/25 visible is not a completion condition. Build for generalization across domain and difficulty; ID lookup, problem fingerprints, visible-story phrase dispatch, per-template solvers accumulated from individual failures, and memorized answer tables are invalid submissions.
• One-domain tunnel vision. A pure code/sympy operator nails arithmetic but misreads derivation-heavy stats; pure CoT reasoning fumbles exact arithmetic. Reconcile the two.
• No clean final answer. Grading parses your Final answer: line — always end with one, with every part for multi-part problems; a buried, approximate-only, or missing answer scores wrong.
• Blowing the budget. Gate extra calls (critique, re-derivation, self-consistency) on low confidence rather than spending them on every problem. Account for both inputs and outputs, especially when drafts are copied into later prompts, and require headroom on the budget smoke.