Design an efficient medical multi-agent workflow for MedAgentsBench

You inherit a reasoning-only medical-QA workflow that orchestrates a fixed base LLM (GPT-4o, injected as an llm proxy) under a tight per-question call/token budget. The starting point in /app/methods/main/ is a deliberately weak single-agent CoT — no stronger reference is shipped, and going weak→strong is the task. You submit an algorithm (the methods/main/ directory exposing answer_batch(examples, llm, budget)); a sealed verifier copies it into a clean box and re-runs it on a hidden MedAgentsBench hard subset (salted ids, shuffled options), scoring macro accuracy averaged per data source — so overfitting the visible split is useless. Optimize macro_accuracy_pct (higher is better).

Hard Constraints

1. You may only edit code under /app/methods/main/; you may add sibling .py helper modules and import them from solver.py.
2. No internet, no model downloads, no external API calls of your own. You may only use the llm proxy passed in by the judge.
3. At most 4 LLM calls per question, at most 6000 total tokens per question, and at most 45s wall-clock per question.
4. The llm proxy uses a fixed model version, fixed temperature=0, fixed seed, and fixed decoding parameters. Do not change these.
5. The output must be exactly one option key per question, e.g. "A", "B", "C". Invalid output, empty output, multiple options, or explanatory text are all scored as wrong.
6. You must not submit a precomputed answer table, look up answers by id / realidx / question hash, read hidden answers, or try to bypass the proxy's budget counting.
7. Prompt search, role design, multi-agent debate, self-refine, answer verification, fallback selectors, and dataset-aware routing are all fair game — but it must generalize to the hidden set.

What You Have

• Visible data (/app/data/): visible_train.jsonl, visible_dev.jsonl, visible_gold.jsonl. Each example has id (a salted id, not the original realidx), dataset (data-source name), question (the stem), options (e.g. {"A": "...", "B": "..."}), and answer_idx — the gold letter, present only in the visible files, stripped from the hidden set before your solver is called.
• The editable baseline /app/methods/main/ — this directory is what gets graded. It is a weak single-agent CoT (solver.py + reusable prompt_utils.py). Improve it in place or rewrite it; multi-agent / self-refine / debate / selector / verifier designs are yours to build.
• Your self-check surface (/app/selfcheck.py): python /app/selfcheck.py (optionally --split visible_dev and --method <dir>) runs on the visible set, calling the same fixed GPT-4o base the grader uses. Reusable, but each run costs real LLM calls — use a small split. Visible scores do not represent the hidden score.

What You Submit

Edit /app/methods/main/solver.py and keep this exact signature:

def answer_batch(examples, llm, budget):
    """
    examples: list[dict] — each has at least id, dataset, question, options
                           (hidden examples do NOT contain answer_idx).
    llm:      callable    — llm(messages, max_tokens=..., stop=...); messages is OpenAI-style
                           [{"role": "...", "content": "..."}]. The proxy forces temperature=0
                           and records tokens and call counts.
    budget:   dict        — max_llm_calls_per_question, max_total_tokens_per_question,
                           max_wall_time_per_question_sec, etc.
    returns:  list[str]   — same length as examples; each element an option key, e.g. "A".
    """

You may add other .py files inside /app/methods/main/ (e.g. router.py, agents.py, verifier.py, prompts.py, parser.py) and import them from solver.py, but the entrypoint function name and signature must stay unchanged. There is no submit step and no per-attempt feedback — work and self-check as long as your run window allows, then leave your best methods/main/ in place; it is graded once at the end on the hidden test set.

How It Is Judged

A sealed verifier copies your methods/main/ into a clean box and runs it on the hidden MedAgentsBench hard subset. It imports solver.answer_batch, passes hidden examples and a controlled llm proxy, and collects the predicted options. The raw metric is:

macro_accuracy_pct = 100 * mean_over_datasets(correct / total)

— per data source first, then averaged across sources, so no single source's sample count dominates. If the budget is exceeded, the signature is wrong, the output length is wrong, the output is an invalid option, or the code tries to read hidden answers / reach the internet directly, the submission is treated as invalid and scored 0. The score is anchored baseline (single-agent CoT) -> 0, reference (AFlow paper SOTA landmark) -> 0.5, upper_bound (perfect accuracy) -> 1; the normalized reward is not shown to you. Beat the baseline and push toward (and ideally past) the SOTA landmark.