Medical-VQA answerability and abstention routing

You inherit a medical visual-question-answering component from a teammate. A fixed, imperfect model proxy is already wired up — a vision-language model that emits an answer for an (image, question) case, and a cross-modal scorer that rates how well the image matches the question — but the teammate's stop-gap simply answers every case at face value. Many cases are not answerable from the image alone: some questions need a fact from the patient's electronic health record, some images were swapped and no longer match the question, and some images are masked or non-diagnostic. Confidently answering those cases is costly. Your job is to submit a better routing policy that decides, case by case, whether to answer (and with what) or to abstain (and why). The policy is graded by re-running it on hidden cases against a REAL GPU proxy (Qwen/Qwen2.5-VL-7B-Instruct as vlm() + BiomedCLIP as clip()), injected by the verifier.

Hard Constraints

1. You may only edit code under /app/methods/main/; you may add sibling .py modules and import them from solver.py. The entrypoint function name and signature must not change.
2. The submission must use ONLY the Python standard library and the resources passed to your function. Do NOT import torch, transformers, a datasets loader, open_clip, huggingface_hub, qwen_vl_utils, or any model/network library, and do NOT open the network or spawn processes — the verifier scans for and rejects these. The model proxy is provided to you as injected callables; you never load a model yourself.
3. Do not attempt to read the sealed grading labels, the held-out files, or re-derive answers from any public dataset. Case identifiers are salted and the hidden grade uses a different proxy instance than your local dry-run.
4. Your function must return exactly one prediction per input case. Each prediction is {"id", "decision", "answer", "reason_code"} with decision in {"answer","abstain"} and reason_code in {"image_answerable","needs_ehr","mismatched_image","insufficient_evidence"}. Malformed output, wrong length, a duplicate or unknown id, or an invalid decision/reason_code is scored as invalid.
5. The proxy has a per-case call budget resources["max_calls"] (= 4): at most that many DISTINCT vlm() generations per case (memoized repeats are free). Exceeding it makes the run invalid. The proxy also rejects vlm()/clip() calls made with a forged/mutated case id.

What You Have

• /app/data/router_grade.jsonl: a labeled development sample of cases (same schema as the hidden set), so you can dry-run locally for free.
• /app/data/case_schema.yaml: the field dictionary (case fields, reason codes, the injected proxy, and the scoring constants).
• /app/methods/main/solver.py: the weak baseline you must improve (answer-by-default with one crude similarity gate and a single fixed abstention reason).
• /app/selfcheck.py: a free self-check that runs your policy on the visible sample against the REAL graded proxy — the SAME Qwen2.5-VL-7B vlm() + BiomedCLIP clip() and the SAME scoring the hidden grader uses — by invoking the actual grading harness (evaluate.py) on /app/data/router_grade.jsonl. It prints the real selective_utility_pct + a per-reason breakdown. Because it is the real proxy on labeled cases, gains you make here TRANSFER to your graded held-out score (subject only to the usual visible→held-out generalization gap — the hidden cases differ). It needs a GPU (provided). Run python /app/selfcheck.py --quick 20 for a fast representative subset, or python /app/selfcheck.py for the full visible split.
• /app/mockproxy/: a deterministic, GPU-free MOCK of the full scoring pipeline (evaluate_mock.py over its own mock data), so you can exercise the entire evaluate mechanics (forbidden-import scan, id/decision validation, floor rescaling) end to end without a GPU: python /app/mockproxy/evaluate_mock.py --submission /app/methods/main --data /app/mockproxy --reward /tmp/r.json (the mock data uses router_grade_visible_mock.jsonl; copy it to router_grade.jsonl in a temp dir, or point --data at a dir containing it).

Each case the policy receives is:

{
  "id": "…",                  # salted id
  "question": "…",            # the question about the image
  "modality": "ct"|"mri"|"xray"|"unknown",   # coarse, inferred
  "organ": "brain"|"chest"|"abdomen"|"heart"|"other",
  "image_b64": "…",           # base64 image; the injected proxy reads the pixels, not you
}

The fixed proxy is handed to you in resources:

resources = {
  "vlm":  callable,    # vlm(case, sample_idx=0) -> str   an imperfect VQA answer (Qwen2.5-VL-7B);
                       #                                   greedy at sample_idx=0, resampled (seeded)
                       #                                   for sample_idx>0 to probe self-consistency
  "clip": callable,    # clip(case) -> float in [0,1]      cross-modal image<->question similarity
                       #                                   (BiomedCLIP cosine mapped to [0,1]); LOW
                       #                                   when the image does not support the question
  "max_calls": 4,
  "reason_codes": ["image_answerable","needs_ehr","mismatched_image","insufficient_evidence"],
}

The vlm is right on most answerable cases but not all, and it will confidently answer a case that is not actually answerable. The vlm emits a short 'blank' when the image is uniform / non-diagnostic (a targeted escape). The clip score tends to be low when the image does not support the question.

What You Submit

Edit /app/methods/main/solver.py and keep the fixed signature:

def route_cases(cases, resources, seed):
    """
    cases:     list[dict] of (image, question) cases as above (no labels).
    resources: {"vlm": callable, "clip": callable, "max_calls": int, "reason_codes": [...]}.
    seed:      int
    returns:   list[dict], one per case, each:
        {"id": str,
         "decision": "answer" | "abstain",
         "answer": str | None,            # your answer string when decision == "answer", else None
         "reason_code": one of resources["reason_codes"]}   # why you abstained (or "image_answerable")
    """

You may add helper modules (detectors.py, vote.py, …) beside solver.py. Approaches that fit here include gating on the cross-modal similarity, detecting questions that ask for record/history/lab facts the image cannot contain, and resampling the vlm to judge how self-consistent (confident) an answer is.

How It Is Judged

A sealed GPU verifier copies /app/methods/main/ into a clean sandbox, injects the fixed REAL proxy, and runs your policy on hidden cases:

python tests/evaluate.py --submission <path> --reward <out.json> --data tests/heldout

Each case contributes a deterministic per-case utility, summed over all cases (as computed by the graded tests/evaluate.py):

answerable case:
    answer, and the answer matches the gold answer  -> +1.0
    answer, but the answer is wrong                  -> -0.5
    abstain                                          ->  0.0
unanswerable case:
    answer (any answer)                              -> -2.0
    abstain with the correct reason_code             -> +0.5
    abstain with a wrong reason_code                 -> +0.1

Answers are compared after a canonicalizer (lowercasing, whitespace trimming, yes/no synonyms). The summed utility is put on a comparable scale by subtracting a floor and dividing by the headroom, where the floor U_floor = max(E[always-answer], best E[always-abstain-with-a-single-reason]) — both scored on the same set with the same proxy. The result is the raw selective_utility_pct (higher is better); a trivial constant policy lands at the floor (~0). The eval reward then maps that metric linearly (bounded_optimum, gamma=1.0): reward = clip(metric / 100, 0, 1), so the theoretical-perfect 100 -> 1 and the shipped baseline 0 -> 0. Invalid or crashing submissions score 0.

(selfcheck.py now runs the REAL proxy and the SAME utility table on the visible split, so its selective_utility_pct is directly comparable to your graded number — improve it, but don't overfit the visible cases, since the hidden set differs. The optional mockproxy/ fallback still uses a coarser GPU-free utility table — mechanics only; don't tune to the mock number.)

Common Pitfalls

• Answering every case (the inherited stop-gap): each unanswerable case then costs you -2.0, and there are many, so a confident answerer scores worse than simply abstaining on everything.
• Always abstaining, or always abstaining with the same reason: this is one of the trivial reference policies the floor is built from, so it earns essentially nothing — you must actually answer the answerable cases and pick the right reason on the rest.
• Treating every "the image does not support this question" case the same: a swapped-but-real image (mismatched_image) is not the same as a masked / non-diagnostic image (insufficient_evidence), and a record/history question (needs_ehr) usually has a perfectly good image — the cross-modal gate alone will not separate these. With the REAL proxy the signals are noisier than the mock: the gap that holds is driven by (1) an EHR-keyword detector recovering the needs_ehr bucket, (2) the vlm 'blank' escape flagging insufficient_evidence, and (3) the cross-modal gate for the rest.
• Trusting a single vlm sample: the proxy is wrong on a fraction of answerable cases; resampling (within max_calls) lets you denoise the answerable answers with a self-consistency vote.
• Spending more than resources["max_calls"] proxy calls per case, or overfitting the visible sample whose proxy instance differs from the hidden one.