Design a label-free predictor that scores how much a pretrained action-conditioned world model will hallucinate on a given rollout, starting from a weak constant baseline. Maximize rank correlation with the true rollout degradation; your submitted predictor is re-run on sealed sequences from held-out tasks for scoring.
Hard Constraints
Modify only /app/methods/main/predictor.py as the final submission.
Keep the public function score_sequence(bundle, frames, actions) -> float returning one finite scalar (higher = more expected hallucination). bundle exposes the frozen tokenizer/dynamics models and sampling utilities.
Label-free means: the predictor must not use the ground-truth future frames (frames beyond the context prefix are never passed to it) and must not fine-tune or modify the model weights.
Scoring the sealed set must finish within 60 minutes on one GPU; budget roughly <=2 seconds of model compute per sequence.
Any files your predictor needs must live under /app/methods/main/.
What You Have
The MMBench2 world model (frozen base checkpoints at /opt/ckpts/base), the repo at /opt/mmbench2 (on PYTHONPATH), and the wm_bundle adapter (Bundle, encode/decode/rollout, rollout_delta_psnr) used by both selfcheck and the sealed grader.
/app/data/visible/: 24-frame ground-truth sequences with actions from 12 training tasks (.npz: frames, actions, task). /app/selfcheck.py computes each sequence's true rollout ΔPSNR with the frozen model on first run and caches it, then reports your predictor's Spearman correlation — so you can measure exactly the sealed metric locally.
/app/methods/main/predictor.py: a weak baseline returning a constant.
The paper's three predictors (tokenizer round-trip residual, flow instability, inter-seed variance) as reference reading in the installed uncertainty.py — reusing or improving on their ideas is legal.
What You Submit
Submit /app/methods/main/predictor.py. The research problem is uncertainty design: find model-internal, label-free signals (latent drift, denoiser disagreement, seed variance, action sensitivity, or new ones) that predict when rollouts will diverge from reality, and combine them so the ranking transfers to unseen tasks.
How It Is Judged
The verifier imports your predictor in a restricted process, runs it on sealed sequences drawn from 12 tasks disjoint from the visible ones (the paper's held-out-sequence protocol), computes the true rollout ΔPSNR with the ground-truth frames it alone holds, and reports the Spearman rank correlation between your scores and the true degradation. Higher rank correlation is better; a predictor with no correlation to the true degradation scores near zero, and a perfect ranking is the theoretical ceiling.
Metric
Spearman rho vs true rollout dPSNR, 300 sealed sequences · higher is better
Predictor score per sequence ranked against negated true dPSNR, which the grader computes from hidden future frames.
anchor
held-out
reward
B
uninformative predictor
0.000
0.00
R
paper single predictor u_s (2-seed)
0.571
0.30
S
expert predictor on this sealed set
0.854
0.60
U
perfect ranking (theoretical max)
1.000
1.00
normalisation
m <= B
0
B < m <= R
0.3 * (m - B) / (R - B)
R < m <= S
0.3 + 0.3 * (m - R) / (S - R)
S < m <= U
0.6 + 0.4 * (m - S) / (U - S)
m > U
1
m = this run's held-out metric · B = uninformative predictor · R = paper single predictor u_s (2-seed) · S = expert predictor on this sealed set · U = perfect ranking (theoretical max)
B/R/S/U = 0, 0.571, 0.854, 1.0. rho is already bounded, so linear, no warp. One pooled rho over all 300 sealed sequences, mapped once.
Rollouts
113 minwall clock
$33.94spend
54.3Mtokens
15versions, 13 kept
keptrevertedno scoreturning point
v0Inherited constant predictor, snapshot onlyconstant, rho undefined3 min · $0.36
v1Context replay: predict frames 4-7 from the first half, graded PSNR ratioTreat the back half of the context as a legal validation horizon, scored by the graded formula, so no future frames are needed.0.7794 min · $0.66
v2Replay lengthened to frames 2-7; steadier on holdouts than the shorter split0.826410 min · $1.73
v3Worst per-frame replay degradation instead of the aggregate over frames0.842518 min · $3.00
v4Adds two-seed disagreement over the 16-step future, normalised by tokenizer errorAdd an epistemic term: two sampled futures agree where the model is at home and diverge where it is not, on the graded horizon.0.884521 min · $3.59
v5Subtracts replay latent mismatch to discount perceptually weak deviations0.897929 min · $5.51
v6Mean pairwise disagreement over three future seeds instead of two0.902833 min · $6.44
v7Worst replay degradation over two context cut points (2 and 3 frames)0.906339 min · $8.15
v8Adds the log ratio of generated to context pixel contrast, no extra compute0.910247 min · $10.66
v9Disagreement anchored on grader seed 0; adds action-ablation latent sensitivity0.914153 min · $12.51
v10Infers each task's active action width from the padding, masks the rest awayActions are zero-padded, so unmasked rollouts feed the model inputs no task issues. The recovered width is exact on 298/300.0.91768 min · $19.85
v11Third reference seed decoded in the same batch as the others0.91779 min · $24.98
v12Decodes replay pairs and seeds as independent batch elements0.917887 min · $28.62
v13Strict B=1 serial decode, the decoder path the trusted metric uses0.917895 min · $32.34
v14Seed-0 disagreement averaged against three independently decoded futures0.9205104 min · $32.98
Fifteen snapshots, 113 min, $34. The last four chase one gap: batching futures lets decoder attention cross seeds; v11 and v12 missed, v13 fixed it.
v1Log mean frame-to-frame MSE over the 8 context frames, no model compute0.69432 min · $3.25
v2Teacher-forced one-step dPSNR inside the visible contextApply the graded quantity where it is legal: decode each context frame from the clean past, take the grader's PSNR ratio.0.80645 min · $4.82
v3Seed disagreement, context distance, teacher-forced error, least-squares weightsPut every term in dB over tokenizer error on the same context, so they estimate the graded ratio instead of just correlating.0.897152 min · $12.69
v4Same three signals, rounded weights (2.4, 2.4, 0.5), horizon from the action tensor0.8986246 min · $21.79
v5Weights refit on raw plus within-task-demeaned targets: (2.4, 1.8, 0.45)0.8988250 min · $23.11
Five snapshots, 252 min, $24. Most of the money sits in v2-v3: caching true dPSNR labels for the 300 visible sequences, then sweeping features.
On the hidden set
held-out metric
reward
uninformative predictor
0.000
0.00
paper single predictor u_s (2-seed)
0.571
0.30
expert predictor on this sealed set
0.854
0.60
perfect ranking (theoretical max)
1.000
1.00
this run
0.8411
0.5863
Re-graded after a verifier fix; this is the corrected reward.
v1Last-step 3x8-seed latent variance, tokenizer residual, pixel motion, teacher-forcedCombine multi-seed rollout disagreement with tokenizer reconstruction error, fit on 9 tasks with 3 held out.0.8127
v2Seed variance averaged over the whole 16-step horizon; weights refit on 9 tune tasksSpread the disagreement estimate over the whole 16-step window the grader averages dPSNR over.0.8338
v3Adds free latent motion ratio, rollout over context; tok weight -1.2, pix 0.60.8443
v4Adds image-space inter-seed variance of the last two decoded framesdPSNR is measured on decoded pixels, so take the seed variance there too, not only in latent space.0.8533
v5Tokenizer residual weight -1.2 to -1.4, no new compute0.856
v0Inherited constant predictor, snapshot onlyconstant, rho undefined1 min · $0.29
v1RMS pixel displacement between first and last context frame, no model computeAsk how much the scene physically moves in the visible context: more motion, worse rollout, at zero model cost.0.539674 min · $19.90
v2Masked two-seed rollout disagreement and drift, over context tokenizer errorSample two futures from the frozen model and measure their divergence in units of the tokenizer's own reconstruction error.0.7796128 min · $36.19
v3Adds a 2-frame-context/6-frame prefix backtest at weight 0.100.7603178 min · $54.06
v4Adds -0.40*log(std of four one-step latent forecast errors)Fit extra log-corrections by leave-one-task-out over the visible tasks, keeping only coefficients stable across folds.0.7522264 min · $86.14
No trajectory curve: this run left one comparable self-check measurement, so there is nothing to plot against spend. The versions and what each one changed are below.
v1Batched 8-seed rollout, L1 mean deviation per step, exponential step weight a=0.20Read hallucination straight off how far eight noise seeds drift apart, weighting late steps more.0.780946 min · $0.62
One snapshot, 50 min, $0.67. Seed count and alpha were swept on all 300 visible labels directly, with no task held out.