Tasks/Computer Vision/Video Understanding

Switch-budgeted candidate routing

Route point tracks through candidates under a switch budget

robotap_switch_budget_candidate_routing_optimizationComputer VisionVideo Understanding
instruction.mdthis is what the agent is given

You inherit a weak persistent router over ten aligned candidate tracks for each queried point. Improve the general routing and visibility method to maximize mean per-video Average Jaccard while using at most four semantic-state changes after each point's query frame. The verifier re-runs your submitted method on a disjoint sealed split for final scoring.

Hard Constraints

  • Put your final method in /app/methods/main/predict.py and keep this exact entry point:

python def predict( query_points, candidate_tracks, occlusion_logits, expected_dist_logits, candidate_model_id, candidate_stage, ) -> tuple[numpy.ndarray, numpy.ndarray]: ...

  • Return (state_token, occluded). Both arrays must have shape [Q,T]; state_token must have a non-boolean integer dtype with values in 0..9, and occluded must have boolean dtype.
  • A state token is semantic: state_token = 5 * candidate_model_id + candidate_stage. It selects the corresponding candidate track at that query and frame.
  • For every query, the state-token sequence on frames strictly after the query frame may contain at most four changes. Frames at or before the query frame are not scored and do not count toward this switch budget.
  • Use only the six arrays passed to predict. Do not read visible labels, case identifiers, index order, filenames, verifier state, process state, credentials, or files outside /app/methods from prediction code.
  • Implement one reusable method. Do not encode visible answers, specialize to individual sequences or queries, infer hidden identities, or branch on array fingerprints, exact shapes, ordering, filenames, or case counts.
  • Treat candidate order as arbitrary and independently permuted for every query. Use candidate_model_id and candidate_stage as semantic metadata; never assume a fixed candidate-axis position.
  • Preserve query equivariance: reordering queries must only reorder outputs, and a query evaluated alone must receive the same prediction as it does in a batch.
  • Prediction must be deterministic for identical inputs. Do not access sealed data, another container, Docker, network resources, or the verifier.
  • Keep runtime and memory practical on CPU. Invalid output, more than four scored-frame switches for any query, timeout, excess resource use, or an exception fails the submission.

Run the complete visible evaluation and public contract audit with:

python /app/selfcheck.py --audit-contract

Use python /app/selfcheck.py --case-limit 3 --audit-contract only for quick smoke tests; the subset score is not comparable to the complete visible score.

What You Have

  • /app/methods/main/predict.py: a weak route that chooses one persistent native-confidence state after each query.
  • /app/data/visible/inputs/: 53 visible candidate-lattice cases.
  • /app/data/visible/labels/: visible labels used by selfcheck.py only. They are development targets, never prediction inputs.
  • /app/data/visible/index.json: shapes and integrity commitments for the visible cases.
  • /app/selfcheck.py: the visible Average Jaccard evaluator, switch-budget validator, and deterministic/query/candidate/singleton audit.

For one case, Q is the number of query tracks, T the number of frames, and K=10 the candidate count. Inputs have these shapes:

  • query_points: float32 [Q,3] in (query_frame, y, x) order;
  • candidate_tracks: float16 [Q,T,K,2] in (x,y) pixel coordinates;
  • occlusion_logits and expected_dist_logits: float16 [Q,T,K];
  • candidate_model_id and candidate_stage: uint8 [Q,K], containing each (model,stage) pair from {0,1} x {0,1,2,3,4} exactly once per query.

Candidate permutations and opaque case identifiers differ between visible and sealed cases. The candidates differ systematically in quality, so the visible evaluator provides an edit-run-measure signal without prescribing how routing or visibility should be modeled.

What You Submit

Harbor collects /app/methods and /app/experiment_log.md. Helper Python files and a compact learned artifact may be placed under /app/methods/main; prediction must not require installation, training, labels, network access, or writable caches at grading time.

Keep every graded helper or learned artifact beside predict.py under /app/methods/main. The staged bundle may contain at most 64 regular files and 64 MiB total; each file may be at most 32 MiB and must end in .py, .json, .joblib, .npz, or .npy.

Keep a concise experiment record in /app/experiment_log.md. Record complete visible scores, the change tested, and whether you kept or reverted it.

How It Is Judged

The verifier maps every predicted semantic state to its candidate track and computes Average Jaccard in first-query mode at pixel thresholds 1, 2, 4, 8, 16; higher is better. Frames at or before each query frame are excluded. Scores are computed per video and macro-averaged across videos.

On a disjoint sealed split, the verifier privately permutes candidate and query order, invokes the submitted method without labels or case identity, validates the four-switch budget, and checks deterministic repeat, semantic candidate-permutation invariance, query equivariance, and singleton equivalence. These are the same behaviors exercised by selfcheck.py --audit-contract.

Metric

mean video Average Jaccard over the 53 sealed videos · higher is better

AJ in first-query mode at 1/2/4/8/16 px, per video then macro-averaged; at most 4 route switches per query.

anchorvisible setheld-outreward
Bweak inherited persistent router0.590480.570870.00
Strusted human reference method0.661400.635390.30
Uperfect tracking1.000001.000001.00
normalisation
m <= B0
B < m <= S0.3 * (m - B) / (S - B)
S < m < U0.3 + 0.7 * (m - S) / (U - S)
m >= U1

m = this run's held-out metric  ·  B = weak inherited persistent router  ·  S = trusted human reference method  ·  U = perfect tracking

One AJ, averaged over videos then mapped once. The grader re-measures B and S on the sealed split and aborts if either drifts by 1e-6.

Rollouts

229 minwall clock
$47.11spend
77.3Mtokens
8versions, 6 kept
0.60 0.62 0.64 0.66 $0 $10 $20 $30 $40 cumulative spend on the run complete visible mean-video AJ, higher is better trusted human reference method · visible · 0.66140 v1 v2 v3 v4 v5 v6 v7 v8
keptrevertedno scoreturning point
  1. v1Inherited persistent native-confidence route, unchanged0.5904841 min · $0.26
  2. v2Fixed semantic state 9; visible when occ_logit + dist_logit < -1.0Hold one fixed (model,stage) slot for the whole video instead of following per-frame native confidence.0.6296396 min · $0.68
  3. v3Boosted localization reward routed by exact DP over 4 switches; utility visibilitySpend the switch budget: score each candidate with a learned reward, then pick the route by exact DP.0.66027767 min · $8.76
  4. v4Route-aware stacked calibrator, 0.65/0.325/0.025 blend; two-state temporal DPDecode visibility as a temporal path over calibrated utility rather than a per-frame threshold.0.6668598 min · $15.83
  5. v5Refit reward with per-frame weights 0.1 + candidate utility range0.667902112 min · $20.33
  6. v6Second utility model from an independent subsample, blended 25% into utility0.667865164 min · $38.38
  7. v7Asymmetric visibility transitions: 0.05 to enter occlusion, 0.50 to return0.667979174 min · $39.12
  8. v8Meta-model sampling cap raised from 15k to 30k frames per video0.667726196 min · $41.73

v6 and v8 were snapshotted then reverted, to v5 and v7. Five-fold OOF picked v7 at 0.65254. 229 min, $47.11, 77.3M tokens.

On the hidden set

held-out metricreward
weak inherited persistent router0.570870.00
trusted human reference method0.635390.30
perfect tracking1.000001.00
this run0.635240.2993
59 minwall clock
$2.64spend
2.4Mtokens
6versions, 5 kept
0.585 0.600 0.615 0.630 0.645 0.660 $0 $0.8 $1.5 $2.2 cumulative spend on the run complete visible mean-video AJ, higher is better trusted human reference method · visible · 0.66140 v0 v1 v2 v3 v4 v5
keptrevertedno scoreturning point
  1. v0Inherited persistent native-confidence route, unchanged0.590484$0.44
  2. v1Persistent semantic token 9; visible when occ + 1.55*dist >= -2.29Hold one (model,stage) slot for the whole video instead of following per-frame confidence.0.630324$0.88
  3. v280-tree per-query semantic ranker: confidence, consensus, motion, refinementLearn which slot to hold per query from label-free features, fitted on cases 1-43 only.0.647242$1.32
  4. v3Median-3 temporal smoothing of the visibility score; threshold -2.310.647353$1.76
  5. v4Frame-level visibility regressor over logits, consensus, motion, route marginDecide visibility per frame from a learned score rather than from one fixed linear cut.0.652028$2.20
  6. v5Retrain the frozen router and visibility heads on all 53 visible cases0.653475$2.64

Six snapshots, but the codex transcript records no tool calls, so the timeline is empty and these points come from the log. 59 min, $2.64.

On the hidden set

held-out metricreward
weak inherited persistent router0.570870.00
trusted human reference method0.635390.30
perfect tracking1.000001.00
this run0.630810.2787
260 minwall clock
$23.43spend
35.0Mtokens
6versions, 6 kept
0.63 0.64 0.65 0.66 0.67 $0 $5 $10 $15 $20 cumulative spend on the run complete visible mean-video AJ, higher is better trusted human reference method · visible · 0.66140 v1 v2 v3 v4 v5 v6
keptrevertedno scoreturning point
  1. v1Always semantic token 9; occluded when occ_logit > 1.5 or dist_logit > -1Candidate quality is ordered by semantic token at every time gap, so hold token 9 rather than route.0.63050234 min · $3.97
  2. v2Visibility from 5 boosted heads over 97 label-free per-query featuresCall a frame visible when the expected AJ gain is positive, not when a logit clears a cut.0.670787127 min · $10.65
  3. v3NaN-free order statistics; T=1..3 and query-on-last-frame edge cases fixed0.670834160 min · $13.64
  4. v4Drop log_frames and log_span_x: near-constant per video, so identity-carryingRemoving the two features that could only encode video identity raised grouped-CV AJ to 0.6410.0.670524224 min · $18.98
  5. v5Comment and readability cleanup; predictions byte-identical to v40.670524242 min · $20.57
  6. v6Final artifact retrained at 413k rows, the density the CV models used0.670491259 min · $23.27

v0 is the inherited router at 0.590484; v4a and v4b were scored on CV only, never snapshotted. First snapshot at 34 min and $3.97; $23.43 over 4h20.

On the hidden set

held-out metricreward
weak inherited persistent router0.570870.00
trusted human reference method0.635390.30
perfect tracking1.000001.00
this run0.629800.274