Tasks/Computer Vision/Video Understanding

Candidate-lattice point-track fusion

Select and fuse point-track candidates over a lattice

robotap_candidate_lattice_optimization Computer Vision Video Understanding
instruction.mdthis is what the agent is given

You inherit a weak confidence-only selector for ten aligned candidates per point track. Improve the general prediction method to maximize mean video Average Jaccard; your submission is re-run on disjoint sealed videos for scoring.

Hard Constraints

  • Put your final method in /app/methods/main/predict.py and keep this exact entry point:

python def predict( query_points, candidate_tracks, occlusion_logits, expected_dist_logits, candidate_model_id, candidate_stage, ) -> tuple[numpy.ndarray, numpy.ndarray]: ...

  • Return (tracks, occluded), where tracks is a finite numeric array of shape [Q,T,2] and occluded is a boolean array of shape [Q,T].
  • Use only the six arrays passed to predict. Do not read visible labels, case identifiers, manifest order, filenames, verifier state, process state, credentials, or files outside /app/methods from prediction code.
  • Implement one reusable method. Do not encode visible answers, specialize to individual videos or queries, infer hidden identities, or branch on array fingerprints, exact shapes, ordering, filenames, or case counts.
  • Treat candidate order as arbitrary and independently permuted for every query. Use candidate_model_id and candidate_stage only as public semantic metadata; never assume a fixed candidate-axis position.
  • Preserve query equivariance: reordering queries must only reorder outputs, and a query evaluated alone must receive the same prediction as it does in a batch, up to ordinary floating-point tolerance.
  • Do not attempt to access sealed data, another container, Docker, network resources, or the verifier. The sealed verifier is offline and runs your method in a restricted child process.
  • Keep runtime and memory practical on CPU. Invalid output, timeout, excess resource use, or an exception fails the submission.

Run the complete visible evaluation with:

python /app/selfcheck.py

Use python /app/selfcheck.py --case-limit 3 only for quick smoke tests; the subset score is not comparable to the complete visible score.

What You Have

  • /app/methods/main/predict.py: the weak max-frame-confidence baseline.
  • /app/data/visible/inputs/: 53 sanitized visible candidate-lattice cases.
  • /app/data/visible/labels/: visible labels used by selfcheck.py only. They are development targets, never prediction inputs.
  • /app/data/visible/MANIFEST.json: shapes and integrity hashes for the visible cases.
  • /app/selfcheck.py: the visible Average Jaccard evaluator.

For one case, Q is the number of query tracks, T the number of frames, and K=10 the candidate count. Inputs have these shapes:

  • query_points: float32 [Q,3] in (query_frame, y, x) order;
  • candidate_tracks: float16 [Q,T,K,2] in (x,y) pixel coordinates;
  • occlusion_logits and expected_dist_logits: float16 [Q,T,K];
  • candidate_model_id and candidate_stage: uint8 [Q,K], containing each (model,stage) pair from {0,1} x {0,1,2,3,4} exactly once per query.

The public metadata distinguishes tracker family and refinement stage. Candidate permutations and opaque case identifiers deliberately change across splits. Use the visible evaluator to study how confidence, spatial and temporal agreement, and visibility decisions affect generalization.

What You Submit

Harbor collects /app/methods/main/predict.py and /app/experiment_log.md. Helper Python files and a compact learned artifact may be placed under /app/methods/main; prediction must be deterministic for identical inputs and must not require installation, training, labels, network access, or writable caches at grading time.

Keep every graded helper or learned artifact beside predict.py under /app/methods/main. The staged bundle may contain at most 64 regular files and 64 MiB total; each file may be at most 32 MiB and must end in .py, .json, .joblib, .npz, or .npy.

Keep a concise experiment record in /app/experiment_log.md. Record complete visible scores, the change tested, and whether you kept or reverted it.

How It Is Judged

The metric is Average Jaccard in first-query mode at pixel thresholds 1, 2, 4, 8, 16; higher is better. Frames at or before each query frame are excluded. Scores are computed per video and macro-averaged across videos.

The verifier loads the same six-field public schema on disjoint sealed videos, privately permutes candidate and query order, invokes the submitted method without labels or case identity, validates the output, and checks deterministic, singleton, and permutation behavior.

Metric

mean video Average Jaccard over the sealed videos · higher is better

AJ in first-query mode at 1/2/4/8/16 px, computed per video and macro-averaged over the 53 sealed videos.

anchorvisible setheld-outreward
Bweak confidence-only selector0.591010.567390.00
Strusted human reference method0.670590.629020.30
Uperfect tracking1.000001.000001.00
normalisation
m <= B0
B < m <= S0.3 * (m - B) / (S - B)
S < m < U0.3 + 0.7 * (m - S) / (U - S)
m >= U1

m = this run's held-out metric  ·  B = weak confidence-only selector  ·  S = trusted human reference method  ·  U = perfect tracking

AJ is averaged over videos first, then mapped once. The grader re-measures B and S on the sealed set and aborts if they drift by 1e-6.

Rollouts

276 minwall clock
$33.66spend
49.8Mtokens
9versions, 9 kept
0.60 0.62 0.64 0.66 $0 $7.5 $15 $22 $30 cumulative spend on the run complete visible mean-video AJ, higher is better trusted human reference method · visible · 0.67059 v0 v1 v2 v3 v4 v5 v6 v7 v8
keptrevertedno scoreturning point
  1. v0Inherited baseline: per-frame argmax of (1-sig(occ))*(1-sig(edl))0.5910140 min · $0.20
  2. v1Learned candidate-quality head, geometric median, learned visibility headFuse all ten candidates under two learned heads over permutation-invariant features instead of picking one by confidence.0.64940687 min · $10.26
  3. v2Temporal second pass over the pass-1 visibility score, nested stacked trainingRebuilt the evaluation so no head sees a model trained on its own test fold; the honest gain of the second pass fell to +0.0015.0.652297141 min · $16.34
  4. v3One candidate head per public (model,stage) token; pairwise candidate geometry0.654824173 min · $20.14
  5. v4Slot-keyed node features: per token, visibility, distance to fused point, score0.65592194 min · $22.00
  6. v5Per-candidate path features and slot-keyed temporal context in the node head0.656566206 min · $23.32
  7. v6Query anchoring: shift each candidate by its own offset at the query frameThe query point is the true position at that frame and the trackers miss it by 0.4 px, so the offset is a free correction.0.657575233 min · $26.51
  8. v7More training rows per head; explicit-lag node features removed0.657811271 min · $31.02
  9. v8Documentation only; numerics identical to v70.657811276 min · $32.88

Every snapshot was kept. 13 rejected ideas - adaptive tau, tree heads, motion lag, smoothing - were measured but never snapshotted. $33.66 over 4h36.

On the hidden set

held-out metricreward
weak confidence-only selector0.567390.00
trusted human reference method0.629020.30
perfect tracking1.000001.00
this run0.624550.2782
130 minwall clock
$35.07spend
57.8Mtokens
11versions, 11 kept
0.60 0.62 0.64 0.66 $0 $7.5 $15 $22 $30 cumulative spend on the run complete visible mean-video AJ, higher is better trusted human reference method · visible · 0.67059 v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10
keptrevertedno scoreturning point
  1. v0Inherited baseline: per-frame max confidence over the ten candidates0.5910141 min · $0.22
  2. v1Take the semantic (model=1,stage=4) track with its TAPIR-style visibilityChoose a slot by its public model/stage metadata instead of per-frame confidence; every fold improved.0.620653 min · $0.49
  3. v2Exact query-coordinate translation; calibrated occ + 2.25*dist < -2.85 cutoffAnchor each track on the one coordinate known exactly, and fit the visibility cutoff instead of fixing it.0.634479 min · $1.24
  4. v3Balanced 50-feature histogram-boosting fusion over the candidate latticeLearn model-0-vs-model-1 coordinate advantage and anchored threshold quality rather than rank by rule.0.65547423 min · $2.92
  5. v4Eight per-query candidate confidence mean/std visibility features, radius-1 smoothing0.6643938 min · $6.57
  6. v5Nine advantage regressors vs anchored model-1/stage-4; semantic wins above +0.0050.66756853 min · $9.89
  7. v6Blend emitted coordinates 75% toward their radius-1 temporal median0.66865766 min · $13.45
  8. v7Extra 30% blend toward the radius-2 median after the primary filter0.66888674 min · $15.74
  9. v8Visibility cutoff aligned from 0.365 to the joint-OOF optimum 0.36750.66901381 min · $17.99
  10. v9Three visibility-only boundary cues: query, track and outside-canvas distance0.670402110 min · $27.34
  11. v10Packaging only: generated .pyc cache removed, predictions unchanged0.670402130 min · $34.81

Every snapshot was kept. Viterbi decoding, Extra Trees, wider medians and absolute x/y were measured and rejected on OOF. $35.07 over 2h10.

On the hidden set

held-out metricreward
weak confidence-only selector0.567390.00
trusted human reference method0.629020.30
perfect tracking1.000001.00
this run0.621830.265
33 minwall clock
$3.23spend
4.7Mtokens
10versions, 9 kept
0.60 0.62 0.64 0.66 $0 $0.8 $1.5 $2.2 $3 cumulative spend on the run complete visible mean-video AJ, higher is better trusted human reference method · visible · 0.67059 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10
keptrevertedno scoreturning point
  1. v1Inherited baseline: framewise max of (1-sig(occ))*(1-sig(dist)), tau 0.50.591014$0.32
  2. v2Take the semantic (model=1,stage=4) track; visibility sig(-(occ+dist)) > 0.70Pick one tracker slot by its public metadata rather than by per-frame confidence. Largest gain of the run, and it stuck.0.629602$0.65
  3. v3Tune-only 61-feature linear reliability model over the kept coordinatesDecide visibility from a learned score over semantic logits, confidence and spatial disagreement, not a fixed threshold.0.634135$0.97
  4. v4Query-local 3-frame median on the linear reliability score0.63424$1.29
  5. v5Small translation-equivariant residual from the nine other semantic tracks0.634712$1.61
  6. v6Weaker calibration C=10 and an OOF-selected 0.55 probability threshold0.631946$1.94
  7. v7C=30 targeting 8 px correctness with a 5-frame median0.631701$2.26
  8. v8Translate each fused track so its query-frame point matches the query pointAnchor the trajectory on the one coordinate known exactly; median raw offset 0.290 px and the sweep improved monotonically.0.636296$2.58
  9. v9Fixed a shadowed stale bias constant; refit residual weights after anchoring0.640066$2.90
  10. v10Refit calibration and residual model on all 53 visible videos0.640028$3.23

No version was snapshotted, so these ten points come from the experiment log alone. It stopped after 33 minutes and $3.23, a tenth of the other run.

On the hidden set

held-out metricreward
weak confidence-only selector0.567390.00
trusted human reference method0.629020.30
perfect tracking1.000001.00
this run0.612550.2198