Tasks/Life Sciences/Protein Structure & Function

Retrieve protein surfaces by shape

Retrieve protein surfaces by shape descriptor

shrec2021 Life Sciences Protein Structure & Function
instruction.mdthis is what the agent is given

You are given a collection of real, structure-derived protein surface meshes, each with per-vertex physicochemical properties, and a deliberately weak starting descriptor. Your job is to design a shape-descriptor method that, given one query protein surface, lets a fixed distance-based ranking retrieve the other surfaces belonging to the same real structural class; your method is re-run unchanged on a sealed held-out set of query surfaces you never see, ranked against the same visible collection, to compute your score.

Hard Constraints

  • Submit a method: methods/main/solver.py defining exactly describe(shape) -> 1-D numpy array (a fixed-length vector; the length may be anything but must be the same on every call within a run). The judge re-runs THIS file, unmodified, on the sealed held-out queries.
  • shape is {"vertices": (P, 3) float64, "properties": (P, 3) float64} -- a real point sample of one protein surface (P real mesh vertices and their real physicochemical properties, same row order). See environment/shape_io.py.
  • The judge, not you, ranks collection items for a query by ascending Euclidean (L2) distance between descriptor vectors -- describe is the only thing you write.
  • Each describe call runs in its own process under a 10-second wall-clock budget (measured outside your process) for a query call, and the whole-collection batch call has a generous shared budget; crashed, over-budget, or inconsistent-length outputs score as the worst case for that call.
  • CPU only, no network; runtime provides Python 3.11 + numpy.
  • Only methods/main/solver.py is graded; do not rely on modifying shape_io.py, retrieval.py, selfcheck.py, or data/.

What You Have

  • data/collection/*.npz -- the visible collection (real protein surfaces) every query is ranked against, plus data/collection_labels.json (each collection item's real structural class). This is the intended primary source of labeled supervision if your method wants any (e.g. fitting a projection) -- 1059 real items across all 65 real structural classes of the evaluation pool. Class sizes are not uniform (this mirrors the real, imbalanced source distribution): from 1 to 109 items per class, averaging about 16.
  • data/visible_queries/*.npz -- a visible slice of real query surfaces, drawn from the same class distribution as the judged queries, split into two label files:
  • data/visible_query_labels_train.json (66 ids) -- labels you may freely use as extra fitting signal. Because classes are imbalanced, not every class has a train example (some small classes contribute 0).
  • data/visible_query_labels_dev.json (112 ids) -- labels used only by selfcheck.py to report an out-of-sample proxy score (again, not every class necessarily has a dev example). Do not fit anything (train a classifier, tune a projection, hard-code per-id logic, etc.) against these ids or their labels -- doing so makes selfcheck.py's number meaningless for predicting your judged score, the same way it would in any real train/validation split.
  • methods/main/solver.py -- a deliberately naive starting descriptor. Replace it freely.
  • python3 selfcheck.py -- free and unlimited: computes mean Average Precision (mAP, higher is better) of your current describe on the visible dev queries against the collection. Because the dev ids are never used above, this is a genuine out-of-sample estimate of how your method generalizes -- unlike training and evaluating on the exact same queries, which would only measure memorization. This mirrors the only feedback you get in practice; the judged queries are held back.

What You Submit

Leave your best methods/main/solver.py in place. There is no submit step and no per-attempt feedback on the judged queries: whatever sits in methods/main/ at the end is what the judge re-runs and scores.

How It Is Judged

The judge draws queries from a sealed held-out split of the same real query pool (same collection, same labeling protocol, never seen during development), runs your describe once per collection item and once per held-out query, ranks each query's retrieved list by ascending descriptor distance, and scores mean Average Precision (mAP) against the real structural-class ground truth -- higher is better. The judge recomputes mAP itself from your descriptors; it does not read any number you report, and you receive no feedback from that run.