RSI-Bench
Tasks/Life Sciences/Protein Structure & Function

Unseen-Epitope TCR Specificity

Match TCRs to epitopes never seen in training

tcr_epitope_unseen_specificity Life Sciences Protein Structure & Function
instruction.mdthis is what the agent is given

Improve the inherited ESM-2 retrieval baseline for TCR–epitope binding specificity. Develop only on the frozen visible split; your final source is re-run once on sealed, epitope-disjoint pairs and scored by Macro AUC0.1.

Hard Constraints

  • Work only inside /app/methods/main. The required entry point is score_pairs(item).
  • Use the provided ESM-2 model only for forward inference. Do not fine-tune it or train a new learned model on the visible labels.
  • Do not retrieve known binders for evaluation epitopes from external databases or services.
  • Return scores computed from the supplied train/test tables; do not submit precomputed final scores.
  • Stay within one GPU, 8 CPUs, 32 GB host memory, and the task time limit.

What You Have

  • data/visible/train_pairs.tsv: labeled pairs from 59 training epitopes.
  • data/visible/test_pairs.tsv: unlabeled pairs from 20 epitopes absent from the training set.
  • data/visible/test_labels.tsv: visible truth used only by the local self-check.
  • methods/main/solver.py: the inherited baseline. It embeds CDR3β with ESM-2 and uses maximum cosine similarity to a positive training TCR; it intentionally ignores the candidate epitope.
  • methods/main/tcr_utils.py: shared loading, embedding, and retrieval utilities.
  • /opt/esm2: a local, pinned protein-language-model snapshot.
  • python selfcheck.py: the canonical complete-visible evaluation. Every comparable experiment must use this exact command and all 20 visible epitopes.

What You Submit

Keep a source-only implementation in methods/main/. solver.py must expose:

def score_pairs(item: dict) -> dict:
    # item contains train_pairs_tsv, test_pairs_tsv, and plm_model_dir.
    # Return one finite score per test row, in row order.
    return {"scores": [...]}

You may add Python helper modules in the same directory. The sealed verifier rejects other file types, symlinks, malformed output, non-finite values, and the wrong score count.

How It Is Judged

The verifier runs your frozen source as an unprivileged process with no network and no access to labels. Its epitopes are absent from training and visible evaluation. For each epitope it computes standardized partial AUROC with max_fpr=0.1, then macro-averages the values. Random ranking is approximately 0.5 and higher is better. Every epitope must be valid; no failed stratum is skipped. Only the final aggregate grade is produced.

Metric

macro standardized partial AUROC at FPR<=0.1, sealed split · higher is better

Per-epitope standardized pAUC with max_fpr=0.1, macro-averaged over the 14 sealed unseen epitopes

anchorvisible setheld-outreward
Bshipped starter solver0.4989580.4974360.00
Rbest known solution (codex v23)0.5350040.5472040.30
Umetric maximum1.0000001.0000001.00
normalisation
m <= B0
B < m <= R0.3 * (m - B) / (R - B)
R < m <= U0.3 + 0.7 * (m - R) / (U - R)
m > U1.0 (cap)

m = this run's held-out metric  ·  B = shipped starter solver  ·  R = best known solution (codex v23)  ·  U = metric maximum

Linear in every segment, no transform. One sealed suite, so the macro metric is mapped once, not per case. B/R/U are the three anchors.

Rollouts

173 minwall clock
$57.44spend
76.0Mtokens
72versions, 22 kept
0.502 0.510 0.517 0.525 0.532 $0 $15 $30 $45 cumulative spend on the run visible self-check macro AUC0.1, higher is better best known solution · visible · 0.535004 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 v25 v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41 v42 v43 v44 v45 v46 v47 v48 v49 v50 v51 v52 v53 v54 v55 v56 v57 v58 v59 v60 v61 v62 v63 v64 v65 v66 v67 v68 v69 v70 v71 v72
keptrevertedno scoreturning point
  1. v1Inherited ESM-2 nearest-positive cosine baseline, unchanged0.4989585 min · $0.47
  2. v2Content-addressed embedding cache; scores identical, experiments faster0.4989587 min · $0.88
  3. v3Swept global positive-vs-negative ESM retrieval, centering, k=1..640.49895810 min · $1.21
  4. v4Conflict-filtered LOEO diagnostic; the contrastive candidate reversed on itsolver unchanged11 min · $1.45
  5. v5Epitope-conditioned and MHC-aware sweep; aborted on evaluator overhead0.49895818 min · $2.53
  6. v6Fast sklearn-equivalent AUC0.1 evaluator, self-neighbour exclusion fixedsolver unchanged19 min · $2.96
  7. v7Fixed the hard-MHC source fallback; reran the staged epitope/allele sweepsolver unchanged21 min · $3.28
  8. v8ESM TCR distance vs endpoint-identity and character-motif similaritiessolver unchanged23 min · $3.70
  9. v9Within-epitope rank ensemble, greedily selected on 39 training epitopessolver unchanged26 min · $4.09
  10. v10Small V-gene and CDR3-length interaction terms on char-motif retrievalsolver unchanged28 min · $4.49
  11. v11Production: MHC-aware ESM epitope mapping plus char3-4 mean-4 retrievalMap the unseen epitope to a compatible training epitope and retrieve inside it, instead of ignoring the epitope.0.51611628 min · $4.80
  12. v12Source-specific pos-vs-neg margins, kNN fractions, KDE correctionssolver unchanged30 min · $5.30
  13. v13Edit, BLOSUM, Atchley and one-hot peptide encodings for source mappingsolver unchanged32 min · $5.86
  14. v14Rank ensembles of size 1-4 over seven distinct source mappingssolver unchanged34 min · $6.34
  15. v1528 CDR3 trim/endpoint variants, Atchley/BLOSUM encodings, 42 n-gramssolver unchanged36 min · $7.20
  16. v16100-geometry Atchley sweep: trim depth, RBF kernels, centered paddingsolver unchanged39 min · $8.11
  17. v17Equal-weight subsets and 1:3 blends of the surviving CDR3 encodingssolver unchanged40 min · $8.55
  18. v18Production: centered trim2/width10 Atchley nearest-binder scoring0.51719741 min · $8.98
  19. v19ESM layer, pooling, centering and top-PC sweep for source mappingsolver unchanged44 min · $9.69
  20. v20Production: epitope mapping moved to penultimate-layer ESM means0.51750645 min · $10.26
  21. v21Label-free within-epitope calibration by V, length and V+length stratasolver unchanged47 min · $10.91
  22. v2220 deterministic exact 4:1 matched LOEO resamples of that calibrationsolver unchanged50 min · $12.36
  23. v23Pairwise V/J match bonuses and length penalties around Atchley retrievalsolver unchanged53 min · $14.58
  24. v24Top-2..59 compatible source epitopes under five aggregation rulessolver unchanged57 min · $15.01
  25. v25Rank blends of 5-75% multi-source residual scorers into productionsolver unchanged58 min · $15.31
  26. v26Shift-tolerant Atchley alignments over trim, width, shift and penaltysolver unchanged60 min · $15.88
  27. v27TCRdist-style single-gap CDR3 retrieval, identity/BLOSUM/Atchley costssolver unchanged64 min · $16.79
  28. v28Robustness audit of the TCRdist candidate; hit a strict min-positive guardaudit crashed65 min · $17.36
  29. v29Audit finished; mean-2 lost on matched resamples, mean-3 looked strongersolver unchanged66 min · $17.76
  30. v30Full robustness audit of mean-3 TCRdist; accepted on train and matchedsolver unchanged67 min · $18.17
  31. v31Production: gapped BLOSUM TCRdist averaged over three nearest bindersCompare CDR3s by gapped BLOSUM alignment rather than by embedding distance, and average the three nearest binders.0.52540670 min · $18.73
  32. v32Uniform, trimmed, geometric and soft-min neighbour aggregation, k=1..12solver unchanged71 min · $19.27
  33. v33Audited geometric k=4 decay 0.75 against mean-3; acceptedsolver unchanged73 min · $19.80
  34. v34Production: four nearest binders weighted geometrically by 0.750.52558373 min · $20.18
  35. v35Re-swept 70 ESM and classical peptide source mappings under TCRdist0.52558378 min · $21.69
  36. v36Audited the four changed source assignments; residue pooling acceptedsolver unchanged80 min · $22.15
  37. v37Production: peptide embeddings become residue-only penultimate means0.52558381 min · $22.56
  38. v38Source-specific negative TCRdist margins and joint pos/neg kNN fractionssolver unchanged82 min · $23.18
  39. v39Audited the 16- and 32-negative margins; both reversed when matchedsolver unchanged83 min · $24.01
  40. v4017 substitution transforms, gap placements and six gap penalties0.52558387 min · $25.51
  41. v41Per-MHC source remapping and exact/two-digit/locus reference filtering0.52558390 min · $26.38
  42. v42Audited coarse reference-level allele filtering; accepted with fallbacksolver unchanged91 min · $27.07
  43. v43Production: positive references filtered to allele-compatible binders0.52568992 min · $27.60
  44. v4413 CDR3 trim pairs, six gap penalties and position-weight profiles0.52568994 min · $28.75
  45. v45Pairwise V/J allele, gene and family mismatch penalties inside TCRdist0.52568996 min · $30.86
  46. v46Label-free CDR3 density inside each unlabelled candidate-epitope cohort0.525689100 min · $31.57
  47. v47Audited the 10% cohort-density blend; all 160 matched audits positivesolver unchanged102 min · $32.29
  48. v48Production: 10% cohort TCRdist-density rank plus 90% retrieval rankTake a label-free signal from the unlabelled test cohort itself: how dense a candidate TCR's own neighbourhood is.0.527388103 min · $32.52
  49. v49Normalized cohort density against hash-selected background cohortssolver unchanged106 min · $33.04
  50. v50Cohort density under nine TCRdist geometries, Atchley, n-grams and ESM0.527388110 min · $33.78
  51. v51Audited char3-4 cohort density at 10-20% weight; accepted at 15%solver unchanged111 min · $34.37
  52. v52Production: cohort residual becomes 15% char 3-4-gram cosine density0.529107112 min · $34.64
  53. v53Two- and three-way cohort-density rank ensembles under 25% total weightsolver unchanged114 min · $35.17
  54. v54Reranked peptide-nearest sources by density/score correlation and tail0.529107117 min · $35.99
  55. v55Audited the top-8 correlation rerank; the gain was one epitope onlysolver unchanged122 min · $37.37
  56. v56Neighbour counts, TF-IDF, Jaccard/Dice and sequence-boundary tokens0.529107124 min · $38.49
  57. v57Audited boundary-marked nearest-neighbour density; accepted at 20%solver unchanged126 min · $39.17
  58. v58Production: 20% boundary-marked char3-4 nearest-neighbour density0.529833126 min · $39.55
  59. v59Restricted cohort search to shared CDR3 length, V gene/family and J genesolver unchanged127 min · $40.14
  60. v60Mutuality, reciprocal rank, indegree, shared neighbours and clustering0.529833130 min · $40.97
  61. v61Audited 2.5-20% internal neighbour propagation; accepted at 10%solver unchanged132 min · $41.74
  62. v62Production: 90% candidate density mixed with 10% of its neighbour's0.530173133 min · $42.64
  63. v63Residue-level peptide mapping audit: endpoint, positional, contact pooling0.530173139 min · $45.11
  64. v64Cohort-kernel audit: 2-5-mers, reduced alphabets, gapped and spaced motifs0.530173143 min · $46.45
  65. v65Nonlinear and gated retrieval/density fusion, adaptive density weights0.530173149 min · $47.86
  66. v66Retested the v65 sigmoid gain on private-clonotype and downsampled cohorts0.530173151 min · $48.56
  67. v67Exact TCRdist over the peptide-nearest 2-8 sources, blended as a residual0.530173155 min · $49.85
  68. v68Audited the 5% top-2 joint-source residual; matched robustness was mixedsolver unchanged160 min · $51.86
  69. v69Reran that audit at 2.5%; matched resampling hit a stale metric cacheaudit crashed163 min · $52.90
  70. v70Cache-independent rerun of the 2.5% audit; the residual was rejected0.530173166 min · $54.42
  71. v71Weak rank ensembles with Atchley, BLOSUM-row and one-hot binder similarity0.530173168 min · $55.63
  72. v72Froze v62, stripped research harnesses, added ambiguous-residue fallback0.530173172 min · $57.15

72 snapshots, 12 of them production promotions, $57.44 over 173 min. The rest were research sweeps that left the graded solver untouched.

On the hidden set

held-out metricreward
shipped starter solver0.4974360.00
best known solution (codex v23)0.5472040.30
metric maximum1.0000001.00
this run0.5104080.0782
taskhttps://huggingface.co/datasets/autoresearch-arena/rsi-tasks-1/tree/0814/tcr_epitope_unseen_specificityrollouthttps://huggingface.co/datasets/autoresearch-arena/rsi-rollout/tree/main/tcr_epitope_unseen_specificity/codex-gpt-5.6-sol-max/tcr_epitope_unseen_specificity__FQYcDD7
PreviousCross-dataset single-cell type annotation — maximize held-out macro-F1 at inference timeNextChest X-ray triage under cross-site distribution shift