Tasks/Life Sciences/Protein Structure & Function

Protein Stability Regression — maximize held-out Spearman correlation

Train and ship a protein-stability regressor

peer_protein Life Sciences Protein Structure & Function
instruction.mdthis is what the agent is given

You inherit a protein sequence regression problem: each example is a short amino-acid sequence (an engineered mini-protein, 43-50 residues) and you must predict its stability score, a real number measured experimentally. The shipped methods/main/solver.py is a deliberately small linear model over amino-acid composition. You train a regressor and save it; a sealed verifier then loads your model and re-runs your predict() on a hidden set of sequences you never see (it does NOT re-train), scoring the held-out Spearman rank correlation between your predictions and the true scores (HIGHER is better) — so only a model that genuinely captures sequence-to-stability structure raises the score.

Hard Constraints

  • CRITICAL (artifact-eval timeout safety): your train() MUST persist the model to out_dir (/app/submission/model) before it can time out — write the model as soon as it is fit, and if you add iterative training, checkpoint periodically rather than only at the very end. The grader scores whatever is in /app/submission/model at the deadline; an empty submission scores 0.
  • You may only edit code under /app/methods/main/; you may add sibling .py modules. The two entrypoints and their signatures must not change — the verifier imports them directly:
  • train(train_data: dict, out_dir: str, device: str = "cpu") -> None — fit a regressor on the labelled sequences and save everything needed to reload it (parameters + any preprocessing config) into out_dir.
  • predict(model_dir: str, test_ids: list, dirs: dict, device: str = "cpu") -> dictload the model you saved in model_dir, read the sequences named in dirs["csv_path"], and return {test_id: predicted_score} (one float per test id).
  • train_data is {"ids": [...], "csv_path": str}, where csv_path is a CSV with columns id,sequence,protein_length,stability_score for every id in "ids". dirs is {"csv_path": str} for the test rows: same columns but WITHOUT stability_score.
  • Do not attempt to read or reconstruct the sealed test labels; do not hardcode answers. The test sequences are NOT in your environment — they are supplied only when the verifier re-runs your predict(), and the true scores live solely in the verifier.
  • A crash or a missing test id scores the whole submission 0.

What You Have

  • The workspace /app/ (/app/README.md orients you):
  • data/train.csv — 53,614 labelled training sequences (id,sequence,protein_length,stability_score).
  • data/valid.csv — 2,512 labelled validation sequences, same columns. This is a pre-defined validation split (not one you need to carve yourself).
  • methods/main/solver.py — the weak starter regressor, split into train() + predict(). This directory is what gets graded, together with the model you save under /app/submission/model/.
  • DATA_CARD.md — what the sequences and labels are and the data layout.
  • python /app/selfcheck.py — a free, unlimited local dry-run: it calls your train() on data/ train.csv, saves a model to a temp dir, runs your predict() on data/valid.csv (labels withheld from predict), and reports the validation Spearman correlation. It is a proxy only — the sealed test set is a different, held-out batch of sequences, so keep a margin and do not over-fit the split.

What You Submit

The workflow:

  1. Edit /app/methods/main/solver.py, keeping the train / predict signatures above.
  2. Run it to produce the model: python /app/methods/main/solver.py trains on data/train.csv and saves the model to /app/submission/model/. (Equivalently call train(train_data, "/app/submission/model") yourself.)
  3. Leave the model in place. The verifier loads /app/submission/model and calls your predict("/app/submission/model", sealed_test_ids, dirs) on the sealed test sequences.

The budget is 8 CPU cores with no GPU, so think carefully about how to train and save your model efficiently. There is no submit step and no per-attempt feedback — iterate against the self-check, then leave your best solver.py and trained model in place; it is graded once at the end.

How It Is Judged

After your run, the verifier copies your methods/main/ and /app/submission/model/ into a sealed sandbox, loads your model and re-runs your predict() on the hidden test sequences (it does NOT re-train), and scores your predictions against the SEALED true stability scores:

metric = Spearman rank correlation between your predicted scores and the true scores   (HIGHER is better)
gate   = score must be strictly above 0.0 (the score of an uninformative, constant prediction) to place

The verifier owns the true scores and recomputes the correlation itself, so the score is honest; it does not read any number your code reports, and you get no feedback from that run.

Metric

Spearman rho on the sealed held-out stability set · higher is better

Rank correlation between predicted and measured stability over 12,851 sealed sequences; higher is better

anchorvisible setheld-outreward
Bshipped starter (composition ridge)0.2591000.0942940.00
Rhuman reference (feature-count)0.6040000.5980070.30
Shidden SOTA (pretrained embeddings)0.6036000.6921540.60
Uupper bound (rho = 1)1.0000001.0000001.00
normalisation
m <= B0
B < m <= R0.3 * (m - B) / (R - B)
R < m <= S0.3 + 0.3 * (m - R) / (S - R)
S < m <= U0.6 + 0.4 * (m - S) / (U - S)
m > U1.0 (clamped, no extrapolation)

m = this run's held-out metric  ·  B = shipped starter (composition ridge)  ·  R = human reference (feature-count)  ·  S = hidden SOTA (pretrained embeddings)  ·  U = upper bound (rho = 1)

B/R/S/U are the four anchors. One sealed set, mapped once, no averaging. Every segment linear; rho <= 0 fails the gate and scores 0.

Rollouts

326 minwall clock
$69.56spend
118.0Mtokens
9versions, 9 kept
0.30 0.45 0.60 0.75 $0 $15 $30 $45 $60 cumulative spend on the run official validation Spearman, higher is better hidden SOTA · visible · 0.603600 v0 v1 v2 v3 v4 v5 v6 v7 v8
keptrevertedno scoreturning point
  1. v0Shipped starter: ridge on 20-residue amino-acid composition, alpha 10.25911 min · $0.12
  2. v1Directed pair frequencies at gaps 1-20, ridge alpha 0.1, length-50 rowsComposition throws away order; directed residue pairs at many gaps keep it while the model stays a ridge.0.68077 min · $0.71
  3. v2Add gapped residue triples, rank targets, cross-regime row weights0.702139 min · $4.29
  4. v3Blend 85% three-seed dilated residual CNN ranks with 15% pair/triple ridgeRead the sequence directly with small fixed-epoch CNNs and average their ranks, not their scores.0.755578 min · $10.80
  5. v4Five-family rank ensemble: scalar/ordinal CNNs, GRUs, masked-pretrained seeds0.7948182 min · $34.81
  6. v5Sixth family: ordinal CNN on broader cross-regime weights; rebalanced blend0.797289223 min · $46.55
  7. v6BLOSUM62 similarity channel in the ordinal CNN, four seeds, 50/50 with v5Hand the network residue chemistry through a fixed BLOSUM62 channel, then replicate seeds instead of picking one.0.808491255 min · $52.33
  8. v7Second BLOSUM CNN family on broad regime weights; 20/35/45 allocation0.816062276 min · $56.87
  9. v8Add BLOSUM ordinal GRUs, prune nine legacy checkpoints, 30/45/25 blend0.819037313 min · $65.87

v0-v3 took 78 min and $10.80; the neural families in v4-v8 took the rest of 5.4 h and $69.56. Nothing was reverted.

On the hidden set

held-out metricreward
shipped starter (composition ridge)0.0942940.00
human reference (feature-count)0.5980070.30
hidden SOTA (pretrained embeddings)0.6921540.60
upper bound (rho = 1)1.0000001.00
this run0.6661520.5171