Tasks/Math & Scientific Computing/Computational Chemistry

Predict the HOMO-LUMO energy gap of a molecule from its SMILES string

Predict a molecule's HOMO-LUMO gap from its SMILES string

molecular_homolumo_gapMath & Scientific ComputingComputational Chemistry
instruction.mdthis is what the agent is given

You inherit a feature-blind baseline: for every molecule it predicts the training-set mean HOMO-LUMO gap, completely ignoring the smiles column it was given. Your goal is to lower the mean absolute error (MAE, in eV) of your predicted gap against the true DFT-computed gap; a sealed verifier re-trains and re-runs your solver on a HIDDEN, disjoint batch of 15,000 molecules and scores it (lower is better, 0 = every prediction exact).

Hard Constraints

  • You may only edit code under /app/methods/main/; you may add sibling .py modules.
  • Keep the entrypoint signatures train(train_records) -> None and predict(test_records) -> list[float], where the returned list has one entry per row of test_records, in the same order as test_records (positional, not keyed by any ID column), each entry a finite float (the predicted gap in eV).
  • numpy + pandas + scikit-learn + scipy + rdkit + the standard library only -- no other third-party packages (in particular, no torch/dgl/xgboost/lightgbm/catboost -- the verifier image does not have them). Any other import makes the submission score 0.
  • Your solver must be deterministic given the shipped train/test data -- the verifier calls train() then predict() fresh in its own process; unseeded randomness makes your local self-check unrepresentative of the graded run (seed every model you fit).
  • Do not fetch chemistry databases or any external data at run time. Only the data shipped in this container may be used. Treat any outbound request as against the spirit of the task even if it happens not to be blocked.
  • Grading budget (declared so you can size your method): your train() + predict() are re-run once, in a fresh process, under a 3600-second wall-clock budget on 8 CPU cores and 8 GiB of memory, CPU only, no network. The graded run re-trains on 80,000 labelled rows (the same count as your visible /app/data/train.csv) and predicts 15,000 held-out rows -- about 1.2x the total rows your local self-check touches. Over-budget, OOM-killed, crashed or wrongly-shaped output scores 0, so leave real margin rather than tuning to the wall.
  • /dev/shm is only 64 MiB (the container default, here and in the grader) and cannot be enlarged. A shared segment larger than that is created successfully and only faults on the first write, killing the process with SIGBUS and no traceback -- no output, so the submission scores 0. If you parallelise with multiprocessing.shared_memory, or hand very large arrays to joblib/loky workers, keep the data in ordinary heap memory or point the temp folder at a normal filesystem path.
  • Your submitted methods/main/ must contain only small .py source files. The verifier rejects (reward 0) any submission that does not look like code: any non-.py file, any symlink, or a total submission size over the guard's byte cap is rejected outright.

What You Have

  • /app/data/train.csv: 80,000 labelled molecules. Each row has ROW_ID, smiles (a SMILES string), and homolumogap (the true HOMO-LUMO gap in eV, computed by DFT on the molecule's relaxed 3D geometry -- your training target). No other columns, no 3D coordinates, no precomputed features -- you featurize smiles yourself.
  • /app/methods/main/solver.py: the baseline solver (predicts the training-set mean gap for every molecule) -- this directory is what gets graded. Improve it in place or rewrite the algorithm entirely. Matching the baseline earns nothing.
  • /app/force_score.py: the exact MAE formula the verifier uses (the formula is public; only the true held-out gaps are sealed). Read it to see precisely how you are scored.
  • /app/selfcheck.py: a free, unlimited local dry-run (python /app/selfcheck.py) that fits on an internal 80/20 split of the visible rows and prints the proxy MAE. It is a proxy only -- the real held-out set is a different, disjoint batch of molecules, so do not overfit to this split.

What You Submit

Edit /app/methods/main/solver.py, keeping the contract:

def train(train_records: list[dict]) -> None:
    # Called once, on the labelled re-training rows (same columns as /app/data/train.csv, string
    # values -- csv.DictReader output). Fit whatever state you need and stash it (module globals are
    # fine).
    ...

def predict(test_records: list[dict]) -> list[float]:
    # Called once, on FEATURE-ONLY rows (ROW_ID + smiles, no homolumogap column). Return one
    # predicted gap (float, eV) per row, same row order as test_records.
    ...

There is no submit step and no per-attempt feedback on the real held-out set -- iterate against selfcheck.py, then leave your best solver.py in place; it is graded once at the end.

How It Is Judged

After your run, the grader copies your methods/main/ into a sealed verifier, spawns a fresh process that imports it and calls train() on the sealed re-training rows, then predict() on a HIDDEN, disjoint batch of 15,000 molecules you never saw, and scores the mean absolute error of your predicted gap against the true gap (lower is better):

score(y_true, y_pred) = mean_i( |y_true[i] - y_pred[i]| )

Your score improves monotonically as MAE falls, so driving MAE down is always the goal -- 0 (exact prediction) is a genuine floor, and there is real headroom between the shipped baseline and a well-tuned solution. Any crash, wrong output length, or a non-finite prediction scores the whole submission 0.

Metric

held-out MAE of the predicted HOMO-LUMO gap, in eV · lower is better

mean |y_true - y_pred| over 15,000 sealed molecules; the verifier re-trains the solver on 80,000 rows, 8 cores, 3600 s.

anchorvisible setheld-outreward
Bshipped starter (training-set mean)0.90500.90900.00
Rreference solution0.39820.39260.30
Sexpert SOTA solution0.21610.20810.60
Utheoretical bound (MAE = 0)0.00.01.00
normalisation
m >= B0
B > m >= R0.3 * (B - m) / (B - R)
R > m >= S0.3 + 0.3 * (R - m) / (R - S)
m < S0.6 + 0.4 * (S - m) / (S - U)

m = this run's held-out metric  ·  B = shipped starter (training-set mean)  ·  R = reference solution  ·  S = expert SOTA solution  ·  U = theoretical bound (MAE = 0)

U = 0. One hidden score, so mapped once, not per case. Linear in the raw MAE: a log rescale is impossible at U = 0. Floor 0, cap 1.

Rollouts

667 minwall clock
$129.97spend
227.2Mtokens
11versions, 11 kept
0.15 0.30 0.45 0.60 0.75 0.90 $0 $30 $60 $90 $120 cumulative spend on the run official seed-0 self-check MAE in eV, lower is better expert SOTA solution · visible · 0.2161 v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10
keptrevertedno scoreturning point
  1. v0Shipped feature-blind training-mean baseline; validation protocol fixed0.9052 min · $0.30
  2. v1217-descriptor RDKit panel with a 160-tree deterministic ExtraTrees0.2615 min · $1.19
  3. v2Two 255-leaf boosters over descriptors plus 2048-bin atom-pair and path countsTrees replaced by boosters, each with a complementary hashed count view; the 50/50 blend buys two views disagreeing.0.219669 min · $5.82
  4. v3192 atom/bond/ring, graph-spectrum, Coulomb and Huckel-like features added0.2035112 min · $10.74
  5. v4Boosters to 1000 rounds, 70% node sampling; clipped MLPs and a SMILES RidgeSmooth function classes alongside the trees. 3D conformers, AUTOCORR2D and Morgan boosters failed the accuracy-per-second gate.0.1922274 min · $38.22
  6. v532 focused adjacency/Huckel eigenvalue-gap and dimensionless pi-system features0.192350 min · $54.73
  7. v676 extended-Huckel features: pi occupancy, on-site terms, gaps computed 12 waysStopped describing the graph and started approximating the target: an extended-Huckel gap under twelve parameterizations.0.1846406 min · $67.32
  8. v736 gap alternatives at occupancy offsets -2/-1/+1 for uncertain lone pairs0.1837473 min · $83.80
  9. v8Spectral-gap and Huckel features fed to the MLPs; third narrow latent MLP kept0.1826522 min · $91.80
  10. v9Each MLP replaced by the deterministic mean of three fits seeded 123/321/7770.181596 min · $108.45
  11. v10Second atom-pair booster on seed 321; AP/AP/path blended 30/30/400.1802666 min · $129.63

Eleven versions, none reverted, 667 minutes and $129.97 — 4.5x the other two runs. Flat after v6: the last four bought 0.0044 eV for $62.

On the hidden set

held-out metricreward
shipped starter (training-set mean)0.90900.00
reference solution0.39260.30
expert SOTA solution0.20810.60
theoretical bound (MAE = 0)0.01.00
this run0.172640.6682
350 minwall clock
$28.84spend
43.8Mtokens
4versions, 4 kept
0.20 0.24 0.28 0.32 $0 $7.5 $15 $22 cumulative spend on the run official seed-0 self-check MAE in eV, lower is better expert SOTA solution · visible · 0.2161 v1 v3 v4 v5
keptrevertedno scoreturning point
  1. v1Sparse count fingerprints (Morgan r2/r3, atom-pair, torsion) + Ridge alpha=20One linear model on hashed count fingerprints, on disk before any tuning. 0.907 to 0.341 in eight minutes.0.34148 min · $0.49
  2. v3Two-tower numpy MLP: dense RDKit descriptors, un-hashed Morgan/AP/torsion vocabsUn-hashed vocabularies instead of hashed bits, descriptors as their own dense tower, near-L1 Huber for an MAE metric.no completed self-check129 min · $11.09
  3. v4Same features, 8-seed fork-pool ensemble; vectorised featurizer ends OOM-killsEnsembling is near-free on 8 cores, but only once the featurizer stopped returning 10M-entry dicts that killed pool workers.0.1879221 min · $19.14
  4. v5Six shape-diverse members instead of eight identical ones, for wall-clock margin0.1863304 min · $25.20

Only four versions were snapshotted; v0 and v2 were not. The v3 self-check never printed a number. Half the 350 minutes went on v3.

On the hidden set

held-out metricreward
shipped starter (training-set mean)0.90900.00
reference solution0.39260.30
expert SOTA solution0.20810.60
theoretical bound (MAE = 0)0.01.00
this run0.175320.663
375 minwall clock
$99.67spend
175.3Mtokens
6versions, 6 kept
0.15 0.30 0.45 0.60 0.75 0.90 $0 $20 $40 $60 $80 cumulative spend on the run official seed-0 self-check MAE in eV, lower is better expert SOTA solution · visible · 0.2161 v1 v2 v3 v4 v5 v6
keptrevertedno scoreturning point
  1. v1Shipped feature-blind training-mean baseline0.9052 min · $0.24
  2. v2217 RDKit 2-D descriptors; 200-tree ExtraTrees and squared-loss booster at 70/30The whole 2-D descriptor panel under two model families at once; 0.9050 to 0.2568 in 18 minutes.0.256818 min · $1.59
  3. v32048-bit Morgan, a 16k-bin sparse-count MLP, 1600-round booster, affine calibrationA new function class, not more descriptors: a sparse Morgan-count MLP beside the trees, calibrated across splits.0.216582 min · $10.28
  4. v4Joint booster over 192 typed/conjugation/Huckel and 32 spectral features; two MLPsPhysics in the features rather than capacity: typed conjugation and Huckel graph terms plus fold-stable spectral coordinates.0.1901208 min · $42.32
  5. v5Compact 200-tree spectral ExtraTrees added; ensemble weights gated on radicals0.1883277 min · $66.78
  6. v6Approximate Huckel frontier gaps, localization coords, disagreement-gated weights0.1859356 min · $93.14

Six versions, none reverted, 375 min for $99.67. Every change was re-checked on an independent seed-17 split; flat after v4, 0.0042 eV.

On the hidden set

held-out metricreward
shipped starter (training-set mean)0.90900.00
reference solution0.39260.30
expert SOTA solution0.20810.60
theoretical bound (MAE = 0)0.01.00
this run0.179470.655
207 minwall clock
-spend
-tokens
9versions, 9 kept
0.22 0.23 0.24 0.25 0 25 50 75 100 agent step (this harness reports no tokens or timestamps) 20k/5k tune-slice MAE in eV, lower is better expert SOTA solution · visible · 0.2161 v0 v1 v2 v3 v4 v6 v7 v9 v10
keptrevertedno scoreturning point
  1. v0Shipped mean predictor kept as the baseline snapshotall-visible 0.9066
  2. v1HGB with MAE loss on Huckel/conjugation, RDKit, MQN and Morgan512 countsHand-built Huckel and conjugation terms in the first model; slice checks name radicals as the dominant error.0.2537
  3. v2Morgan dropped; radical extras, pi-GNN pools, 2-seed HGB + radical specialistA specialist model for radicals, blended in at 0.55, because their MAE ran twice the closed-shell one.0.2429
  4. v3Ring-size and fragment counts; ExtraTrees blended 0.50 with the HGB pair0.2383
  5. v4Huckel +/-1e fillings, chromophore and lone-pair counts; squared-error HGB addedPush the Huckel picture further: occupancy shifted by one electron, isolated and conjugated chromophores counted.0.2317
  6. v6Stronger squared-error HGB (127 leaves, 480 iters); A/B blend re-weighted 0.40/0.600.2304
  7. v7Radical specialist enlarged; threadpool_limits(1) around ExtraTrees0.23
  8. v9167-bit MACCS keys appended to the feature vector0.2278
  9. v10Thread env defaults restored, n_jobs = min(4, cpu_count); model unchangedselfcheck 0.1982

Nine snapshots in 207 min; this harness reports no tokens or cost. Screening ran on a 20k/5k slice; the official selfcheck ran only at v6 and v10.

On the hidden set

held-out metricreward
shipped starter (training-set mean)0.90900.00
reference solution0.39260.30
expert SOTA solution0.20810.60
theoretical bound (MAE = 0)0.01.00
this run0.189720.6353
175 minwall clock
$28.46spend
44.0Mtokens
9versions, 8 kept
0.15 0.30 0.45 0.60 0.75 0.90 $0 $5 $10 $15 $20 cumulative spend on the run official seed-0 self-check MAE in eV, lower is better expert SOTA solution · visible · 0.2161 v0 v1 v2 v3 v4 v5 v6 v7 v8
keptrevertedno scoreturning point
  1. v0Shipped feature-blind training-mean predictor0.9051 min · $0.17
  2. v1All RDKit 2D descriptors, median imputation, 300-tree ExtraTreesThe whole 2D descriptor panel straight into a tree ensemble, no feature engineering. 87% of the run's gain in five minutes.0.26217 min · $0.66
  3. v250/50 blend of 300-tree ExtraTrees and a 500-iteration squared-loss booster0.251623 min · $1.92
  4. v3Morgan r2/2048-bit branch: Ridge 0.45, sparse ExtraTrees 0.32, blend gained 0.0012rejected, no self-check43 min · $3.83
  5. v4Booster capacity sweep; 127 leaves / minleaf 20 / 600 rounds, 30-70 vs trees0.245758 min · $5.71
  6. v546 composition, conjugation, ring and Huckel-spectral descriptors addedFirst physics in the features, not more model capacity. Ablation: booster 0.2494 to 0.2212, trees 0.2597 to 0.2176.0.212482 min · $8.83
  7. v6Sweep on augmented features: 400-tree ExtraTrees + 255-leaf booster at 45/550.21110 min · $13.22
  8. v7Structural block 46 to 188: pi-component summaries, spectra, electron-filling gapsThe same idea pushed further: describe each pi component separately. Custom-features-only ablation improved both model families.0.2053127 min · $17.09
  9. v8Loss sweep: 500-tree ET, equal slow-squared and Poisson boosters, tail calibration0.2034152 min · $22.70

Nine snapshots, 175 minutes, $28.46 — the Opus run's money for half its wall clock. It stopped while still improving, and never tried neural models.

On the hidden set

held-out metricreward
shipped starter (training-set mean)0.90900.00
reference solution0.39260.30
expert SOTA solution0.20810.60
theoretical bound (MAE = 0)0.01.00
this run0.196040.6232
79 minwall clock
$0.82spend
4.5Mtokens
3versions, 2 kept
0.242 0.244 0.246 0.248 0.250 $0 $0.2 $0.3 $0.5 $0.6 cumulative spend on the run official seed-0 self-check MAE in eV, lower is better expert SOTA solution · visible: 0.2161 · off this scale v1 v2 v3
keptrevertedno scoreturning point
  1. v1HistGB (300 iters, 63 leaves, lr 0.05) on ECFP+FCFP+MACCS and 217 descriptorsOne boosted-tree model on the whole fingerprint-plus-descriptor block; 0.9050 to 0.2492 in 17 minutes.0.249217 min · $0.28
  2. v2Boosting iterations 300 to 500 on the same features0.241121 min · $0.33
  3. v3Back to 300 iterations at lr 0.10; trains in about 1.5 minutes0.241962 min · $0.68

Three snapshots in 79 min for $0.82. v2's 500 iterations blew the 5-min shell timeout; v3 traded 0.0008 eV for speed. The graded child crashed, rc 1.

On the hidden set

held-out metricreward
shipped starter (training-set mean)0.90900.00
reference solution0.39260.30
expert SOTA solution0.20810.60
theoretical bound (MAE = 0)0.01.00
this runno predictions0