Tasks/Math & Scientific Computing/Astrophysics

Retrieve exoplanet atmospheres from transmission spectra

Retrieve exoplanet atmospheres from transmission spectra

exoplanet_transit_spectroscopy Math & Scientific Computing Astrophysics
instruction.mdthis is what the agent is given

You inherit a spectrum-blind baseline: for every planet it predicts the training-set marginal mean and standard deviation for each atmosphere parameter, completely ignoring the 52-bin transmission spectrum it was given. Your goal is to raise the mean Gaussian-Log-Likelihood (GLL) of your predicted posterior mean/std against the true injected atmosphere parameters; a sealed verifier re-trains and re-runs your solver on a HIDDEN, disjoint batch of planets and scores it (higher is better).

Hard Constraints

  • You may only edit code under /app/methods/main/; you may add sibling .py modules.
  • Keep the entrypoint signatures train(train_records) -> None and predict(test_records) -> (mean, sigma), where mean/sigma are each a list of lists of shape [len(test_records), 6], one row per input record in the same order as test_records (positional, not keyed by planet ID), columns in the order of TARGETS = ["planet_temp", "log_H2O", "log_CO2", "log_CH4", "log_CO", "log_NH3"]. sigma must be > 0 everywhere.
  • numpy + pandas + scikit-learn + scipy + the standard library only — no internet at run time, no other third-party packages. The verifier image has exactly these; any other import makes the submission score 0. Network access during your own session is restricted to the model API as well (an allowlist is enforced outside your container), so solve the task from the data in /app/data rather than trying to fetch anything.
  • Grading budget (the box your submission is re-run in). train() + predict() together run once, in a fresh process, under a 7200-second wall-clock budget with 4 CPUs and 1 GB of memory — the same shape as this session's container, so a self-check that fits here fits there. Over-budget or out-of-memory means the whole submission scores 0, and no partial result is kept. The graded call is modestly larger than selfcheck.py's: it fits on 3200 labelled planets (vs 2560 in the self-check split) and predicts 800 (vs 640) — 1.25x on both, so scale your self-check timings by about that before deciding you have room.
  • Your solver must be deterministic given the shipped train/test data — the verifier calls train() then predict() fresh in its own process; unseeded randomness makes your local self-check unrepresentative of the graded run (seed every model you fit).
  • /dev/shm is only 64 MiB (the container default, here and in the grader) and cannot be enlarged. A shared segment larger than that is created successfully and only faults on the first write, killing the process with SIGBUS and no traceback -- no output, so the submission scores 0. If you parallelise with multiprocessing.shared_memory, or hand very large arrays to joblib/loky workers, keep the data in ordinary heap memory or point the temp folder at a normal filesystem path.

What You Have

  • /app/data/train.csv: 3200 visible planets. Each row has spec_0..spec_51 (52-bin transmission spectrum, (Rp/Rs)^2, dimensionless), noise_0..noise_51 (1-sigma instrument noise, same bins), star_radius_m, planet_radius_m, star_temperature (aux physical features), and the 6 true target columns. /app/data/wavelength_grid.csv gives the wavelength (microns) and bin width for each of the 52 spectral bins — identical grid for every planet, shipped once.
  • /app/methods/main/solver.py: the baseline solver (spectrum-blind constant mean/std) — this directory is what gets graded. Improve it in place or rewrite the algorithm entirely. Matching the baseline earns nothing.
  • /app/gll_metric.py: the exact GLL formula the verifier uses (the formula is public; only the true target values are sealed). Read it to see precisely how you are scored.
  • /app/selfcheck.py: a free, unlimited local dry-run (python /app/selfcheck.py) that fits on an 80/20 internal split of the visible planets and prints the proxy mean GLL. It is a proxy only — the real held-out planets are a different, disjoint, sealed batch.

What You Submit

Edit /app/methods/main/solver.py, keeping the contract:

def train(train_records: list[dict]) -> None:
    # Called once, on the labelled re-training set (same columns as /app/data/train.csv). Fit
    # whatever state you need and stash it (module globals are fine).
    ...

def predict(test_records: list[dict]) -> tuple[list[list[float]], list[list[float]]]:
    # Called once, on FEATURE-ONLY rows (no target columns). Return (mean, sigma), each
    # [len(test_records), 6], same row order as test_records, columns in TARGETS order.
    ...

There is no submit step and no per-attempt feedback on the real held-out set — iterate against selfcheck.py, then leave your best solver.py in place; it is graded once at the end.

How It Is Judged

After your run, the grader copies your methods/main/ into a sealed verifier, spawns a fresh process that imports it and calls train() on the sealed re-training planets, then predict() on a HIDDEN, disjoint batch of held-out planets, and scores the mean Gaussian-Log-Likelihood of your posterior mean/std against the true injected atmosphere parameters (higher is better):

GLL(x, mu, sigma) = -0.5 * log(2*pi*sigma^2) - (mu - x)^2 / (2*sigma^2)

averaged over every (planet, target) pair. Your score rises monotonically with the mean GLL, so pushing GLL up is always the goal. Any crash, wrong output shape, non-finite value, or sigma <= 0 scores the whole submission 0.

Metric

mean Gaussian log-likelihood over the 800 sealed planets · higher is better

gll = -0.5*log(2*pi*s^2) - (mu-x)^2/(2*s^2), s >= 1e-6, over every (planet, target) pair

anchorvisible setheld-outreward
Bshipped starter (spectrum ignored)-2.7917-2.79990.00
Rauthor's best ordinary solution-1.2099-1.11050.30
Sexpert pre-fitted on 30k outside planets-0.57830.60
Uanalytic max of gll (mu=x, s=1e-6)12.89661.00
normalisation
m <= B0
B < m <= R0.3 * (m - B) / (R - B)
R < m <= S0.3 + 0.3 * (m - R) / (S - R)
S < m < U0.6 + 0.4 * (m-S)/(m-S+tau) * (W+tau)/W
m >= U1

m = this run's held-out metric  ·  B = shipped starter (spectrum ignored)  ·  R = author's best ordinary solution  ·  S = expert pre-fitted on 30k outside planets  ·  U = analytic max of gll (mu=x, s=1e-6)

Mapped once on the aggregate mean, no per-case step. W = U-S; tau = 0.4W/(0.3W/(S-R) - 0.4) = 0.7491 keeps the slope continuous at S.

Rollouts

720 minwall clock
$14.41spend
20.7Mtokens
2versions, 2 kept
-1.2 -0.9 -0.6 -0.3 0 0.3 $0 $2.5 $5 $7.5 $10 cumulative spend on the run 5-/4-fold CV mean GLL on visible planets, higher is better author's best ordinary soluti… · visible · -1.2099 v1 v2
keptrevertedno scoreturning point
  1. v1Physics features, per-target HGB mean, HGB variance on OOF squared residualsHand-built physics features under boosted trees, a second booster fitting sigma from the residuals.-1.139110 min · $0.91
  2. v220-net numpy MLP ensemble trained on the Gaussian NLL, 800 epochs, wd 6e-3Train on the graded loss itself: one fit gives both moments, and ensemble spread supplies the width (K=1 -20.2, K=10 -0.03).0.2003466 min · $10.78

Two snapshots in 12 h: v1 at 10 min / $0.91, v2 at 466 min / $10.78 after ~20 background configs. A last 4 h sweep produced no v3.

On the hidden set

held-out metricreward
shipped starter (spectrum ignored)-2.79990.00
author's best ordinary solution-1.11050.30
expert pre-fitted on 30k outside planets-0.57830.60
analytic max of gll (mu=x, s=1e-6)12.89661.00
this run+0.36650.8355
302 minwall clock
$87.52spend
158.8Mtokens
18versions, 16 kept
-3.00 -2.25 -1.50 -0.75 0 $0 $20 $40 $60 $80 cumulative spend on the run self-check mean GLL on visible planets, higher is better author's best ordinary soluti… · visible · -1.2099 v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17
keptrevertedno scoreturning point
  1. v0Inherited spectrum-blind baseline: training-set marginal mean and std per target-2.79171 min · $0.20
  2. v1Per-target HGB on normalised spectra, exact-config temperature lookup, OOF sigmaSwap the marginal baseline for one booster per target on normalised spectra, and read temperature off repeated configurations.-0.83724 min · $2.79
  3. v2Global OOF sigma replaced by 10-bin shrunk conditional RMSE curves per prediction-0.686543 min · $5.60
  4. v3Fixed cross-split ensembles of spectral views per target; curves refit on their OOF-0.657156 min · $8.05
  5. v4Target-specific boosting capacity: 250/15 for temperature, 300/31 for molecules-0.630973 min · $11.71
  6. v5Calibrated view disagreement folded into sigma, clamped 0.5-4x; seed-0 worse-0.647490 min · $15.58
  7. v6Drops the 40-case variance shrinkage; 12/20 OOF bins use local RMSE directly-0.5874110 min · $20.38
  8. v7Exact-configuration temperature sigma cut from 0.2 K to 0.02 KRepeated configurations pin temperature: 47,502 exact matches had 0.0067 K pooled RMSE, so its width can shrink tenfold.-0.4411116 min · $22.22
  9. v8Gamma-boosted OOF squared residuals on 15 meta-features replace the 1-D sigma curvesLearn sigma as a nonlinear function of mean, spectral and noise meta-features instead of binning one OOF prediction.-0.2995139 min · $28.05
  10. v9Depth-diverse HGB blends for the single-view targets CH4 (75/25) and CO (50/50)-0.2977154 min · $32.06
  11. v10Half-strength OOF linear de-shrink calibration for CO only, slope near 1.04-0.2949168 min · $36.43
  12. v11Temperature mean blends 30% ExtraTrees, fold-wise for OOF and full-data for final-0.2862177 min · $39.25
  13. v12Adds a linmulti view of Gaussian line residuals at widths 1/2/4, blended into 3 gases-0.2599198 min · $46.30
  14. v13Weighted mean-model disagreement added as a 16th feature to the Gamma sigma model-0.2357213 min · $51.61
  15. v14Half-strength stacked residual-mean HGBs on the same meta-features, cross-fitted-0.2225257 min · $68.53
  16. v15Stacked mean-bias correction disabled for NH3, the one target it had worsened-0.2226281 min · $78.75
  17. v16Mean OOF calibration moved from 4 folds to 5; seed-0 worse at 25% more cost-0.2269292 min · $83.27
  18. v17Submission cleanup: dead sigma-curve code and unused views out, predictions identical-0.2226301 min · $87.42

Eighteen snapshots in 302 min and $88, two reverted (v5, v16). Sigma modelling drove the jumps: v7 and v8 add 0.29 GLL between them.

On the hidden set

held-out metricreward
shipped starter (spectrum ignored)-2.79990.00
author's best ordinary solution-1.11050.30
expert pre-fitted on 30k outside planets-0.57830.60
analytic max of gll (mu=x, s=1e-6)12.89661.00
this run-0.02450.7795
135 minwall clock
$22.78spend
37.6Mtokens
12versions, 12 kept
-3.0 -2.5 -2.0 -1.5 -1.0 -0.5 $0 $5 $10 $15 $20 cumulative spend on the run self-check mean GLL on visible planets, higher is better author's best ordinary soluti… · visible · -1.2099 v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11
keptrevertedno scoreturning point
  1. v0Inherited spectrum-blind baseline: training-set marginal mean and std per target-2.79170 min · $0.09
  2. v1500-tree multi-output ExtraTrees on continuum-normalised shape; constant sigma-2.12732 min · $0.24
  3. v2Per-gas HGB regressors, 5-fold OOF calibration, separate temperature modelOne model per target instead of a shared multi-output forest, each calibrated out-of-fold with its own sigma.-1.31446 min · $0.69
  4. v3Physics features: scale-height depth, local differences, noise-normalised contrastsFeed the boosters physics, not raw bins: relative depth, atmospheric height, four difference scales, band contrasts.-0.915812 min · $1.22
  5. v4Gas boosting capacity to 31 leaves / 300 rounds; temperature on the same matrix-0.802623 min · $2.54
  6. v516th/84th-percentile models give each planet its own sigma, shrunk to constantOne width per planet, not per target: quantile models, log-shrunk with a per-gas power, calibrated on OOF standardised errors.-0.594232 min · $3.77
  7. v6Temperature mean blends ExtraTrees and HGB; tree disagreement becomes its sigma-0.492841 min · $5.00
  8. v7Gas mean averages full- and 70%-feature HGB; repeat-system sigma floor to 0.02 K-0.344653 min · $6.78
  9. v8Standardised multi-output MLP added at 50% weight to the log_CO mean only-0.325964 min · $8.64
  10. v9Same MLP reused for H2O (30%) and NH3 (40%); CO2 and CH4 blends rejected-0.3097100 min · $15.20
  11. v10Unknown-system temperature mean adds a 25% group-cross-fitted neural part-0.3084112 min · $17.86
  12. v11Frees fold models before the full-data fit; predictions identical to v10-0.3084123 min · $20.09

Twelve snapshots in 123 min and $20, all kept, each better than the last. v11 is memory hygiene only: audited peak 854 -> 800 MiB under the 1 GB cap.

On the hidden set

held-out metricreward
shipped starter (spectrum ignored)-2.79990.00
author's best ordinary solution-1.11050.30
expert pre-fitted on 30k outside planets-0.57830.60
analytic max of gll (mu=x, s=1e-6)12.89661.00
this run-0.14220.7553
192 minwall clock
-spend
-tokens
12versions, 8 kept
-2.8 -2.4 -2.0 -1.6 -1.2 0 30 60 90 120 agent step (this harness reports no tokens or timestamps) self-check mean GLL on visible planets, higher is better author's best ordinary soluti… · visible · -1.2099 v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11
keptrevertedno scoreturning point
  1. v0Inherited spectrum-blind baseline: training-set marginal mean and std per target-2.7917
  2. v1HGB on relative, log, derivative, height and band features; OOF-calibrated sigmaNormalise each spectrum by its own optical continuum and scale height, then calibrate a per-planet width out-of-fold.-1.2237
  3. v2Two-stage: temperature first, abundances on the T-normalised excess spectrum-1.2107
  4. v3Temperature regressed in sqrt space; optical-slope features; 4-fold OOF sigma-1.2055
  5. v4Three-way temperature blend, 0.2 linear / 0.4 sqrt / 0.4 log, weights fit leakily-1.2118
  6. v50.75 two-stage HGB + 0.25 per-target MLP on compact relative-spectrum featuresBlend in a network whose errors differ from the trees', at a fixed 3:1 weight rather than a fitted one.-1.1512
  7. v6Milder sigma floors plus a 0.88 post-OOF shrink; full fit sharper than OOF-1.1466
  8. v7Averages two MLP seeds to cut neural variance-1.1528
  9. v8Bags two HGB seeds in the final temperature and molecule models only-1.1466
  10. v9Final MLP refit on every label, iteration count taken from the OOF medianSpend the early-stopping holdout on OOF only: the submitted network then trains on all rows at a borrowed iteration budget.-1.0951
  11. v10Final-MLP iteration budget raised to 1.25x the OOF median, floor 80-1.0901
  12. v11Temperature blends 0.75 squared-error and 0.25 absolute-error HGB-1.0977

Twelve snapshots over 192 min, four reverted (v4, v7, v8, v11). Every keep also had to hold on a locked seed-123 holdout, not just the self-check.

On the hidden set

held-out metricreward
shipped starter (spectrum ignored)-2.79990.00
author's best ordinary solution-1.11050.30
expert pre-fitted on 30k outside planets-0.57830.60
analytic max of gll (mu=x, s=1e-6)12.89661.00
this run-1.06290.3268
40 minwall clock
$0.62spend
2.2Mtokens
1versions, 1 kept

No trajectory curve: this run left one comparable self-check measurement, so there is nothing to plot against spend. The versions and what each one changed are below.

  1. v1Scale-height normalised features, HGB mean, nested-OOF residual model for sigmaDivide out baseline depth and star-planet scale to leave atmospheric height, then fit sigma on nested OOF absolute residuals.-1.848534 min · $0.48

One snapshot only, at 34 min and $0.48 of a 40 min $0.62 run; the last fourteen steps re-ran the self-check and tidied the solver.

On the hidden set

held-out metricreward
shipped starter (spectrum ignored)-2.79990.00
author's best ordinary solution-1.11050.30
expert pre-fitted on 30k outside planets-0.57830.60
analytic max of gll (mu=x, s=1e-6)12.89661.00
this run-1.80760.1762