Tasks/Health & Medicine/Trials & Pharmacometrics

Longitudinal joint IRT-NLME on PRO data (R-only, no mirt)

Fit a joint IRT-NLME model in pure R

irt_nlme_pro_longitudinal Health & Medicine Trials & Pharmacometrics
instruction.mdthis is what the agent is given

You inherit a half-finished pure-R joint Item Response Theory + Nonlinear Mixed Effects model for longitudinal patient-reported outcomes (14 ordinal items over 11 time points, three latent factors). The shipped entry is deliberately weak: item parameters frozen at a baseline GRM, an exponential trajectory, and no covariates — none of which the engine requires. Your job is to make the model generalize, minimizing the held-out OFV (= −2·logL, lower is better). You submit an algorithm; the sealed verifier re-runs it on a hidden, disjoint set of subjects you never see and independently recomputes the OFV from your emitted parameters, so overfitting the visible subjects or faking a number is useless.

Hard Constraints

  • R 4.3 compatible only. No internet at run time (all packages are preinstalled).
  • mirt / mirtCAT are banned, and no Python bridges (reticulate, system("python"), system2("python")). Any banned import scores the submission invalid (reward 0).
  • Model within the provided engine. The verifier recomputes your OFV from the params you emit using its own sealed copy of engine.R, so your model must be expressible as per-factor {item GRM params a, b} + one supported trajectory (linear/exp/biexp/triexp) + the SEX/ECOG covariates.
  • Cover the 14-item instrument exactly once. You may regroup items across factors, but every item must be modeled exactly once (no dropping hard items, no double-counting easy ones).
  • Keep the solver CLI contract (see What You Submit).

What You Have

Under environment/ (baked into the agent container at /app):

  • data/data_AI_visible.csv — visible subjects (with truth) for free local development. Long format columns: ID, TIME, VARIABLE_num, ITEM_NAME, SCORE (0-3), ECOG, SEX. TIME = 0, 30, …, 300 (11 points). Three latent factors: PhysDys (Pain×3 + Dyspnea×2), GI (Appetite, Nausea×2, Vomit, Diarrhea, Constipation), Psych (Activity, Depression, Insomnia_Severity).
  • methods/main/engine.R — the pure-R joint-MLE engine (Samejima GRM + 2D Gauss-Hermite quadrature + L-BFGS-B). Trajectories linear/exp/biexp/triexp; helpers fit_joint(), fit_baseline_grm(), pack_*(), and emit_factor_params().
  • methods/main/solver.R — your submission entry (currently the V1 baseline).
  • selfcheck.R — local OFV on the visible set; run Rscript selfcheck.R from /app freely before submitting.

Covariates the verifier builds deterministically (use the SAME recipe): SEX_M = (SEX=="Male"), ECOG3 = min(ECOG,2), ECOG_1 = (ECOG3==1), ECOG_2 = (ECOG3==2); time is scaled tau = TIME/300.

What You Submit

Edit methods/main/solver.R, keeping the CLI contract:

Rscript solver.R --data <held-out csv> --out <dir>     # writes <dir>/results.json

results.json MUST contain a params object — one entry per factor — that decodes your fitted model. The easiest way is to call emit_factor_params(traj, items, fit) from the engine on each factor's fit (see the baseline solver.R). Each factor record holds: traj ("linear"/"exp"/"biexp"/"triexp"), items (order matching a), a (J), b (J×3), the trajectory scalars (mu, sigma, rho, k/k_slow/k_mid/k_fast, C/C1/C2), and beta0, beta_slope (length-3 covariate effects on baseline / steady-state-or-slope; zeros for unused). total_OFV / per_factor_OFV may also be written but are advisory only — the verifier ignores them. You may add helper .R files under methods/main/ and source() them.

How It Is Judged

The sealed verifier copies your submission into a clean sandbox, runs solver.R on the hidden held-out subjects, and then re-derives the OFV independently from your emitted params: it rebuilds the latent trajectories + Samejima GRM likelihood with its own sealed engine.R at a fixed quadrature and sums the per-factor OFV. It never trusts a self-reported number — a faked low OFV, or params that do not actually fit, both yield a (correctly) bad score.

  • Metric = held-out total OFV, lower is better. A stronger fit to the hidden subjects scores higher; the shipped starter is a deliberately weak floor.
  • Invalid (scores 0): banned import; missing params; the 14 items not covered exactly once; a non-finite or degenerate recomputed OFV (below the disclosed 8000 sanity floor).

Metric

held-out total OFV, recomputed by the sealed verifier · lower is better

-2 log-likelihood of the joint IRT-NLME model over 14 ordinal items, summed over the three factors

anchorvisible setheld-outreward
Bshipped V1 baseline83,465.9822,905.170.00
Rstrong reference solution19,727.270.30
degenerate floor, rejected below8,0000.00
normalisation
x <= 00
0 < x <= 10.3 * x
x > 11 - 0.7 / x

m = this run's held-out metric  ·  B = shipped V1 baseline  ·  R = strong reference solution

x = (B - m) / (B - R), B 22905.166, R 19727.27. One hidden set, mapped once. No sota anchor, so past R reward is soft-capped.

Rollouts

315 minwall clock
$34.34spend
54.0Mtokens
7versions, 7 kept
82.25k 82.50k 82.75k 83.00k 83.25k 83.50k $0 $7.5 $15 $22 $30 cumulative spend on the run total OFV on the 320 visible subjects, lower is better v1 v2 v3 v4 v5 v6 v7
keptrevertedno scoreturning point
  1. v1Shipped baseline: frozen baseline-GRM items, exponential trajectory, no covariates83,465.981 min · $0.34
  2. v2Joint item estimation, warm-start exp->biexp->triexp, SEX/ECOG covariatesEstimate the GRM items with the trajectory instead of freezing them; a 15x faster likelihood made that search affordable.82,328.6670 min · $6.38
  3. v3Widened the binding box constraints; baseline-jump and random multistartsShipped |C|<=3 and sigma<=15 were binding; widening opens a baseline-jump basin that decouples theta(0) from the curve.82,303.47100 min · $9.58
  4. v4Per factor: screen 16 triexp candidates at maxit 120, polish top 3, 8 basin hops60 cold starts found the best PhysDys basin once, so the likelihood is multimodal and fitting must itself be a global search.82,268.77142 min · $14.29
  5. v530 basin hops with a cheap probe and patience, plus a wall-clock budget guard82,260.53202 min · $21.13
  6. v6|C| bound 12 -> 40, full-precision JSON, missing-item and fallback guards82,262.87241 min · $25.16
  7. v7v6 re-confirmed end to end, wall-clock guard documented; submitted82,262.87306 min · $32.76

Seven versions, all kept, $34.34 over 5h15. Rejected offline: 28 item-to-factor moves, other partitions, free rates, tighter tolerances.

On the hidden set

held-out metricreward
shipped V1 baseline22,905.170.00
strong reference solution19,727.270.30
degenerate floor, rejected below8,0000.00
this run19,609.470.325
337 minwall clock
$85.74spend
153.7Mtokens
17versions, 16 kept
82.25k 82.50k 82.75k 83.00k 83.25k 83.50k $0 $20 $40 $60 $80 cumulative spend on the run total OFV on the 320 visible subjects, lower is better v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17
keptrevertedno scoreturning point
  1. v1Shipped baseline: frozen TIME=0 GRM items, exponential trajectory, no covariates83,465.982 min · $0.33
  2. v2All GRM item parameters estimated jointly from all 11 visits, not just TIME=0Frozen items are the baseline's real weakness: refitting them across every visit is worth 785 OFV alone, in every factor.82,681.2614 min · $1.35
  3. v3Independent factors fitted concurrently with mclapply; same fit, 513 s vs 699 s82,681.2623 min · $2.40
  4. v4Bi-exponential trajectory (acute + rebound), items frozen, as a cheap shape screen83,235.5125 min · $2.72
  5. v5SEX_M/ECOG_1/ECOG_2 on baseline and steady state, items still frozen82,917.1930 min · $3.43
  6. v6Full combination: joint all-visit items + bi-exponential + all covariatesCombines both frozen-item screens with joint items. Every later version only re-optimises this model class; 1843 s forces staging.82,314.2462 min · $7.70
  7. v7Block fitting: cached item sweeps alternating with fixed-item trajectory fits82,362.9588 min · $12.25
  8. v8Capped 40-iteration q=10 all-parameter L-BFGS polish after the block fitting82,317.72109 min · $16.60
  9. v9Tri-exponential screen with covariates, initialized to three separated rates82,938.39124 min · $19.54
  10. v10Factor-specific optimisers: block+polish for PhysDys/GI, direct for PsychFactors sit in different basins, so each gets the optimiser that reaches its own. Everything after this is basin selection.82,309.73150 min · $26.91
  11. v11PhysDys switched from block fitting to direct 80-iteration joint fitting82,309.49205 min · $41.35
  12. v12Cohorts under 150 subjects restart once, accepting only a decrease82,309.49215 min · $44.24
  13. v13GI fits direct and block candidates at exact q=10, keeps the lower OFV82,309.49228 min · $48.05
  14. v14Third GI candidate started at (k_slow,k_fast,C)=(0.5,4,2), picked by exact OFVaudit slice only246 min · $53.55
  15. v15Submission hardening: schema and exact-once item assertions, core detectionnot re-scored250 min · $54.68
  16. v16Second PhysDys direct candidate at (0.3,15,0.3), picked by exact OFV82,309.49276 min · $63.63
  17. v17Small-cohort restart cap raised from one to three; frozen and submitted82,309.49322 min · $79.92

Seventeen versions, $85.74 over 5h37. The visible total froze at v10; v11-v17 were judged on a 63-subject audit slice, 15,846.93 -> 15,792.87.

On the hidden set

held-out metricreward
shipped V1 baseline22,905.170.00
strong reference solution19,727.270.30
degenerate floor, rejected below8,0000.00
this run19,635.210.3197
287 minwall clock
$73.77spend
126.1Mtokens
5versions, 5 kept
82.25k 82.50k 82.75k 83.00k 83.25k 83.50k $0 $15 $30 $45 $60 cumulative spend on the run total OFV on the 320 visible subjects, lower is better v1 v2 v3 v4 v5
keptrevertedno scoreturning point
  1. v1Shipped baseline: frozen TIME=0 GRM items, exponential trajectory, no covariates83,465.982 min · $0.38
  2. v2All-visit joint GRM item estimation after a fixed-item exponential warm startFrozen TIME=0 items are the baseline's weak point; refitting them across all 11 visits is worth 732 OFV, on every factor.82,733.5633 min · $3.94
  3. v3Bi-exponential trajectory plus SEX/ECOG effects on baseline and steady stateOne jump to the full model class, curvature plus covariates. Every later version only re-optimises this same class.82,348.6562 min · $8.03
  4. v4Analytic gradient replaces finite differences; joint init from the baseline GRMA hand-derived analytic gradient cuts a fit from 1687 s to 238 s, which is what makes the later multi-start search affordable.82,320.6996 min · $15.23
  5. v5Multi-start exact per-factor selection, triexp candidates, confluent limit form82,309.65225 min · $59.09

Five versions, all kept, $73.77 over 4h47. Structure judged by held-out-subject OFV over folds; a 64-subject audit split picked v5 over v4.

On the hidden set

held-out metricreward
shipped V1 baseline22,905.170.00
strong reference solution19,727.270.30
degenerate floor, rejected below8,0000.00
this run19,657.440.3151
103 minwall clock
-spend
-tokens
4versions, 4 kept
82.25k 82.50k 82.75k 83.00k 83.25k 83.50k 0 30 60 90 120 agent step (this harness reports no tokens or timestamps) total OFV on the 320 visible subjects, lower is better v1 v2 v3 v4
keptrevertedno scoreturning point
  1. v1Shipped baseline: frozen TIME=0 GRM items, exponential trajectory, no covariates83,465.98
  2. v2Analytic-gradient staged fit: SEX/ECOG covariates, shape screen, joint item refineScreen exp/biexp/triexp with items fixed, then refine items jointly, and judge on a 256/64 subject split, not the visible total.tune split only
  3. v3Cap baseline item a to [0.35,2.8]; triexp/biexp screen; L-BFGS restart on maxitRunaway TIME=0 discriminations were the bad basin; bounding them and restarting on maxit is what lets the joint refine finish.82,316.49
  4. v4Two-core mclapply over factors, GRM/EAP fallback, 14-item coverage assert; submittedN=48 smoke only

Four versions, all kept, 1h43, no cost or token data. Reverted offline: GI regrouping, PhysDys split, alternative item init, ucminf polish.

On the hidden set

held-out metricreward
shipped V1 baseline22,905.170.00
strong reference solution19,727.270.30
degenerate floor, rejected below8,0000.00
this run19,684.760.3092
34 minwall clock
$0.43spend
1.7Mtokens
1versions, 1 kept

No trajectory curve: this run left one comparable self-check measurement, so there is nothing to plot against spend. The versions and what each one changed are below.

  1. v1Shipped baseline: frozen TIME=0 GRM items, exponential trajectory, no covariates83,465.983 min · $0.12

Only the baseline was snapshotted, $0.43 over 34 min. A biexp two-step joint-item solver with covariates went into main last and was never re-scored.

On the hidden set

held-out metricreward
shipped V1 baseline22,905.170.00
strong reference solution19,727.270.30
degenerate floor, rejected below8,0000.00
this run19,840.400.2893