Examples
gate_sizing
Pick a library cell per instance to cut power under timing limits
mean_score · lower is betteragent_harness_tbench
Improve the harness around a frozen base model, not the model
test_resolve_rate · higher is betterqlib_alpha_factor_icir
Predict cross-sectional returns on an anonymized factor panel
icir · higher is betterai4sci_reasoning_workflow_design
One workflow that solves both maths and statistics problems
accuracy_pct · higher is betterdiscrete_truss_section_sizing
Size every bar of a transmission tower
mean over the hidden instances of 100 x weight / weight_baseline · lower is betterexoplanet_transit_spectroscopy
Retrieve exoplanet atmospheres from transmission spectra
mean_gll · higher is betterfem_stiffness_field_inversion
Localize stiffness loss in a steel structure
sealed_fleet_mean_recovery_score · higher is betterlean_formal_proof_workflow_design
Prove Lean theorems under a per-problem inference budget
lean_compile_accuracy_pct · higher is bettermolecular_homolumo_gap
Predict a molecule's HOMO-LUMO gap from its SMILES string
heldout_mae_ev · lower is betterphotometric_transient_class
Classify transients and variables from their light curves
weighted_multiclass_logloss · lower is betterqec_decoder_arena
Reimplement a colour-code decoder without the usual libraries
setting_score · higher is bettershadow_noisy
Design 128 noisy shallow-circuit measurements
macro_family_weight_mse · lower is bettersymbolic_regression
Propose an equation family that extrapolates, not just fits
mean_heldout_extrapolation_accuracy · higher is bettertddn_contraction_planning
Plan the contraction order for exact circuit equivalence checking
combined_log_gain · higher is betterbigann_filtered_vector_search
Build a filtered ANN index over 10M real vectors
qps_at_recall0.90 · higher is betterdevice_iv_regime_extrapolation
Extrapolate power-device I–V curves outside the safe window
sealed_lot_mean_abs_log10_current_error · lower is betterflashattention_varlen_feature_full_vjp_speedup
Variable-length attention with the full backward pass
raw_speedup · higher is betterflashfftconv_multistream_gated_forward_speedup
Gated long convolutions across concurrent streams
raw_speedup · higher is betterincremental_placement
Relocate, resize and buffer a placed netlist for better PPA
mean_leaderboard_score · higher is betterjoin_order_query_plan_runtime
Plan join orders from noisy selectivity estimates alone
mean_true_plan_cost · lower is bettermamba3_siso_longragged_multichunk_joint_states_full_vjp_speedup
Mamba-3 scan with joint states and a full backward pass
raw_speedup · higher is bettermbff_banking_placement
Bank and place multi-bit flip-flops at minimum weighted cost
mean_final_score · lower is betterpaged_ragged_gqa_decode_speedup
Decode attention over a paged, ragged KV cache
raw_speedup · higher is betterw4a16_quantized_gemm_epilogue_speedup
INT4 weight-only GEMM with a fused inference epilogue
raw verifier-owned dynamic-baseline stratified speedup after exact correctness gatesclevr_cogent_grpo_qwen2vl
Train with GRPO so it survives a compositional distribution shift
mean_accuracy_pct · higher is betterdiscoveryworld_agent_harness_low2
Lift a ReAct harness on the tasks where it scores worst
mean_paper_style_scientific_discovery · higher is betterlocomo_longterm_memory
Redesign a conversational memory system so it generalizes
holdout_token_f1 · higher is betterparallel_tts_qwen3_vnext
Spend a fixed evidence budget across opaque branches
mean_panel_group_exact_answer_accuracy · higher is bettersmall_scaling_law_1xgpu_scaled
Predict the compute-optimal run from smaller ones
dist_to_optimal · lower is betterteacher_student_math_posttraining
Post-train a 1.7B student with a fixed 8B teacher
sealed_exact_integer_accuracy · higher is bettervisualclawarena_system_optimization
Choose which frames to look at under a fixed evidence budget
macro_normalized_complete_event_coverage · higher is betteraerial_road_connection_repair
Reconnect broken road graphs from aerial evidence
family_balanced_symmetric_delta_polyline_f1 · higher is betterfew_step_diffusion_sampler_optimization
Fit a ten-evaluation generative solver
fid50k_10nfe_vnext · lower is bettergaussian_splatting_compression
Drop Gaussians to a budget while holding novel-view fidelity
mean_scene_novel_view_psnr_db · higher is betternuisance_equivariant_image_editing
Keep one semantic edit correct across nuisance variation
mean_group_score · higher is betterparaphrase_worst_case_image_editing
Keep paraphrased edit requests mutually consistent
mean_group_score · higher is betterrobotap_candidate_lattice_optimization
Select and fuse point-track candidates over a lattice
mean_video_average_jaccard · higher is betterrobotap_switch_budget_candidate_routing_optimization
Route point tracks through candidates under a switch budget
mean_video_average_jaccard_under_four_switch_budget · higher is betterthumos_temporal_action_localization
Post-process action proposals with no retraining allowed
mean_mAP_tIoU · higher is bettervggt_multiview_camera_pose
Recover consistent camera poses from noisy correspondences
equal_family_mean_pairwise_relative_pose_mAA_1_to_10_degrees · higher is bettervggt_multiview_dense_reconstruction
Fuse noisy multi-view observations into one point cloud
accuracy_gated · lower is bettervideo_temporal_denoising_schedule_optimization_vnext
Tune a five-pass temporal latent refinement policy
balanced_normalized_temporal_latent_mse · lower is betteraedificium_map_recovery
Reconstruct a hidden maze through an interactive judge
mean_rel_cost_vs_champion · higher is betterahc066_macro_controller
Drive a robot with a record-once, replay-many macro button
mean_relative_score_vs_rank1 · higher is betterbbo_noisy_continuous_v1
Optimize a noisy multimodal function on a fixed query budget
oracle_normalized_auc70_final30 · higher is betterbbo_simopt_inventory_v1
Tune an (s,S) inventory policy through a stochastic simulator
oracle_normalized_auc70_final30 · higher is betterbbo_synthetic_shifted_v1
Ship a self-contained ask/tell optimizer
oracle_normalized_auc70_final30 · higher is betterjohnson1991_leighton_graph_coloring
Minimize conflicts at a fixed color budget on hard DIMACS graphs
kpi · lower is betterorlib_set_covering
Beat Chvátal-greedy on hidden set-covering instances
kpi · lower is betterperiodic_timetabling_pesplib
Schedule cyclic events with standard library only
mean_family_normalized_periodic_cost · lower is betterquadrotor_time_optimal_racing
Fly a quadrotor through gates in minimum time
mean_hidden_percent_gap_to_expert_line · lower is betterroadef_glass_cutting
Cut glass orders from the fewest stock plates on a guillotine line
mean_relative_score_vs_reference · higher is betterterminal_crane_scheduling
Choreograph five terminal cranes that cannot pass each other
mean_relative_score_vs_reference · higher is betteratac_seq2atac
Predict base-resolution chromatin accessibility from DNA
mean_per_window_pearson_r · higher is betterlung_fewshot_celltype_annotation
Annotate lung cell types from five labeled examples
macro_f1 · higher is betterpbmc_batch_correction
Integrate scATAC batches into one clean embedding
cell-type NMI · higher is betterpbmc_gene_expression_prediction
Predict expression from chromatin accessibility alone
gene_srcc · higher is betterpeer_protein
Train and ship a protein-stability regressor
heldout_spearman · higher is betterprotein_ligand_cofolding_posebusters
Co-fold protein and ligand, inference only
pb_valid_success_rate · higher is bettershrec2021
Retrieve protein surfaces by shape descriptor
mAP · higher is bettersinglecell_celltype_annotation_cellxgene
Annotate cells across datasets with a frozen foundation model
macro_f1 · higher is bettertcr_epitope_unseen_specificity
Match TCRs to epitopes never seen in training
macro_auc01 · higher is bettercxr_ood_triage_policy
Triage chest X-rays under a fixed resource cap
triage_utility_pct · higher is betterirt_nlme_pro_longitudinal
Fit a joint IRT-NLME model in pure R
heldout_total_OFV · lower is bettermedagents_workflow_design
Design a multi-agent medical QA workflow under a call budget
macro_accuracy_pct · higher is bettermedvqa_answerability_router
Decide when a medical VQA model should abstain
selective_utility_pct · higher is betterbilevel_network_capacity_design
Choose which links to widen and how to price them
mean_hidden_normalised_score · lower is bettercitywide_signal_coordination
Retime ~600 signalised junctions across a whole city
mean_norm_cost · lower is bettercommuter_rail_network_design
Grow a rail network that pays for its own next track
mean_relative_score_vs_reference · higher is betterdarp_robotaxi_dispatch
Dispatch robotaxis under two-sided time windows
mean_hidden_normalised_cost · lower is betterevrp_electric_fleet_routing
Route electric vans that must detour to charge
mean_hidden_normalised_cost · lower is bettergearshift_strategy_wltc
Choose the gear every second to burn less fuel
mean_hidden_percent_fuel_excess_over_dp_bound · lower is betterhead_to_head_racing_strategy
Race, overtake and defend against scripted opponents
mean_hidden_race_cost · lower is betternarrow_slot_parking_trajectory
Park an Ackermann car in a tightly constrained slot
mean over the hidden cases of T_park seconds (invalid or missing -> 240) · lower is betterpdptw_paired_request_routing
Serve 500 paired pickup-delivery requests on time
mean_hidden_normalised_cost · lower is betterpmsm_fcs_control
Switch a traction inverter above base speed
mean_hidden_weighted_episode_cost · lower is betterrailway_rescheduling_flatland
Dispatch trains through breakdowns you cannot see coming
mean_hidden_percent_excess_over_per_train_lower_bound · lower is bettersuspension_kc_hardpoint_design
Place suspension hardpoints and pick the tuning that goes with them
mean_hidden_normalised_score · lower is bettertransit_network_mumford
Design a bus network that covers every node
normalized_demand_weighted_travel_time · lower is betterbuilding_thermal_extreme_extrapolation
Predict summer indoor heat from one mild-season record
sealed_stock_mean_abs_hourly_indoor_temp_error_degC · lower is betterclimsim_convection_param
Emulate sub-grid convection for a climate model
combined 27-channel mean R^2 on the sealed later-block holdout (26 ptend_t levels + PRECC, ClimSim calc_R2, degenerate grid columns masked) · higher is betterdual_doppler_wind_retrieval
Recover 3D wind where the vertical component is underdetermined
wind vector RMSE · lower is betterems_causal_strategy
Split power in a hybrid using only what the car can see now
mean_hidden_percent_gap_to_dp · lower is betterfast_charge_protocol
Design a fast-charge current table that spares the cell
mean_hidden_degradation_loss · lower is betterfeeder_phase_impedance_inversion
Repair phase labels and impedances on a distribution feeder
sealed_fleet_mean_calibration_score · higher is bettertidal_friction_inverse
Invert seabed friction from sparse tide gauges
Mean gauge RMSE · lower is betterwell_log_lithology
Classify lithology facies from wireline well logs
mean_penalty · lower is betterwindfarm_wake_layout
Place turbines so wakes cost the least energy
mean_hidden_normalised_aep · higher is betterfinscope_dcf_valuation
Forecast value drivers from a messy fundamentals panel
MAVE · lower is betteriv_surface_noarb_fit
Fit an arbitrage-free vol surface from one noisy snapshot
sealed_snapshot_mean_composite · higher is betterstructural_break_detection
Flag structural breaks in a fleet of series
sealed_fleet_roc_auc · higher is bettermmbench2_hallucination_predictor
Predict when a world model is about to hallucinate, without labels
rank_corr_spearman · higher is betterrobomme_memory_policy
Give a manipulation policy usable memory
success_rate · higher is bettertrifinger_offline_rl_push_refined
Learn to push a cube from a fixed offline dataset
mean_episode_return · higher is bettersafeguard_jailbreak_autoresearch
Improve a white-box optimizer that breaks a safety classifier
asr_pct · higher is betterdart_style_haskell
Rebuild the Dart formatter in Haskell, byte-for-byte
hidden_pipeline_unit_pass_rate · higher is betterlibexpat_x86asm_port
Port libexpat's XML core to hand-written x86-64 assembly
module_weighted_pass_rate * (0.5 + 0.5*capped_speed_ratio) · higher is betterpg_wire_sqlite
Reimplement the PostgreSQL server over SQLite storage, in Zig
pooled_pass_rate_358 · higher is better2048_strategy
Play 2048 well on seeds you have never seen
per_seed_2048_score · higher is betteropenttd_transport_ai
Run a transport company for 20 simulated years
mean_final_company_value · higher is betterparabox_recursive_solver
Solve Sokoban where rooms nest inside themselves
sealed_level_solve_rate · higher is bettertetris_survival_agent
Survive Battle Tetris with holds, combos and garbage
per_game_battle_performance · higher is betterapex_gdpr_compliance_memory
Distil a reusable memory of a GDPR engagement
held_out_rubric_pass_fraction · higher is betterapex_telemarketing_reg_memory
Distil a reusable memory of a telemarketing-regulation matter
held_out_rubric_pass_fraction · higher is betterbanking_deal_caseload
Answer banking questions from the underlying deal files
held_out_pass_at_1 · higher is betterconsulting_engagement_caseload
Answer consulting questions from the engagement files
held_out_pass_at_1 · higher is betterlegal_matter_caseload_regulatory
Work regulatory legal matters across 1,000+ case documents
held_out_pass_at_1 · higher is betterlegal_matter_caseload_transactional
Work transactional legal matters across 745 case documents
held_out_pass_at_1 · higher is better