Task bank

Tasks

Browse benchmark tasks by category, intro, evaluation setting.

Examples

ai4sci_reasoning_workflow_design

Math & Scientific ComputingMaths & Statistics

One workflow that solves both maths and statistics problems

metricaccuracy_pct · higher is better
1 rollout

discrete_truss_section_sizing

Math & Scientific ComputingComputational Mechanics

Size every bar of a transmission tower

metricmean over the hidden instances of 100 x weight / weight_baseline · lower is better
1 rollout

exoplanet_transit_spectroscopy

Math & Scientific ComputingAstrophysics

Retrieve exoplanet atmospheres from transmission spectra

metricmean_gll · higher is better
2 rollouts

fem_stiffness_field_inversion

Math & Scientific ComputingComputational Mechanics

Localize stiffness loss in a steel structure

metricsealed_fleet_mean_recovery_score · higher is better
1 rollout

lean_formal_proof_workflow_design

Math & Scientific ComputingFormal Maths

Prove Lean theorems under a per-problem inference budget

metriclean_compile_accuracy_pct · higher is better
2 rollouts

molecular_homolumo_gap

Math & Scientific ComputingComputational Chemistry

Predict a molecule's HOMO-LUMO gap from its SMILES string

metricheldout_mae_ev · lower is better
3 rollouts

photometric_transient_class

Math & Scientific ComputingAstrophysics

Classify transients and variables from their light curves

metricweighted_multiclass_logloss · lower is better
3 rollouts

qec_decoder_arena

Math & Scientific ComputingQuantum Computing

Reimplement a colour-code decoder without the usual libraries

metricsetting_score · higher is better
1 rollout

shadow_noisy

Math & Scientific ComputingQuantum Measurement

Design 128 noisy shallow-circuit measurements

metricmacro_family_weight_mse · lower is better
1 rollout

symbolic_regression

Math & Scientific ComputingSymbolic Regression

Propose an equation family that extrapolates, not just fits

metricmean_heldout_extrapolation_accuracy · higher is better
2 rollouts

tddn_contraction_planning

Math & Scientific ComputingQuantum Computing

Plan the contraction order for exact circuit equivalence checking

metriccombined_log_gain · higher is better
2 rollouts

bigann_filtered_vector_search

Chips & Compute SystemsData Systems

Build a filtered ANN index over 10M real vectors

metricqps_at_recall0.90 · higher is better
2 rollouts

device_iv_regime_extrapolation

Chips & Compute SystemsDevice Modelling

Extrapolate power-device I–V curves outside the safe window

metricsealed_lot_mean_abs_log10_current_error · lower is better
2 rollouts

flashattention_varlen_feature_full_vjp_speedup

Chips & Compute SystemsGPU Kernels

Variable-length attention with the full backward pass

metricraw_speedup · higher is better
1 rollout

flashfftconv_multistream_gated_forward_speedup

Chips & Compute SystemsGPU Kernels

Gated long convolutions across concurrent streams

metricraw_speedup · higher is better
1 rollout

incremental_placement

Chips & Compute SystemsChip Design Automation

Relocate, resize and buffer a placed netlist for better PPA

metricmean_leaderboard_score · higher is better

join_order_query_plan_runtime

Chips & Compute SystemsData Systems

Plan join orders from noisy selectivity estimates alone

metricmean_true_plan_cost · lower is better
2 rollouts

mamba3_siso_longragged_multichunk_joint_states_full_vjp_speedup

Chips & Compute SystemsGPU Kernels

Mamba-3 scan with joint states and a full backward pass

metricraw_speedup · higher is better
1 rollout

mbff_banking_placement

Chips & Compute SystemsChip Design Automation

Bank and place multi-bit flip-flops at minimum weighted cost

metricmean_final_score · lower is better
2 rollouts

paged_ragged_gqa_decode_speedup

Chips & Compute SystemsGPU Kernels

Decode attention over a paged, ragged KV cache

metricraw_speedup · higher is better
1 rollout

w4a16_quantized_gemm_epilogue_speedup

Chips & Compute SystemsGPU Kernels

INT4 weight-only GEMM with a fused inference epilogue

metricraw verifier-owned dynamic-baseline stratified speedup after exact correctness gates

clevr_cogent_grpo_qwen2vl

LLM Systems & TrainingTraining Methods & Scaling

Train with GRPO so it survives a compositional distribution shift

metricmean_accuracy_pct · higher is better

discoveryworld_agent_harness_low2

LLM Systems & TrainingAgent Systems & Memory

Lift a ReAct harness on the tasks where it scores worst

metricmean_paper_style_scientific_discovery · higher is better
2 rollouts

locomo_longterm_memory

LLM Systems & TrainingAgent Systems & Memory

Redesign a conversational memory system so it generalizes

metricholdout_token_f1 · higher is better
2 rollouts

parallel_tts_qwen3_vnext

LLM Systems & TrainingInference Scaling

Spend a fixed evidence budget across opaque branches

metricmean_panel_group_exact_answer_accuracy · higher is better
1 rollout

small_scaling_law_1xgpu_scaled

LLM Systems & TrainingTraining Methods & Scaling

Predict the compute-optimal run from smaller ones

metricdist_to_optimal · lower is better

teacher_student_math_posttraining

LLM Systems & TrainingPost-training

Post-train a 1.7B student with a fixed 8B teacher

metricsealed_exact_integer_accuracy · higher is better
1 rollout

visualclawarena_system_optimization

LLM Systems & TrainingAgent Systems & Memory

Choose which frames to look at under a fixed evidence budget

metricmacro_normalized_complete_event_coverage · higher is better

aerial_road_connection_repair

Computer VisionRemote Sensing

Reconnect broken road graphs from aerial evidence

metricfamily_balanced_symmetric_delta_polyline_f1 · higher is better

few_step_diffusion_sampler_optimization

Computer VisionGeneration & Editing

Fit a ten-evaluation generative solver

metricfid50k_10nfe_vnext · lower is better
1 rollout

gaussian_splatting_compression

Computer Vision3D Reconstruction

Drop Gaussians to a budget while holding novel-view fidelity

metricmean_scene_novel_view_psnr_db · higher is better
2 rollouts

nuisance_equivariant_image_editing

Computer VisionGeneration & Editing

Keep one semantic edit correct across nuisance variation

metricmean_group_score · higher is better

paraphrase_worst_case_image_editing

Computer VisionGeneration & Editing

Keep paraphrased edit requests mutually consistent

metricmean_group_score · higher is better

robotap_candidate_lattice_optimization

Computer VisionVideo Understanding

Select and fuse point-track candidates over a lattice

metricmean_video_average_jaccard · higher is better
2 rollouts

robotap_switch_budget_candidate_routing_optimization

Computer VisionVideo Understanding

Route point tracks through candidates under a switch budget

metricmean_video_average_jaccard_under_four_switch_budget · higher is better
2 rollouts

thumos_temporal_action_localization

Computer VisionVideo Understanding

Post-process action proposals with no retraining allowed

metricmean_mAP_tIoU · higher is better
1 rollout

vggt_multiview_camera_pose

Computer Vision3D Reconstruction

Recover consistent camera poses from noisy correspondences

metricequal_family_mean_pairwise_relative_pose_mAA_1_to_10_degrees · higher is better

vggt_multiview_dense_reconstruction

Computer Vision3D Reconstruction

Fuse noisy multi-view observations into one point cloud

metricaccuracy_gated · lower is better
2 rollouts

video_temporal_denoising_schedule_optimization_vnext

Computer VisionVideo Understanding

Tune a five-pass temporal latent refinement policy

metricbalanced_normalized_temporal_latent_mse · lower is better
1 rollout

aedificium_map_recovery

Operations ResearchCombinatorial Optimization

Reconstruct a hidden maze through an interactive judge

metricmean_rel_cost_vs_champion · higher is better

ahc066_macro_controller

Operations ResearchHeuristic Scheduling

Drive a robot with a record-once, replay-many macro button

metricmean_relative_score_vs_rank1 · higher is better
1 rollout

bbo_noisy_continuous_v1

Operations ResearchBlack-Box Optimization

Optimize a noisy multimodal function on a fixed query budget

metricoracle_normalized_auc70_final30 · higher is better
2 rollouts

bbo_simopt_inventory_v1

Operations ResearchBlack-Box Optimization

Tune an (s,S) inventory policy through a stochastic simulator

metricoracle_normalized_auc70_final30 · higher is better
1 rollout

bbo_synthetic_shifted_v1

Operations ResearchBlack-Box Optimization

Ship a self-contained ask/tell optimizer

metricoracle_normalized_auc70_final30 · higher is better
1 rollout

johnson1991_leighton_graph_coloring

Operations ResearchCombinatorial Optimization

Minimize conflicts at a fixed color budget on hard DIMACS graphs

metrickpi · lower is better
1 rollout

orlib_set_covering

Operations ResearchCombinatorial Optimization

Beat Chvátal-greedy on hidden set-covering instances

metrickpi · lower is better

periodic_timetabling_pesplib

Operations ResearchCombinatorial Optimization

Schedule cyclic events with standard library only

metricmean_family_normalized_periodic_cost · lower is better

quadrotor_time_optimal_racing

Operations ResearchOptimal Control

Fly a quadrotor through gates in minimum time

metricmean_hidden_percent_gap_to_expert_line · lower is better
1 rollout

roadef_glass_cutting

Operations ResearchCombinatorial Optimization

Cut glass orders from the fewest stock plates on a guillotine line

metricmean_relative_score_vs_reference · higher is better
2 rollouts

terminal_crane_scheduling

Operations ResearchHeuristic Scheduling

Choreograph five terminal cranes that cannot pass each other

metricmean_relative_score_vs_reference · higher is better
2 rollouts

atac_seq2atac

Life SciencesEpigenomics

Predict base-resolution chromatin accessibility from DNA

metricmean_per_window_pearson_r · higher is better
1 rollout

lung_fewshot_celltype_annotation

Life SciencesSingle-Cell Omics

Annotate lung cell types from five labeled examples

metricmacro_f1 · higher is better

pbmc_batch_correction

Life SciencesSingle-Cell Omics

Integrate scATAC batches into one clean embedding

metriccell-type NMI · higher is better
1 rollout

pbmc_gene_expression_prediction

Life SciencesSingle-Cell Omics

Predict expression from chromatin accessibility alone

metricgene_srcc · higher is better
1 rollout

peer_protein

Life SciencesProtein Structure & Function

Train and ship a protein-stability regressor

metricheldout_spearman · higher is better
1 rollout

protein_ligand_cofolding_posebusters

Life SciencesProtein Structure & Function

Co-fold protein and ligand, inference only

metricpb_valid_success_rate · higher is better
1 rollout

shrec2021

Life SciencesProtein Structure & Function

Retrieve protein surfaces by shape descriptor

metricmAP · higher is better

singlecell_celltype_annotation_cellxgene

Life SciencesSingle-Cell Omics

Annotate cells across datasets with a frozen foundation model

metricmacro_f1 · higher is better
1 rollout

tcr_epitope_unseen_specificity

Life SciencesProtein Structure & Function

Match TCRs to epitopes never seen in training

metricmacro_auc01 · higher is better
1 rollout

cxr_ood_triage_policy

Health & MedicineImaging & Triage

Triage chest X-rays under a fixed resource cap

metrictriage_utility_pct · higher is better
2 rollouts

irt_nlme_pro_longitudinal

Health & MedicineTrials & Pharmacometrics

Fit a joint IRT-NLME model in pure R

metricheldout_total_OFV · lower is better
2 rollouts

medagents_workflow_design

Health & MedicineClinical QA & Routing

Design a multi-agent medical QA workflow under a call budget

metricmacro_accuracy_pct · higher is better
1 rollout

medvqa_answerability_router

Health & MedicineClinical QA & Routing

Decide when a medical VQA model should abstain

metricselective_utility_pct · higher is better

bilevel_network_capacity_design

Transport & LogisticsSignals & Road Networks

Choose which links to widen and how to price them

metricmean_hidden_normalised_score · lower is better
2 rollouts

citywide_signal_coordination

Transport & LogisticsSignals & Road Networks

Retime ~600 signalised junctions across a whole city

metricmean_norm_cost · lower is better
2 rollouts

commuter_rail_network_design

Transport & LogisticsRail Planning

Grow a rail network that pays for its own next track

metricmean_relative_score_vs_reference · higher is better
2 rollouts

darp_robotaxi_dispatch

Transport & LogisticsVehicle Routing

Dispatch robotaxis under two-sided time windows

metricmean_hidden_normalised_cost · lower is better
2 rollouts

evrp_electric_fleet_routing

Transport & LogisticsVehicle Routing

Route electric vans that must detour to charge

metricmean_hidden_normalised_cost · lower is better
2 rollouts

gearshift_strategy_wltc

Transport & LogisticsPowertrain Control

Choose the gear every second to burn less fuel

metricmean_hidden_percent_fuel_excess_over_dp_bound · lower is better
1 rollout

head_to_head_racing_strategy

Transport & LogisticsDriving & Motion Planning

Race, overtake and defend against scripted opponents

metricmean_hidden_race_cost · lower is better
2 rollouts

narrow_slot_parking_trajectory

Transport & LogisticsDriving & Motion Planning

Park an Ackermann car in a tightly constrained slot

metricmean over the hidden cases of T_park seconds (invalid or missing -> 240) · lower is better
1 rollout

pdptw_paired_request_routing

Transport & LogisticsVehicle Routing

Serve 500 paired pickup-delivery requests on time

metricmean_hidden_normalised_cost · lower is better
1 rollout

pmsm_fcs_control

Transport & LogisticsPowertrain Control

Switch a traction inverter above base speed

metricmean_hidden_weighted_episode_cost · lower is better
1 rollout

railway_rescheduling_flatland

Transport & LogisticsRail Dispatching

Dispatch trains through breakdowns you cannot see coming

metricmean_hidden_percent_excess_over_per_train_lower_bound · lower is better
1 rollout

suspension_kc_hardpoint_design

Transport & LogisticsVehicle Dynamics

Place suspension hardpoints and pick the tuning that goes with them

metricmean_hidden_normalised_score · lower is better
2 rollouts

transit_network_mumford

Transport & LogisticsSignals & Road Networks

Design a bus network that covers every node

metricnormalized_demand_weighted_travel_time · lower is better
1 rollout

building_thermal_extreme_extrapolation

Earth & EnergyBuilding Energy

Predict summer indoor heat from one mild-season record

metricsealed_stock_mean_abs_hourly_indoor_temp_error_degC · lower is better
2 rollouts

climsim_convection_param

Earth & EnergyWeather & Climate

Emulate sub-grid convection for a climate model

metriccombined 27-channel mean R^2 on the sealed later-block holdout (26 ptend_t levels + PRECC, ClimSim calc_R2, degenerate grid columns masked) · higher is better
2 rollouts

dual_doppler_wind_retrieval

Earth & EnergyWeather & Climate

Recover 3D wind where the vertical component is underdetermined

metricwind vector RMSE · lower is better
2 rollouts

ems_causal_strategy

Earth & EnergyRenewables & Storage

Split power in a hybrid using only what the car can see now

metricmean_hidden_percent_gap_to_dp · lower is better
1 rollout

fast_charge_protocol

Earth & EnergyRenewables & Storage

Design a fast-charge current table that spares the cell

metricmean_hidden_degradation_loss · lower is better
1 rollout

feeder_phase_impedance_inversion

Earth & EnergyPower Grid

Repair phase labels and impedances on a distribution feeder

metricsealed_fleet_mean_calibration_score · higher is better
1 rollout

tidal_friction_inverse

Earth & EnergyGeoscience

Invert seabed friction from sparse tide gauges

metricMean gauge RMSE · lower is better
2 rollouts

well_log_lithology

Earth & EnergySubsurface Interpretation

Classify lithology facies from wireline well logs

metricmean_penalty · lower is better
1 rollout

windfarm_wake_layout

Earth & EnergyRenewables & Storage

Place turbines so wakes cost the least energy

metricmean_hidden_normalised_aep · higher is better
1 rollout

finscope_dcf_valuation

FinanceValuation

Forecast value drivers from a messy fundamentals panel

metricMAVE · lower is better
1 rollout

iv_surface_noarb_fit

FinanceQuantitative Investing

Fit an arbitrage-free vol surface from one noisy snapshot

metricsealed_snapshot_mean_composite · higher is better
2 rollouts

structural_break_detection

FinanceQuantitative Investing

Flag structural breaks in a fleet of series

metricsealed_fleet_roc_auc · higher is better
2 rollouts

mmbench2_hallucination_predictor

RoboticsWorld Models

Predict when a world model is about to hallucinate, without labels

metricrank_corr_spearman · higher is better
2 rollouts

robomme_memory_policy

RoboticsEmbodied Memory

Give a manipulation policy usable memory

metricsuccess_rate · higher is better
1 rollout

trifinger_offline_rl_push_refined

RoboticsOffline RL Manipulation

Learn to push a cube from a fixed offline dataset

metricmean_episode_return · higher is better
2 rollouts

safeguard_jailbreak_autoresearch

AI SafetyJailbreak & Backdoor

Improve a white-box optimizer that breaks a safety classifier

metricasr_pct · higher is better

dart_style_haskell

Software EngineeringCross-Language Porting

Rebuild the Dart formatter in Haskell, byte-for-byte

metrichidden_pipeline_unit_pass_rate · higher is better
2 rollouts

libexpat_x86asm_port

Software EngineeringCross-Language Porting

Port libexpat's XML core to hand-written x86-64 assembly

metricmodule_weighted_pass_rate * (0.5 + 0.5*capped_speed_ratio) · higher is better
2 rollouts

pg_wire_sqlite

Software EngineeringCross-Language Porting

Reimplement the PostgreSQL server over SQLite storage, in Zig

metricpooled_pass_rate_358 · higher is better
1 rollout

2048_strategy

GamesStrategy & Puzzles

Play 2048 well on seeds you have never seen

metricper_seed_2048_score · higher is better
2 rollouts

openttd_transport_ai

GamesStrategy & Puzzles

Run a transport company for 20 simulated years

metricmean_final_company_value · higher is better
2 rollouts

parabox_recursive_solver

GamesStrategy & Puzzles

Solve Sokoban where rooms nest inside themselves

metricsealed_level_solve_rate · higher is better
2 rollouts

tetris_survival_agent

GamesStrategy & Puzzles

Survive Battle Tetris with holds, combos and garbage

metricper_game_battle_performance · higher is better
1 rollout

apex_gdpr_compliance_memory

Professional Knowledge WorkLaw

Distil a reusable memory of a GDPR engagement

metricheld_out_rubric_pass_fraction · higher is better

apex_telemarketing_reg_memory

Professional Knowledge WorkLaw

Distil a reusable memory of a telemarketing-regulation matter

metricheld_out_rubric_pass_fraction · higher is better

banking_deal_caseload

Professional Knowledge WorkInvestment Banking

Answer banking questions from the underlying deal files

metricheld_out_pass_at_1 · higher is better
1 rollout

consulting_engagement_caseload

Professional Knowledge WorkManagement Consulting

Answer consulting questions from the engagement files

metricheld_out_pass_at_1 · higher is better
1 rollout

legal_matter_caseload_regulatory

Professional Knowledge WorkLegal

Work regulatory legal matters across 1,000+ case documents

metricheld_out_pass_at_1 · higher is better
1 rollout

legal_matter_caseload_transactional

Professional Knowledge WorkLegal

Work transactional legal matters across 745 case documents

metricheld_out_pass_at_1 · higher is better
1 rollout