RSI-Bench
New release RSI-Bench v0.1

RSI-Bench: Benchmarking Recursive Self-Improvement through Executable Research

It evaluates whether an AI agent can improve itself over a long horizon and generalize to unseen data. Hours of autonomous experimentation on its own method, model weights or agent harness, then one final run on a hidden test set.

Browse tasks How it works GitHub
Overview

The evaluation pipeline

In the working container an agent receives the task instruction and the autoresearch protocol. It then improves its approach round after round, guided by feedback on the visible data. When it is done, the artifact it produced is handed to the verifier container and scored on hidden data.

AGENT IMAGE · VISIBLE DATA Task instruction interface · metric · rules Autoresearch protocol the standing research procedure Agent under evaluation Weak baseline · /app/methods/main rewrite it freely, keep the contract valid Self-check on visible data the only feedback available many rounds VERIFIER IMAGE · SEALED, OFFLINE Fresh container scoring code the agent can never touch Hidden data never entered the agent image reward 0 0.3 0.6 1.0 baseline reference solution SOTA solution upper bound anchors measured end to end by the task author THE SEAL · ONE WAY artifact only
The autoresearch protocol identical for every task
  • Loop and keep the trail. Experiment, log the version, snapshot it, go again. Reverted versions are kept too.
  • Aim at ideas that transfer. The visible score is a proxy. What counts is a method that still holds on data you never saw.
Two axes

Recursive improvement & Generalization

Three artifact types

The method, the model weights, or the agent harness itself. Each task targets one of them.

Method algorithm Model weights Harness scaffold
whichever one you submit, the verifier runs it

One-shot sealed generalization

Iterate on the visible data as long as the budget allows. The score comes from one run on the hidden set.

VISIBLE Development set split it as you like iterate freely self-check any time SEAL artifact HIDDEN Rerun from scratch fresh container, offline data never touched this number is the score
topping the visible set is not the same as improving
Authoring

How a task is built

Every task is written by a domain expert, then implemented and reviewed by other people.

01 · Author

Domain expert

Proposes the problem and the metric, and runs the weak baseline and at least one stronger reference end to end to calibrate the range.

02 · Developer

Implementation

Turns the idea into our required format: two isolated images, a declared artifact contract, sealed data and trusted scoring code.

03 · Reviewer A

Cross-check

Audits value, measurability, data provenance and licensing, and every leakage path that could be looked up or memorised.

04 · Final review

Last look

Re-runs the anchors, and reads a full agent trajectory end to end: did the score move for a real reason, or did the task leak, saturate or collapse to a floor?

Authors are senior PhDs, postdocs, faculty and industry practitioners.

Task bank

Tasks by scenario

Grouped by the problem a task solves. Hover a domain to see the scenarios inside it.

110tasks
15domains
51scenarios
13 Math & Scientific Computing 8 scenarios
Maths & Statistics1
  • One workflow that solves both maths and statistics problemsai4sci_reasoning_workflow_design
Formal Maths1
  • Prove Lean theorems under a per-problem inference budgetlean_formal_proof_workflow_design
Quantum Computing2
  • Reimplement a colour-code decoder without the usual librariesqec_decoder_arena
  • Plan the contraction order for exact circuit equivalence checkingtddn_contraction_planning
Astrophysics2
  • Retrieve exoplanet atmosphere parameters with calibrated error barsexoplanet_transit_spectroscopy
  • Classify transients from multi-band light curvesphotometric_transient_class
Computational Chemistry1
  • Regress the DFT HOMO-LUMO gap from a SMILES stringmolecular_homolumo_gap
Symbolic Regression1
  • Propose an equation family that extrapolates, not just fitssymbolic_regression
Computational Mechanics2
  • Localize stiffness loss in a steel structurefem_stiffness_field_inversion
  • Size every bar of a transmission towerdiscrete_truss_section_sizing
11 Chips & Compute Systems 4 scenarios
GPU Kernels5
  • Decode attention over a paged, ragged KV cachepaged_ragged_gqa_decode_speedup
  • Variable-length attention with the full backward passflashattention_varlen_feature_full_vjp_speedup
  • INT4 weight-only GEMM with a fused inference epiloguew4a16_quantized_gemm_epilogue_speedup
  • Mamba-3 scan with joint states and a full backward passmamba3_siso_longragged_multichunk_joint_states_full_vjp_speedup
  • Gated long convolutions across concurrent streamsflashfftconv_multistream_gated_forward_speedup
Chip Design Automation3
  • Pick a library cell per instance to cut power under timing limitsgate_sizing
  • Relocate, resize and buffer a placed netlist for better PPAincremental_placement
  • Bank and place multi-bit flip-flops at minimum weighted costmbff_banking_placement
Data Systems2
  • Build a filtered ANN index over 10M real vectorsbigann_filtered_vector_search
  • Plan join orders from noisy selectivity estimates alonejoin_order_query_plan_runtime
Device Modelling1
  • Extrapolate power-device I–V curves outside the safe windowdevice_iv_regime_extrapolation
8 LLM Systems & Training 3 scenarios
Agent Systems & Memory4
  • Improve the harness around a frozen base model, not the modelagent_harness_tbench
  • Lift a ReAct harness on the tasks where it scores worstdiscoveryworld_agent_harness_low2
  • Redesign a conversational memory system so it generalizeslocomo_longterm_memory
  • Choose which frames to look at under a fixed evidence budgetvisualclawarena_system_optimization
Inference-Time Compute1
  • Allocate a fixed inference budget across opaque branchesparallel_tts_qwen3
Training Methods & Scaling3
  • Train with GRPO so it survives a compositional distribution shiftclevr_cogent_grpo_qwen2vl
  • Predict the compute-optimal run from smaller onessmall_scaling_law_1xgpu_scaled
  • Distil a larger teacher into a small model by preference optimization plannedmodel_distillation_preference
9 Computer Vision 4 scenarios
Generation & Editing3
  • Fit a ten-evaluation generative solverfew_step_diffusion_sampler_optimization
  • Keep one semantic edit correct across nuisance variationnuisance_equivariant_image_editing
  • Keep paraphrased edit requests mutually consistentparaphrase_worst_case_image_editing
Video Understanding2
  • Select and fuse point-track candidates over a latticerobotap_candidate_lattice_optimization
  • Post-process action proposals with no retraining allowedthumos_temporal_action_localization
3D Reconstruction3
  • Drop Gaussians to a budget while holding novel-view fidelitygaussian_splatting_compression
  • Recover consistent camera poses from noisy correspondencesvggt_multiview_camera_pose
  • Fuse noisy multi-view observations into one point cloudvggt_multiview_dense_reconstruction
Remote Sensing1
  • Reconnect broken road graphs from aerial evidenceaerial_road_connection_repair
12 Operations Research 4 scenarios
Combinatorial Optimization5
  • Minimize conflicts at a fixed color budget on hard DIMACS graphsjohnson1991_leighton_graph_coloring
  • Beat Chvátal-greedy on hidden set-covering instancesorlib_set_covering
  • Schedule cyclic events with standard library onlyperiodic_timetabling_pesplib
  • Cut glass orders from the fewest stock plates on a guillotine lineroadef_glass_cutting
  • Reconstruct a hidden maze through an interactive judgeaedificium_map_recovery
Heuristic Scheduling3
  • Choreograph five terminal cranes that cannot pass each otherterminal_crane_scheduling
  • Lay a commuter rail network one tile per turncommuter_rail_network_design
  • Drive a robot with a record-once, replay-many macro buttonahc066_macro_controller
Black-Box Optimization3
  • Optimize a noisy multimodal function on a fixed query budgetbbo_noisy_continuous_v1
  • Tune an (s,S) inventory policy through a stochastic simulatorbbo_simopt_inventory_v1
  • Ship a self-contained ask/tell optimizerbbo_synthetic_shifted_v1
Optimal Control1
  • Fly a quadrotor through gates in minimum timequadrotor_time_optimal_racing
11 Life Sciences 4 scenarios
Protein Structure & Function4
  • Train and ship a protein-stability regressorpeer_protein
  • Co-fold protein and ligand, inference onlyprotein_ligand_cofolding_posebusters
  • Match TCRs to epitopes never seen in trainingtcr_epitope_unseen_specificity
  • Retrieve protein surfaces by shape descriptorshrec2021
Single-Cell Omics4
  • Annotate lung cell types from five labeled exampleslung_fewshot_celltype_annotation
  • Integrate scATAC batches into one clean embeddingpbmc_batch_correction
  • Predict expression from chromatin accessibility alonepbmc_gene_expression_prediction
  • Annotate cells across datasets with a frozen foundation modelsinglecell_celltype_annotation_cellxgene
Epigenomics1
  • Predict base-resolution chromatin accessibility from DNAatac_seq2atac
Virtual Cell2
  • Predict a cell's transcriptome under an unseen perturbation plannedvirtual_cell_perturbation_response
  • Model how a cell state moves after intervention plannedvirtual_cell_state_transition
5 Health & Medicine 4 scenarios
Physiological Signals1
  • Screen 12-lead ECGs for Chagas diseasephysionet25_chagas_ecg
Clinical QA & Routing2
  • Decide when a medical VQA model should abstainmedvqa_answerability_router
  • Design a multi-agent medical QA workflow under a call budgetmedagents_workflow_design
Imaging & Triage1
  • Triage chest X-rays under a fixed resource capcxr_ood_triage_policy
Trials & Pharmacometrics1
  • Fit a joint IRT-NLME model in pure Rirt_nlme_pro_longitudinal
11 Transport & Logistics 5 scenarios
Signals & Road Networks3
  • Retime ~600 signalised junctions across a whole citycitywide_signal_coordination
  • Choose which links to widen and how to price thembilevel_network_capacity_design
  • Design a bus network that covers every nodetransit_network_mumford
Powertrain Control2
  • Choose the gear every second to burn less fuelgearshift_strategy_wltc
  • Switch a traction inverter above base speedpmsm_fcs_control
Driving & Motion Planning2
  • Race, overtake and defend against scripted opponentshead_to_head_racing_strategy
  • Park an Ackermann car in a tightly constrained slotnarrow_slot_parking_trajectory
Vehicle Routing3
  • Dispatch robotaxis under two-sided time windowsdarp_robotaxi_dispatch
  • Route electric vans that must detour to chargeevrp_electric_fleet_routing
  • Serve 500 paired pickup-delivery requests on timepdptw_paired_request_routing
Rail Dispatching1
  • Dispatch trains through breakdowns you cannot see comingrailway_rescheduling_flatland
9 Earth & Energy 5 scenarios
Weather & Climate2
  • Emulate sub-grid convection for a climate modelclimsim_convection_param
  • Recover 3D wind where the vertical component is underdetermineddual_doppler_wind_retrieval
Geoscience2
  • Classify lithology from wireline well logsforce2020_well_log_lithology
  • Invert seabed friction from sparse tide gaugestidal_friction_inverse
Power Grid1
  • Repair phase labels and impedances on a distribution feederfeeder_phase_impedance_inversion
Building Energy1
  • Predict summer indoor heat from one mild-season recordbuilding_thermal_extreme_extrapolation
Renewables & Storage3
  • Place turbines so wakes cost the least energywindfarm_wake_layout
  • Design a fast-charge current table that spares the cellfast_charge_protocol
  • Split power in a hybrid using only what the car can see nowems_causal_strategy
4 Finance 2 scenarios
Quantitative Investing3
  • Fit an arbitrage-free vol surface from one noisy snapshotiv_surface_noarb_fit
  • Predict cross-sectional returns on an anonymized factor panelqlib_alpha_factor_icir
  • Flag structural breaks in a fleet of seriesstructural_break_detection
Valuation1
  • Forecast value drivers from a messy fundamentals panelfinscope_dcf_valuation
3 Robotics 3 scenarios
Offline RL Manipulation1
  • Learn to push a cube from a fixed offline datasettrifinger_offline_rl_push_refined
Embodied Memory1
  • Give a manipulation policy usable memoryrobomme_memory_policy
World Models1
  • Predict when a world model is about to hallucinate, without labelsmmbench2_hallucination_predictor
3 AI Safety 1 scenario
Jailbreak & Backdoor3
  • Improve a white-box optimizer that breaks a safety classifiersafeguard_jailbreak_autoresearch
3 Software Engineering 1 scenario
Cross-Language Porting3
  • Reimplement the PostgreSQL server over SQLite storage, in Zigpg_wire_sqlite
  • Port libexpat's XML core to hand-written x86-64 assemblylibexpat_x86asm_port
  • Rebuild the Dart formatter in Haskell, byte-for-bytedart_style_haskell
4 Games 1 scenario
Strategy & Puzzles4
  • Play 2048 well on seeds you have never seen2048_strategy
  • Run a transport company for 20 simulated yearsopenttd_transport_ai
  • Solve Sokoban where rooms nest inside themselvesparabox_recursive_solver
  • Survive Battle Tetris with holds, combos and garbagetetris_survival_agent
4 Professional Knowledge Work 2 scenarios
Law2
  • Distil a reusable memory of a GDPR engagementapex_gdpr_compliance_memory
  • Distil a reusable memory of a telemarketing-regulation matterapex_telemarketing_reg_memory