Tasks/LLM Systems & Training/Agent Systems & Memory
Design an Agent Harness for Terminal Tasks
Improve the harness around a frozen base model, not the model
agent_harness_tbenchLLM Systems & TrainingAgent Systems & Memory
instruction.mdthis is what the agent is given
A fixed base model is given to you behind a proxy, and a set of terminal tasks it must solve. The harness around that model — everything between a task description arriving and the work being done — is yours to design. You are given one working harness; it is a reference point, not a foundation, and you may keep it, rebuild it, or replace it entirely. Your submission is re-run by a sealed verifier on a hidden set of terminal tasks, and your score is the fraction it resolves.
Hard Constraints
The base model is fixed. All model traffic must go through the per-task proxy at $TBENCH_LLM_BASE_URL (OpenAI-compatible /chat/completions). Calling any model backend directly, or working around the proxy's budget, is cheating and voids the run.
Per-task budget (enforced by the proxy and the runner, identical during grading): max 60 LLM calls, max 400k total tokens, wall-clock capped at 240s per task. Over budget → the proxy returns 429 and the harness must finish up.
Fixed entry contract. The verifier invokes exactly:
python3 /app/methods/main/agent_main.py --instruction-file <file> --workdir <dir>
with env TBENCH_LLM_BASE_URL, TBENCH_LLM_MODEL, TBENCH_SOLVE_DEADLINE, TBENCH_TASK_LOG_DIR. Keep this entry point working. Everything behind it is yours.
Self-contained submission. Everything your harness needs at run time must live inside /app/methods (pure-Python vendoring is fine) or be preinstalled in the image. There is no package installation at run time, in either the agent phase or the verifier: packages you pip-install into the container at agent time do NOT exist in the verifier.
/workspace does not persist. It is wiped and re-materialized for every task run, so harness state kept there is gone by the next task; keep it in /app/methods.
Process model. Your harness is spawned with cwd = the task workdir, runs unprivileged, and its whole process group is killed at the deadline — background daemons do not survive into the test phase. Each task's tests then run separately, in their own venv, from their own declared dependencies.
No per-task hardcoding. Solving logic must be general; the hidden tasks are different from the dev tasks.
What You Have
/app/methods/main/ — the reference harness: agent_main.py (entry point and model callback) plus a small package implementing one particular design. Read it to see what a working answer looks like and what it costs; you are not required to build on it.
/app/tasks_dev/ — the visible dev split of terminal tasks: per task meta.json (instruction, category, timeouts), the baked environment snapshot, and its actual pytest files (tests/, run-tests.sh) — read them to understand how tasks are judged.
/app/selfcheck.py — runs your harness on dev tasks and prints the dev resolve rate:
python3 /app/selfcheck.py (full split) · --ids a,b,c / --n 4 (subsets for fast iteration). Per-task logs (harness stdout, LLM usage, test output) land in /app/selfcheck_logs/.
/app/runner/ — the trusted runner (task materialization into /workspace, proxy, test execution). It is replaced by the verifier's own copy at grading; changing it changes nothing about your score.
What You Submit
The harness in /app/methods (entry contract above). Keep experiment snapshots under /app/methods/versions/ and your log in /app/methods/experiment_log.md; only /app/methods/main/ is executed by the verifier.
How It Is Judged
The verifier re-runs /app/methods/main/agent_main.py on every task of the sealed hidden split under the same budgets, runs each task's hidden tests, and computes the resolve rate (resolved tasks / all tasks). A higher resolve rate is better. An exception inside your harness counts as an unresolved task, not a retry, and a task whose work is unfinished at the deadline scores as unresolved. Dev and hidden tasks are drawn category-stratified from the same pool with matching subcategory proportions, so honest dev-split gains are expected to transfer.
Metric
resolve rate on the sealed 20-task Terminal-Bench 2.0 split · higher is better
resolved / 20 hidden tasks, the harness re-run on a fixed base under 60 calls, 400k tokens, 240 s per task
anchor
visible set
held-out
reward
B
inherited Terminus-2 starter harness
0.30
0.30
0.00
U
every hidden task resolved
—
1.00
1.00
normalisation
m <= B
0
m > B
(m - B) / (U - B)
m = this run's held-out metric · B = inherited Terminus-2 starter harness · U = every hidden task resolved
B=0.30, U=1.00 from the sealed anchors.json. One linear band on the pooled rate, never per case; under 3 tasks making an LLM call fails it.
v1Full rewrite: persistent bash executor, tagged actions, free recon, forced auditReplace the terminal pane with a bounded executor that captures exact output, and end an episode on a self-audit, not a claim.0.531 min · $3.38
v2Old turns replaced by one-line action digests; tiered history; reply cap 8k to 5k0.358 min · $4.71
v3Path checklist in every observation, adaptive prompt allowance, late compact reconaborted after 2 tasks81 min · $7.10
v4History summaries rewritten as prose so the model stops imitating truncated blocks0.388 min · $9.10
v5Never-dropped journal of earlier turns; ~24 wide turns planned instead of ~45 thinCompression dropped every record of the model's own work, so it restarted finished tasks. A never-evicted journal keeps one.0.4123 min · $10.65
v6Mechanical before/after workspace diff at audit time; inline syntax check on writesStop asking the model what it changed and show it a measured diff of the workspace, including files it never authored.0.5158 min · $13.25
v7Three critics fired in parallel at the first audit, findings merged into one fix turn0.4194 min · $14.52
v8Reply cap tied to remaining clock; forced write once 60% of the episode is gone0.4227 min · $16.82
v9Agent asked to write a self-check script the harness re-runs after every turn0/6 subset, v8 got 2/6263 min · $18.67
v10Rollback to the last syntax-clean version of each written file; advisers when stuck0.4292 min · $20.68
v11Salvage cut-off write blocks via append; bigger recon; tolerate 4 proxy failures0.1330 min · $22.95
v12No v12 harness: the parallel requirements-checklist build never entered main/never deployed502 min · $32.23
v13v11 reverted, then salvage re-added only for files that do not exist yet0.5371 min · $26.14
v14Work-loop sampling temperature pinned to 0.25; the advisers keep the default0.5439 min · $29.73
Repeats of unchanged code bracket the noise: v13 measured 0.50 then 0.30, v14 0.50 then 0.40. Plotted is each first run. Ended on API 529.
No trajectory curve: this run left one comparable self-check measurement, so there is nothing to plot against spend. The versions and what each one changed are below.
v1Native tool-calling agent (bash/read/write/edit/submit) replaces the tmux paneGive the model bounded real tools and a workspace survey instead of keystrokes, plus cleanup and no-oracle rules.tuning subset: 4/4
v3Submit gates, forced write of missing files, dump truncation, early-write promptGate the submit on the deliverable existing, and write the file for the model when it never wrote one.hard+extract: 1/4
v4Forced write narrowed to source files; verify-output nudge after the first writehard 0/3, path 0.945
v5Conversation reset after the first write, plus a fragile-speedup nudgepath-tracing fail
v6Reset reverted; ELF strings and symbols in the survey; dumps blocked after a writeportfolio PASS, path 0/1
v7Dump block removed; forced write delayed until tokens or time run lowpath 0.927, still fail
v8Plausible .rodata floats auto-extracted from local ELFs into the surveypath 0.845, gzip over
v9Content-filter recovery and a compressed-size reminder; first full-split run0.4
v10Binary words declared unsigned; survey dropped after 6 turns; tighter compactionextract-elf PASS
v11Unused terminus package deleted from the graded treeconfirm subset: 4/5
v12Non-deliverable gcc outputs deleted on submit, deadline or budget exhaustionTreat the task's file contract as part of the answer and delete build by-products it never asked for.polyglot PASS
Only v9 was measured on the full 10-task dev split (4/10); every other row is a 1-4 task subset. Submitted v12.
v1pty-backed persistent bash replaces tmux keystrokes; history pruned to 4 turnsDrive one real shell through a pty and read its exact output, then prune old turns so an episode fits the token cap.121 min · $0.75
v2write_file action executed in Python; workspace file tree injected in turn 1Let the model name a file and its contents and have the harness write it, so heredoc escaping stops eating turns.not re-measured40 min · $1.13
No full-split run: all three rows are single-task self-checks, v2 unmeasured. The log's 0.0 for v0 belongs to an unsnapshotted draft.
v2Direct bounded shell tools instead of the tmux pane; sliding context, 8k repliesDrop the terminal pane: the model issues bounded shell commands and gets exact output back, plus cleanup and deadline rules.0.57 min · $1.72
v4Exploratory-only command batches rejected after turn 2; adaptive late capshard-2 subset: 0/252 min · $23.27
v5One independent mid-course critic, full trace logs, late diagnostic-only gatesanity 3/3, hard-2 0/260 min · $30.71
v6Critic hard-capped at 1200 tokens; the LLM loop stops with a 20 s reservepair subset: 1/274 min · $41.18
v7First artifact-aware gate; extraction over-included input filenamespath fail at 0.932284 min · $48.95
v8Precise artifact extraction: output triggers no longer leak into input mentionssame fail, sooner88 min · $52.82
v9Persistent distilled notes, anti-hardcoding review, final validate-or-edit gatehard-2 subset: 0/2101 min · $66.31
v108k/65 s generation limits, timeout recovery, direct late runs and writeshard-2 subset: 0/2109 min · $75.03
v11Truncated-command salvage, 60k/6-round context, HTTP 400 prune and retryAttack transport, not prompting: salvage a clipped command, prune and retry on HTTP 400, keep six bounded rounds of context.core subset: 4/4116 min · $82.65
v12Completion caps cut to 6k and critic to 700 tokens; 55 s request timeoutscheduler fail: 3.88e11129 min · $96.40
v138k/65 s implementation capacity retained, critic shortened to 700 tokens0.3134 min · $100.77
v14Review only on evidence; preserve build and runtime products a task may needtargeted pair: 1/2164 min · $135.64
v15Deterministic cleanup only for explicit single-file contracts, new siblings onlyNarrow cleanup from 'delete my by-products' to 'enforce the contract the task states', after v13 deleted a needed shared object.0.3170 min · $138.43
v16Mutations detected before read-prefix filtering; single-file cleanup in a finallyaffected pair: 1/2204 min · $153.73
Timeline stops at v16, 204 min in, where the trajectory is truncated; the log runs to v40, which was submitted. Only v2, v13, v15 saw the full split.
v0Inherited Terminus baseline: full transcript replay, 50-line pane, strict JSON0.31 min · $0.45
v1Rolling bounded context, scrollback deltas, JSON repair, one-call completionStop replaying the whole transcript: carry a rolling bounded context, repair malformed JSON in place, and finish in one call.diverse 4-task: 2/440 min · $9.78
v3One-turn memory, recent-action list, hard implementation prompt after turn 5targeted pair: 1/257 min · $16.37
v4Conservative repeated-analysis action guard after the reconnaissance windowscheduler: 0/1, 1.38e1263 min · $19.02
v5Compound-command classification fixed; one analysis-only action per concrete stepscheduler: 0/1, 4.35e1168 min · $22.08
v6Permanent analysis ban replaced by an every-other-probe throttlescheduler: 0/1, 4.02e1173 min · $24.56
v7Sampling temperature pinned to 0.2 for lower-variance coding decisionsstability subset: 2/379 min · $27.91
v8Verifier and agent dependency environments explained; repeat installs suppressedmodernization: 1/181 min · $29.59
v9Post-completion audit capped at two action-bearing responses0.284 min · $30.89
v10Recency-weighted progress nudge after two inspection-only batches, nothing blockedpath/rev/polyglot: 0/3131 min · $41.68
v11Maximum completion cut from 4096 to 3072 tokens to bound post-action ramblingpolyglot/path: 0/2140 min · $46.01
v12Early return on sub-18 s completions removed so every finish enters the auditpolyglot repeat: 1/1143 min · $47.22
v13Task-generic rule to preserve supplied skeletons, build config and public APIsportfolio slice: 1/2150 min · $49.67
v14Verifier-transfer reminder repeated after any package install attemptportfolio: 0/1, 5/6156 min · $52.45
v15Baked /workspace/.tbench-venv activated in the shell; prepared envs checked firstportfolio: 0/1, 5/6162 min · $56.02
v16Persistent skeleton contract reminder: edit bodies, keep exports and signaturesPreserve the supplied interface - exports, method tables, build files - while filling in bodies, instead of renaming them.0.5168 min · $58.53
v17Per-request transport timeout cut from 80 s to 45 s, one retry retainedscheduler: 0/1, no plan206 min · $76.65
v18Every seventh consecutive diagnostic batch declined; edits and builds untouchedsched/rev/poly: 1/3218 min · $80.87
v19Probe allowance stays exhausted after the first rejection until an edit is triedsched/rev/poly: 1/3229 min · $83.69
v20First late rejection replaced by one bounded senior-strategist blueprint call4-task slice: 1/4240 min · $87.00
v21HTTP 400 content filters handled: clear only the pane, two recoveries allowedpath/rev/elf: 1/3253 min · $91.38
v22Keep a measured baseline after any end-to-end score; change it in bounded stepssched/path/elf: 1/3265 min · $96.17
v23Terminal input echo disabled so heredoc source stops burying build output4-task slice: 2/4283 min · $104.12
v24Every action labelled DELIVERED/POLL/FAILED; silent writes explained in promptpolyglot: 0/1287 min · $108.31
v25Quote-aware detection auto-appends an omitted declared heredoc delimiterpolyglot: 0/1294 min · $112.57
v26Visible input restored; only the heredoc auto-closing kept from the echo branch0.5297 min · $114.69
v27Hybrid late guard: one focused probe after a rejection, then an edit requiredsched/path/poly: 1/3308 min · $116.07
v28Format-first rule: parse magic, dimensions and endianness before raw bytespath/polyglot: 1/2346 min · $126.58
v29tmux foreground-process awareness: polls and interrupts pass, commands waitTell the model when a job is still running so it interrupts or polls, instead of queueing commands into a busy shell.0.5356 min · $130.91
v30Exact verifier metric reproduction required; proxy-metric completion forbiddenpath: 0/1, no image.c361 min · $133.10
v31Empty polls and control keys counted as progress and never declinedpath: 0/1, 3/5 tests366 min · $135.53
v32Mid-object truncation marks a recovery turn and lifts that reply to 4096 tokensregex: 0/1, dormant372 min · $139.66
v33Completion audit checks the exact invocation and the acceptance criterionpath/polyglot: 0/2381 min · $144.00
v34Compact plan kept across content-filter recovery; deliverables protectedpath 0/1, reverse 0/1410 min · $155.84
v35Filter recovery redacts xxd/od/disassembly blocks and keeps safe siblingspath: 0/1, 0.755 sim421 min · $158.82
v36Dedup ordering fixed: observations before the risky block stay authoritativepath: 0/1, dormant427 min · $160.60
v37Deliverable filenames extracted; actions reserved for naming one after 90 spath/rev/regex: 0/3434 min · $163.38
v38In the last 45 s, inspection-only responses withheld; edits and runs allowedpath: 0/1, 4/5 tests448 min · $169.19
v39Deadline-adaptive first-attempt timeout under 100 s left, one attempt reservedpath 0/1, regex 0/4454 min · $172.07
v40Adaptive retry reservation narrowed to 55 s or less remainingpath: 0/1, 4/5 tests465 min · $177.71
v41Retry reservation restricted to the 36-55 s band onlypath: 0/1, 0.933 sim470 min · $180.59
v42Bounded ledger of early concrete facts: formats, dimensions, paths, scorespath: 0/1, 1/5 tests477 min · $183.99
v43Terminal scripts that only print the model's own reasoning prohibitedregex: 0/1, no re.json484 min · $187.35
v448 KB archive of the first two recon observations replayed in every later promptregex: 0/1, no re.json489 min · $188.15
v45Metadata-only workspace snapshots plus a per-turn file create/modify/delete reportregex: 0/1, no re.json495 min · $188.91
v46Replies capped near 6 KB and split into complete executable incrementsregex: 0/1, no re.json501 min · $189.89
v47Establish-the-oracle rule: validate one real call to a supplied evaluator firstscheduler: 0/1, 5/6506 min · $190.94
v48Two candidate replies sampled after three diagnostic batches, action preferredscheduler: 0/1, 4/6512 min · $192.17
v49Edit-intent and checked-completion priority added to candidate rankingscheduler: 0/1, 5/6518 min · $193.48
v50Initial-sample gate fixed so the second candidate pair fires five turns laterscheduler: 0/1, 5/6523 min · $194.67
v51Backend-enforced JSON object response format, parser and loop unchangedpolyglot: 1/1, 180 s528 min · $196.15
v52Free-text JSON replaced by a native terminal_actions function callpolyglot: 0/1532 min · $197.58
v53Post-completion audit cap raised from two to three non-confirming responsespolyglot: 0/1, dormant534 min · $198.57
v54Read-only file inventory run before the first call; broad recon discouragedpath: 0/1, 0.763 sim541 min · $200.67
v55System prompt compressed from 4.9 KB to 2.9 KB, semantics retainedpolyglot: 0/1546 min · $202.57
v56Check-only byproducts marked temporary; finalization audits the exact directory0.5551 min · $204.55
v57Content-filter recovery ceiling raised from two recoveries to four; submittedpath: 0/1, 4/5 tests594 min · $216.03
v58Requests starting with 55 s or less left capped at 1536 tokens0.5609 min · $219.93
v59Acceptance heuristic: requested persistent results over incidental log outputreverse: 0/1, 2/3 tests614 min · $221.12
v60Unclosed replies over 6 KB trigger one bounded truncation-recovery turnregex: 0/1, no re.json621 min · $223.11
v61Native terminal_actions transport plus v56 cleanup and the v58 late capscheduler: 0/1, 1/6629 min · $225.64
Only v0, v9, v16, v26, v29, v56 and v58 saw the full dev split; the rest are 1-4 task subsets. Submitted v57, whose dev 5/10 did not transfer.