RSI-Bench
Tasks/Software Engineering/Cross-Language Porting

libexpat in hand-written x86-64 assembly

Port libexpat's XML core to hand-written x86-64 assembly

libexpat_x86asm_port Software Engineering Cross-Language Porting
instruction.mdthis is what the agent is given

Reimplement the XML parsing core of libexpat R_2_6_4 — the public API surface implemented by xmlparse.c, xmlrole.c and xmltok.c — as a shared library written entirely in hand-written x86-64 assembly, and make it fast.

The unmodified upstream C source is provided at /app/upstream/expat/lib/ as a read-only behavioural specification, together with the fixed build configuration /app/upstream/expat_config.h (XML_NS, XML_DTD, XML_GE=1, XML_CONTEXT_BYTES=1024, XML_TESTING, little-endian). Your library must reproduce the behaviour of that C code under exactly that configuration.

Deliverable

/app/asm-port/ containing:

  • GNU assembler sources (*.s, optionally *.S; GNU as syntax, no preprocessing) implementing the library.
  • build.sh — already provided; it performs the canonical build: every .s/.S file under /app/asm-port/ sorted by path, each assembled with as --64, all objects linked with ld -shared -soname libexpat.so -o libexpat.so. The verifier performs exactly this sequence itself, from your sources — it does not run your build.sh. Anything your build needs beyond that sequence will not exist at grading time.

The built libexpat.so must export the complete public ABI — every XML_* function declared in expat.h under this configuration (69 symbols). The starter file anchor_min.s already exports all of them as stubs; grow it into a real implementation rather than deleting exports. A library missing public symbols fails the test-driver link and scores 0.

Your library may call libc functions (malloc, memcpy, …): leave them as undefined symbols and the dynamic loader resolves them at run time.

Hard rules (checked mechanically; any violation scores 0)

  1. At least one assembly source file must exist under /app/asm-port/.
  2. No C source files under /app/asm-port/ — this is a port, not a repackaging.
  3. The assembly must not reference the dlopen/dlsym family or RTLD_ constants outside comments — no delegating the work to another library at run time.
  4. The built .so must not depend on another expat: no NEEDED entry matching libexpat and no embedded path string pointing at a system libexpat.
  5. The built .so must not carry a compiler-generated .comment section — hand-assembled objects have none; compiled C leaves one. (There is no C compiler in this environment, and the verifier rebuilds only with as+ld, so compiled code cannot enter the submission anyway.)

How you are graded

score = correctness × (0.5 + 0.5 × performance), graded by a separate verifier you never see, against the complete test pipeline of the unmodified upstream release. Performance only counts for what you have implemented correctly: a fully correct port at reference speed scores 1.0, a fully correct but very slow one 0.5, and a half-correct one caps near its pass rate no matter how fast it is.

Correctness — the verifier links upstream libexpat R_2_6_4's own full test suite (several hundred registered test cases across six driver modules) against your .so and runs the whole suite 12 times (parse-chunk sizes 0–5 × reparse-deferral on/off). A test case counts as passed only when all 12 iterations pass. Scoring is a weighted mean of per-module pass rates (weights: basic 3, namespace 2, misc 1, allocation 2, ns-allocation 1; the internal byte-accounting module runs but is not scored). The denominator is fixed by a static scan of the suite — a test that never ran counts as failed. Each test case runs in its own isolated process with a watchdog: a crash or hang in one test fails only that test, the rest of the suite is still graded (your local test runner isolates the same way).

Performance — upstream's own benchmark.c (source at /app/bench/benchmark.c), buffer size 8192, on three deterministic XML corpora (/app/bench/data/, generator included): small ~0.5 KB, medium ~169 KB, large ~1.4 MB. Per tier the verifier measures reference-C time and your time the same way (pinned to one CPU, strictly serial, median of 5; every tier is timed as whole-process CPU time measured by the verifier, not by anything the benchmark process prints) and takes min(1.0, ref_time / your_time) — parity with the reference C build is full marks, beating it earns no extra. Tier weights: small 1, medium 1, large 2. The performance component is multiplied by 0.5 for each graded binary that terminates abnormally (a benchmark tier that crashes/times out also scores ratio 0). Because the total is multiplicative, speed is worth nothing beyond what your correctness has earned.

Visible vs hidden tests

/app/vtests/runtests_visible is a precompiled driver holding 129 test cases from all five of the suite's modules: 102 basic tests (a success-path ladder — well-formed documents over UTF-8, UTF-16 BE/LE and Latin-1, character references, CDATA, DTD and entity paths, suspend/resume, unknown-encoding handlers, lifecycle and position-reporting APIs — plus a hard slice of error-path and corner-case tests: malformed input and bad encodings, error codes and positions, abort/suspend in edge states, buffer API edges, DTD/entity corners), 11 namespace-processing tests, 5 misc API tests, and 11 allocation-failure robustness tests (8 alloc / 3 nsalloc: they drive your allocator entry points to fail at every possible point and expect clean recovery without crashes or leaks of parser state). These 129 are a strict subset of the hidden suite — the hidden pipeline is a superset containing them plus a couple hundred more corner-case tests it does not show you, across the same families and beyond: entity expansion defenses, reparse-deferral machinery, oversized-token defenses, and the bulk of the allocator failure-injection matrix. The visible 129 carry ~30% of the correctness weight and deliberately contain a long tail of hard tests: the success-path rungs come first, but a strong port is expected to keep finding failing visible tests to fix for a long time — full visible score is neither required nor expected for a good result, and most of your score still comes from how much of the rest of the library you actually implement correctly. The C source is the complete specification — what the hidden tests check is exactly "does your port behave like the C code". Like the hidden pipeline, the visible driver runs each test in its own isolated process, so a crash in one test fails only that test and the run completes.

Run python3 /app/selfcheck.py any time: it does the canonical build, runs the visible driver (per-test 12-iteration results), and times the three benchmark corpora with your library (reference timings are measured verifier-side only).

Environment

  • Tools: GNU binutils (as, ld, objdump, readelf, nm), gdb, strace, make, python3. There is no C compiler. No network.
  • 4 CPUs, 1 GB RAM. The run ends after 6 hours; the verifier runs after that.
  • Grading budget (stated so you can size your work; all values are the verifier's, not your own container's):
Stage Verifier budget Your selfcheck.py
canonical build (as per file, then ld) 120 s each same sequence, build.sh
anti-cheat checks 300 s not run locally
linking the test drivers against your .so 600 s precompiled
full test suite (362 cases × 12 iterations) 900 s total, 15 s watchdog per test case 600 s total for the 129 visible cases
benchmark, per tier per run 90 s, 5 interleaved cycles 120 s, 3 runs
whole verifier stage 3600 s wall-clock, 4 CPUs / 1 GiB agent container is also 4 CPUs / 1 GiB

Two of those are looser locally than at grading: your visible run gets 600 s for 129 cases (4.6 s/case) where grading gives 900 s for 362 cases (2.5 s/case), and your benchmark gets 120 s per run where grading gives 90 s. Do not tune to the local wall. A test case that is killed by the 15 s watchdog, and a benchmark tier that is killed by the 90 s cap, both count exactly as a failure of that case/tier — never as a grader error. - Each graded child process may write at most 64 MiB to stdout and 64 MiB to stderr; beyond that the verifier kills it and counts every test that had not reported yet as failed. A normal full run prints well under 1 MiB, so this only ever bites runaway output. - The test drivers define two test-support globals (g_reparseDeferralEnabledDefault, g_bytesScanned) and fall-back stubs for the XML_TESTING-internal helpers, exactly as a C build under this configuration interacts with them (see internal.h and the XML_TESTING-guarded code in xmlparse.c). Your library reaches full marks on the deferral-measurement tests only by honouring those globals the way the C code does (import them as undefined symbols; they resolve to the test executable's definitions).

Work autonomously; do not stop to ask for input.

Metric

module-weighted hidden pass rate x capped speed factor · higher is better

corr*(0.5+0.5*perf); corr = weighted pass rate (basic 3, ns 2, misc 1, alloc 2, nsalloc 1) over 362 cases

anchorvisible setheld-outreward
Ball-stub anchor_min.s (starting point)0.0077520.0098040.00
Ustructural bound (all pass, ref speed)1.01.01.00
normalisation
m <= B0
B < m < U(m - B) / (U - B)
m >= U1

m = this run's held-out metric  ·  B = all-stub anchor_min.s (starting point)  ·  U = structural bound (all pass, ref speed)

B=0.009804, U=1.0. One clipped linear segment; the 0.3/0.6 anchors are deliberately absent. Mapped once on the whole suite, not per module.

Rollouts

215 minwall clock
$88.13spend
128.0Mtokens
53versions, 53 kept
0 30 60 90 120 $0 $20 $40 $60 $80 cumulative spend on the run visible-suite tests passing (of 129), higher is better v0 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 v25 v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41 v42 v43 v44 v45 v46 v47 v48 v49 v50 v51 v52
keptrevertedno scoreturning point
  1. v0Inherited baseline: 69 zero-valued stubs; all three benchmark tiers crash11 min · $0.23
  2. v169-symbol ABI layer: lifecycle, allocator suite, handlers; tokenizer still a stubFix the 69-symbol ABI, parser state and allocator contract first, before any scanner exists to drive them.98 min · $1.48
  3. v2Streaming UTF-8/Latin-1 tokenizer, element stack, refs, CDATA, suspend/resume5216 min · $2.52
  4. v3Buffer capacity kept across realloc; partial UTF-8 and entity tails held per chunk5417 min · $3.35
  5. v4Error enum tail, XML name ranges, XML declaration grammar, unknown-encoding maps6426 min · $7.94
  6. v5Retained UTF-16 LE/BE path with BOM detection and surrogate validation7433 min · $11.00
  7. v6Declaration scanner made quote- and bracket-aware; internal subsets survive chunking7935 min · $11.96
  8. v7Internal general-entity declarations retained; entity callbacks, lookup and expansion8442 min · $13.59
  9. v8ATTLIST type tables, tokenized-value whitespace collapse, ID-attribute indexes8645 min · $14.47
  10. v9Scoped namespace bindings, QName expansion, triplets, reverse end-namespace callbacks9349 min · $16.27
  11. v10ATTLIST defaults applied before namespace expansion; defaulted xmlns become bindings9451 min · $16.89
  12. v11Pool growth aligned with the configured realloc suite; ASCII data now alloc-free9958 min · $21.63
  13. v12Final/mismatch byte metadata, errors kept on retry, CDATA default events split10765 min · $24.09
  14. v13Context-sensitive UTF truncation errors, CRLF accounting, DefaultCurrent and skipping11275 min · $29.90
  15. v14External general entities: metadata, ref callbacks, inherited child-parser state11584 min · $33.27
  16. v15External TextDecl split from XML declaration; protocol encoding wins over the entity11685 min · $33.43
  17. v16Validated DOCTYPE header: name, system and public IDs, declaration callbacks11788 min · $34.03
  18. v17NULL-context children read external parameter subsets; entities owned by parent12290 min · $34.55
  19. v18Standalone state retained; the undeclared-entity relaxation limited to non-standalone12390 min · $34.71
  20. v19Foreign-DTD last-chance read at the first element, with the not-standalone veto12491 min · $34.89
  21. v20XmlDecl and TextDecl handler with owned strings and the standalone tri-state12593 min · $35.62
  22. v21Entity lookup separates parameter, internal, external parsed and unparsed entities12695 min · $36.98
  23. v22Parameter references inside declaration values; nested external-PE callbacks12796 min · $37.32
  24. v23External-PE IGNORE scanner: nesting, UTF-8 validation, exact partial-section errors12898 min · $37.87
  25. v24Resumable internal-entity input frame; suspend and resume inside replacement textA frame that tokenizes replacement text through the parent parser makes suspension work anywhere, not only in document data.129102 min · $39.26
  26. v25Per-entity open tracking; recursive-reference error 12, unparsed reference error 15129105 min · $41.25
  27. v26UnparsedEntityDeclHandler dispatch and its precedence over EntityDeclHandler129106 min · $41.54
  28. v27NOTATION declarations in all three forms; PubidChar validation and normalization129108 min · $42.02
  29. v28PubidChar rules extended to external ENTITY and DOCTYPE public identifiers129108 min · $42.63
  30. v29Nested NULL-context external PEs share one DTD; child parsers inherit parser state129111 min · $44.06
  31. v30Reserved xml and xmlns prefix and URI rules with exact errors 28/38/39/40/2129114 min · $45.66
  32. v31Second colon rejected in declaration prefixes and expanded QNames, per NCName129114 min · $46.09
  33. v32Namespace bindings inherited across external general-entity parser chains129117 min · $46.54
  34. v33ParserReset reinitializes DTD, namespace, security and reparse state129122 min · $47.87
  35. v34Buffer and state API edges: negative GetBuffer, suspended ParseBuffer, root hash salt129124 min · $48.67
  36. v35DTD default attributes applied across parent chains, with local-first shadowing129125 min · $49.10
  37. v36Grammar-aware XML_Content builder: MIXED, SEQ, CHOICE, nested groups, quantifiers129133 min · $50.86
  38. v37Entity amplification accounting shared at the root; threshold and limit errors129137 min · $52.11
  39. v38First-declaration-wins entities; each entity keeps its declaration-time base URI129142 min · $54.83
  40. v39Serialized external-entity contexts; a qword error check read the adjacent field128150 min · $59.80
  41. v40Dispatch error check narrowed to its 32-bit field; contexts and imports retained129151 min · $60.32
  42. v41Markup-bearing internal entities nest by saving and reinstalling the outer frame129153 min · $61.63
  43. v42ATTLIST semantics completed: types, enumerations, defaults, AttlistDeclHandler129161 min · $63.35
  44. v43Differential runs against reference expat on 1,053 documents: 482 to 450 mismatchesStopped grading itself on the visible 129: documents diffed against a reference expat expose accepts the suite never probes.129170 min · $66.38
  45. v44Exact DTD keyword dispatch; unknown markup declarations no longer silently accepted129172 min · $67.61
  46. v45DeclSep, entity-value PE rules and pseudo-attribute grammar: 30 accept diffs to 0129176 min · $70.10
  47. v46Streaming external-DTD INCLUDE sections with nesting and chunk-boundary retention129179 min · $72.21
  48. v47Internal parameter-entity expansion: root-shared tables, dispatch, recursion error 12129183 min · $73.94
  49. v48Root parser kept as handler arg during PE expansion; PE refs inside entity values129190 min · $75.96
  50. v49QName shape, duplicate xmlns, triplet duplicates: 136-run namespace corpus to zero129200 min · $79.75
  51. v50Replacement text normalized at declaration time; PE refs in external entity values129207 min · $83.02
  52. v51Expat's doubling heuristic for deferred reparse: 30 KB attribute 0.61 s to 0.038 s129211 min · $85.32
  53. v52Unknown-encoding callback gets a stack-initialized XML_Encoding before allocation129214 min · $87.72

53 versions, 3.6 h, $88. 129/129 visible by v24 (105 min, $41); the last 28 moved no visible test and aimed at the hidden set.

On the hidden set

held-out metricreward
all-stub anchor_min.s (starting point)0.0098040.00
structural bound (all pass, ref speed)1.01.00
this run0.89590.8948
taskhttps://huggingface.co/datasets/autoresearch-arena/rsi-tasks-1/tree/main/libexpat_x86asm_portrollouthttps://huggingface.co/datasets/autoresearch-arena/rsi-rollout/tree/main/libexpat_x86asm_port/codex-gpt-5.6-sol-max/libexpat_x86asm_port__Wyg69Go
68 minwall clock
$33.74spend
50.9Mtokens
9versions, 9 kept
0 25 50 75 100 125 $0 $7.5 $15 $22 $30 cumulative spend on the run visible-suite tests passing (of 129), higher is better v0 v1 v2 v3 v4 v5 v6 v7 v8
keptrevertedno scoreturning point
  1. v0Inherited baseline: 69 all-zero public stubs; benchmarks crash on all three tiers10 min · $0.15
  2. v1Allocator-aware parser object, full API surface, append-only Parse, no tokenizer yetBuilt the object, memory suite and every setter/getter first, without a scanner: the semantics the whole suite links against.385 min · $0.91
  3. v2Final-buffer XML scanner with UTF-8 events plus UTF-16 LE/BE and Latin-1 transcodingThe parsing engine, with encoding conversion built in rather than bolted on later: basic and misc then moved together.5616 min · $4.03
  4. v3Post-attribute tag transitions, attribute normalisation, IDs, error positions7321 min · $7.11
  5. v4Scoped namespaces and triplets, fast tag storage, UTF validation, entity tables8938 min · $15.56
  6. v5Entity-declaration callbacks, attribute reference expansion, XML declaration grammar9546 min · $17.65
  7. v6Strict encoding/name errors, external subset and foreign DTD, event boundaries+22 visible tests at once across the error, DTD and namespace families, with the new DTD scans kept off the content hot path.11764 min · $30.55
  8. v7Raw epilog tokens preserved for default-only parsing; CR abort semantics fixed11866 min · $32.91
  9. v8CR is valid epilog whitespace, so default-handler aborts see the raw token11867 min · $33.19

Nine versions, none reverted: 1 -> 118/129 in 67 minutes and $33. The last two were single-behaviour fixes worth one test between them.

On the hidden set

held-out metricreward
all-stub anchor_min.s (starting point)0.0098040.00
structural bound (all pass, ref speed)1.01.00
this run0.70830.7054
taskhttps://huggingface.co/datasets/autoresearch-arena/rsi-tasks-1/tree/main/libexpat_x86asm_portrollouthttps://huggingface.co/datasets/autoresearch-arena/rsi-rollout/tree/main/libexpat_x86asm_port/codex-gpt-5.6-sol-high/libexpat_x86asm_port__5j4FH2o
220 minwall clock
$172.90spend
279.7Mtokens
44versions, 42 kept
75 90 105 120 $0 $40 $80 $120 $160 cumulative spend on the run visible-suite tests passing (of 129), higher is better v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 v13 v14 v15 v16 v17 v18 v19 v20 v21 v22 v23 v24 v25 v26 v27 v28 v29 v30 v31 v32 v33 v34 v35 v36 v37 v38 v39 v40 v41 v42 v43 v44
keptrevertedno scoreturning point
  1. v1First full port from its own code generator: tokenizer, processors, 69 entry points7732 min · $9.00
  2. v2Fixed overlapping temp slots, PLT interposition, stale encoding pointer9037 min · $12.40
  3. v3Namespace mode enabled, xml prefix bound at create, text decls in external entities9738 min · $13.29
  4. v4External-entity handler signature; expat's negative-token rule in the prolog scanner10042 min · $15.68
  5. v5userData at parser offset 0; ATTLIST defaults, external subset, conditional sectionsXML_GetUserData is a macro reading *(void**)parser, so userData must be the first struct member. Only expat.h states that.10965 min · $24.74
  6. v6Full DTD declaration machine: entity/attlist/notation handlers, content models11472 min · $28.47
  7. v7Performance pass: per-encoding scan bitmaps, reusable tag buffers, codegen inlining11587 min · $36.37
  8. v8Tag scanner records name and attribute extents; handler layer stops rescanning11589 min · $37.69
  9. v9Call-site guards for the no-attribute and no-DTD element paths; scan_lt inlined11590 min · $37.99
  10. v10Open-internal-entity stack, so expansion can suspend and resume as expat does11894 min · $40.81
  11. v11External encoding overrides the declared one; larger initial pools12095 min · $41.57
  12. v12Alloc-failure and leak stress harness; external parsers share the root DTD tables12099 min · $44.68
  13. v13Differential harness against system libexpat 2.8.2: 20 divergences fixedStopped treating the visible 129 as the objective: 92 documents x 3 chunk sizes diffed event by event against a real libexpat.124106 min · $51.65
  14. v1423 scripted API scenarios diffed against the reference; three lifecycle fixes124109 min · $54.40
  15. v15Matched expat's malloc-vs-realloc pattern per allocation site125113 min · $59.14
  16. v16XML_SetParamEntityParsing defaults to NEVER, survives reset, standalone downgrades125115 min · $60.76
  17. v17Parameter entities expanded between declarations; ] and ) deferrable at buffer end125120 min · $66.16
  18. v18Mutation fuzzer found 83/500 divergences; prolog tokenizer rewritten from C125127 min · $74.27
  19. v19Multi-byte char legal only inside a name now ends the token as INVALID125128 min · $75.93
  20. v20IS_INVALID_CHAR checked before the naming bitmap; ATTLIST types validated125130 min · $78.12
  21. v21Cached event-pointer slot addresses in do_content; no gain, reverted125133 min · $81.56
  22. v22Pinned the parser pointer in callee-saved %rbx; slightly slower, reverted125135 min · $84.15
  23. v23Seven API scenarios: CDATA default reporting, both suspension points in CDATA125138 min · $87.30
  24. v24Corpus grown to 157 docs; xml-prefix redeclaration and entity-error pointers fixed125141 min · $90.98
  25. v25Allocation-failure sweep over the whole corpus, ~57k parses: no crashes, no leaks125142 min · $94.09
  26. v26NULL-parser guards on the ten entry points that segfaulted where C guards125144 min · $96.80
  27. v27State guards on every configuration entry point; float setter hand-written125147 min · $101.53
  28. v28Billion-laughs accounting implemented in SSE; costs about 6% on the benchmark125152 min · $105.00
  29. v29Root-parser pointer hoisted out of the content loop's accounting125157 min · $106.88
  30. v30Entities carry expat's is_internal flag; ENTITY_DECLARED_IN_PE in standalone cases125158 min · $110.20
  31. v31Last unproduced error code: XML_ERROR_INCOMPLETE_PE, PE between declarations125159 min · $113.50
  32. v32prologTok follower sets: '<' starts the instance unvalidated, ')' needs its own125161 min · $117.77
  33. v33Converted character data delivered in 1024-byte chunks, as expat fills m_dataBuf125167 min · $121.36
  34. v34XML fragments mined out of the visible driver binary: 16 divergences fixed125169 min · $126.37
  35. v35153 driver documents recovered; epilogProcessor made its own loop as in C125174 min · $133.06
  36. v36288 documents split at every byte position; found the CR/LF-pair split rule125175 min · $135.18
  37. v37Random document generator; duplicate-QName and failed-start-tag binding fixes125180 min · $144.19
  38. v3824 external-subset bodies x 3 docs x ns on/off: four parameter-entity bugs fixed125183 min · $149.61
  39. v39External parameter-entity references go through the external entity ref handler125184 min · $152.31
  40. v40External-parser namespace context writes the default binding first, as bare '=uri'125186 min · $155.59
  41. v41External subsets and general entities in the generator; 1000 more docs, 0 diffs125188 min · $158.31
  42. v42Read the 2.6.4 source, not 2.8.2 diffs: int len, PE in entity values, ParserResetDifferential testing was blind here: ctypes sign-extends int len, no corpus doc chained subsets, none called ParserReset.129208 min · $166.48
  43. v431656 parameter-entity configurations: entity created at ENTITY_NAME, bare % invalid129217 min · $171.59
  44. v44Ignored entity declarations still report their literals through the default handler129219 min · $172.60

44 versions, 3.7 h, $173. v1 runs at 77/129 (32 min, $9); 125/129 by v15 (113 min, $59), then 27 versions and $99 before v42 closed the last four.

On the hidden set

held-out metricreward
all-stub anchor_min.s (starting point)0.0098040.00
structural bound (all pass, ref speed)1.01.00
this run0.69320.6901
taskhttps://huggingface.co/datasets/autoresearch-arena/rsi-tasks-1/tree/main/libexpat_x86asm_portrollouthttps://huggingface.co/datasets/autoresearch-arena/rsi-rollout/tree/main/libexpat_x86asm_port/claude-code-opus-5-high/libexpat_x86asm_port__5BrevDv
PreviousDart code formatter in HaskellNextPostgreSQL 18 wire-compatible server on SQLite (Zig)