You are studying one legal matter — a folder of ~130 case documents ("the world")
about the M/V Red Room Ohio River oil spill. You inherit a near-empty
methods/main/memory.md. Your job is to fill it with a distilled, transferable
case memory: the key parties, figures, controlling statutes and clauses, the
established conclusions, and — critically — the exact filenames where each piece
of knowledge lives. Your memory is graded by having a fresh analyst LLM, which
sees ONLY your memory (plus the documents your memory names), answer a set of
held-out questions about this same matter; the more rubric criteria that analyst
satisfies, the higher your score.
Hard Constraints
- You submit
methods/main/memory.mdonly — a plain-text case memory. Do not submit answers to the tasks; per-question answers do not transfer and are not graded. - The graded solver only reads documents your memory names. A document is unlocked for it only if your memory mentions that document's filename (or path). An empty or vague memory unlocks nothing and scores ~0.
- CPU only. Python 3 is available; the world documents are pre-extracted to
text under
tools/world_corpus/. Do not fetch anything external. - Whatever is in
methods/main/memory.mdat the end is graded.
What You Have
- The world —
tools/world_corpus/holds all ~130 case documents as text (contracts, insurance policies, statutes, memos, charter party), plus_index.jsonlisting every document's path. - Visible tasks —
tools/visible_tasks.json: 5 example questions about this matter, each with its binary rubric criteria. These show you the kinds of detail that matter (exact figures, statute subsections, policy names). - A local dev bench —
python3 selfcheck.pyruns the same fixed solver + judge used for grading on the 5 visible tasks with your current memory, and prints which rubric criteria a memory-only solver passed or failed. Iterate: improve memory → re-run → close the gaps.
What You Submit
methods/main/memory.md — a distilled case memory. Good memories name the exact
documents that hold each fact, state the established conclusions (figures,
statutory subsections, coverage determinations), and organise knowledge by topic
so a fresh analyst can find and apply it.
How It Is Judged
- The sealed evaluation runs the same fixed solver + judge on 5 held-out questions about the same matter (different from the visible ones). For each, a fresh solver LLM answers using only your memory and the documents your memory names; a judge LLM scores the answer against that question's binary rubric.
- Your score rises with the fraction of held-out rubric criteria satisfied. A memory that transfers — capturing the matter comprehensively, not just the visible questions — scores highest. How raw scores map to reward is not shown.
Common Pitfalls
- Naming no documents. If your memory never mentions a filename, the solver
reads nothing and answers blind. Name the exact files (as in
_index.json). - Only covering the visible questions. The held-out questions probe other parts of the same matter; distil the whole case, not just the 5 examples.
- Vague pointers without facts. "See the insurance policy" is weak; state the conclusion (which policy covers what, the exact figure, the statutory cite) and name the file.