Tasks/Professional Knowledge Work/Law

Distil a case memory that a fresh analyst can work from

Distil a reusable memory of a telemarketing-regulation matter

apex_telemarketing_reg_memory Professional Knowledge Work Law
instruction.mdthis is what the agent is given

You are studying one legal matter — a folder of ~130 case documents ("the world") about the M/V Red Room Ohio River oil spill. You inherit a near-empty methods/main/memory.md. Your job is to fill it with a distilled, transferable case memory: the key parties, figures, controlling statutes and clauses, the established conclusions, and — critically — the exact filenames where each piece of knowledge lives. Your memory is graded by having a fresh analyst LLM, which sees ONLY your memory (plus the documents your memory names), answer a set of held-out questions about this same matter; the more rubric criteria that analyst satisfies, the higher your score.

Hard Constraints

  • You submit methods/main/memory.md only — a plain-text case memory. Do not submit answers to the tasks; per-question answers do not transfer and are not graded.
  • The graded solver only reads documents your memory names. A document is unlocked for it only if your memory mentions that document's filename (or path). An empty or vague memory unlocks nothing and scores ~0.
  • CPU only. Python 3 is available; the world documents are pre-extracted to text under tools/world_corpus/. Do not fetch anything external.
  • Whatever is in methods/main/memory.md at the end is graded.

What You Have

  • The worldtools/world_corpus/ holds all ~130 case documents as text (contracts, insurance policies, statutes, memos, charter party), plus _index.json listing every document's path.
  • Visible taskstools/visible_tasks.json: 5 example questions about this matter, each with its binary rubric criteria. These show you the kinds of detail that matter (exact figures, statute subsections, policy names).
  • A local dev benchpython3 selfcheck.py runs the same fixed solver + judge used for grading on the 5 visible tasks with your current memory, and prints which rubric criteria a memory-only solver passed or failed. Iterate: improve memory → re-run → close the gaps.

What You Submit

methods/main/memory.md — a distilled case memory. Good memories name the exact documents that hold each fact, state the established conclusions (figures, statutory subsections, coverage determinations), and organise knowledge by topic so a fresh analyst can find and apply it.

How It Is Judged

  • The sealed evaluation runs the same fixed solver + judge on 5 held-out questions about the same matter (different from the visible ones). For each, a fresh solver LLM answers using only your memory and the documents your memory names; a judge LLM scores the answer against that question's binary rubric.
  • Your score rises with the fraction of held-out rubric criteria satisfied. A memory that transfers — capturing the matter comprehensively, not just the visible questions — scores highest. How raw scores map to reward is not shown.

Common Pitfalls

  • Naming no documents. If your memory never mentions a filename, the solver reads nothing and answers blind. Name the exact files (as in _index.json).
  • Only covering the visible questions. The held-out questions probe other parts of the same matter; distil the whole case, not just the 5 examples.
  • Vague pointers without facts. "See the insurance policy" is weak; state the conclusion (which policy covers what, the exact figure, the statutory cite) and name the file.